Introduction: A Hearing MeArm, Google Coral TPU Accelerator Driven
In the following I would like to describe a voice-controlled version of the MeArm, a small xyz robot arm with a gripper. I used the MeArm Pi from MIME industries, but the system should be applicable to any version of he MeArm, or similar servo-driven devices.
Using the Google Coral TPU Accelerator allows to run rapid offline TensorFlow voice recognition scripts on the Raspberry Pi, and hereby control physical devices by spoken orders, with a latency below one second.
The device described herein is a combination, and extension, of concepts described in two previous instructables. It is a extension of an earlier implementation of the Google Coral voice control, a Jumping Jack, described here and a vast improvement of a Google AIY voice controlled MeArm described here.
The voice-controlled MeArm using the Google Voice AIY system required online access, was not easy to implement, required to press a button to activate listening for voice orders and had a long latency time. The Google Coral TPU Accelerator used now allows to run TensorFlowLite models offline with a high velocity on a Raspberry Pi or other Linux devices. Among the examples on the Google Coral Github page there is an example called "hearing snake" for a voice recognition system that can understand 140 key phrases (Sept 2019) , which are then mapped to virtual keystrokes. Coupling these "keystrokes" with the execution of some functions programmed in Python makes it possible to build a a voice command controlled device. I recently had described a first implementation, a voice-controlled electromechanical jumping jack.The implementation herein is a bit more complex and allows to control all four servos of the MeArm to either move the MeArm continuously or have it to move to a number of predefined positions, or to perform some more complex tasks.
Using the script provided here as example, it should be relatively simple to construct other voice-controlled devices, e.g. robotic cars or assistive tech units.
Supplies
- MeArm. Used here: MeArm Pi from MIME Industries
- Raspberry Pi 4
- Google Coral TPU Accelerator
- Adafruit 16 channel servo bonnet
- some jumper cables
- optional: capacitor for servo bonnet, about 400 µF for 4 servos (recommended by Adafruit)
- 5-6 V power source for servo bonnet. I here used an old 6V charger, a 4x AA battery pack works as well
- Microphone. I used an old Microsoft HD3000 webcam as microphone.
Step 1: Setting Up the System
Download the preconfigured Raspian image for the Google Coral TPU Accelerator from the Google Coral Github page and install it on a µSD card. The image contains also a number of example scripts. Set up the Pi as indicated.
Install the example Keyword spotter from the Google Coral GitHub site, if not included in the image, and all required programs. Attach the microphone to the Pi. I would recommend to play with the "Hearing Snake" example to make sure everything is working.
Download and install the Adafruit 16 channel bonnet software, as described here.
Install the bonnet and play with the Adafruit examples to ensure everything is working properly.
Download the files attached to this instructable and copy them to the "Project Keyword Spotter" folder. The "commands_v1_MeArm.txt" file must be copied to the "config" subfolder.
Connect your MeArm's servos to the servo bonnet as indicated. I used port 15 for up/down, port 11 for forward/backwards, port 7 for turn and port 3 for the gripper servos.
Within the script you may have to adjust the min/center/max values for each servo to your configuration, These settings help to avoid damage to the servos. You may also have to modify the included "positions", "transport1" and "transport2" lists.
Run the script. So far I had been running it from the IDE.
In case you would like modify the key phrases that evoke a certain function according to your need.
A complete list of available KeyPhrases are found in the "labels_gc2 raw.txt" file in the config subfolder.
The system has a latency time of about 1 second, but depending much on which actions are performed. In some cases the key phase has to be repeated, accuracy of recognition is not always 100%.
Step 2: Using the Device
If everything is set up and checked, you may run the device.
A current limitation is that a given order is executed repetitively as long as it is not stopped (using "stop game") or another order is given. Complex multistep tasks, e.g. "transport1" (evoked by the phrase "launch game") are always executed to the final step.
So by "turn right" the device will move in small steps to the right until stopped, or the preset maximum value is reached. "launch game", "next game" or "start_video" will start a series of moves that are defined by lists containing the setting for each servo at a given step. "random game" will the device to jump from one to another step, picked randomly from a list of settings.
As you may see in the accompanying video, I had build a diabolo shaped object from LEGO that can be picked up by the MeArm and be transported from one location to another by a predefined set of movements. You may define your own functions by modification of the 'transport1' or 'transport2' lists.
Step 3: The Script
The script listed here is a modification of the "Hearing Snake" example from "Project Keyword Spotter". The example has been stripped down to a minimum, then the part for driving the servos was added, based on the software and examples provided for the Adafruit servo bonnet.
The script has not been optimized by now. Use on your own risk, feel free to modify and optimize.
In addition to the python script there is the commands-file and the used labels-file. Place it in the config-subfolder.
As mentioned before, several adjustments of parameters might be required to adapt the script for your special MeArm or some other device.
# Copyright 2019 Google LLC# # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # href="https://www.apache.org/licenses/LICENSE-2.0" href="https://www.apache.org/licenses/LICENSE-2.0" https://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # the original "hearing_snake" code was modified for an implemention for the MeArm by Dr H. ''' Instructions My implementation uses a Raspbery Pi 4 with a Google Coral accelerator and an Adafruit 16 channel servo bonnet attached. The servos of a MeArm (MIME industries) were attached to ports 3, 7, 11 and 15 of the bonnet. For details please have a look on the "Hearing MeArm" Instructable. Commands: "position x", x= 0 to 9, moves the device to a given predefined position. "move/go up", "move/go down", "go/turn forwards", "go/turn backwards", "turn/go left" and "turn/go right" evoke a slow, stepwise movement in the given direction, "stop game" stops the movements. "open tab" and "close tab" opens or closes the gripper. "start video" evokes the device to go follow a preset order of positions, defined by the list 'positions'. "random game" results in a random pattern of movements, "stop game" ends it. "launch game" starts another series of moves predefined by the list 'transport1', "next game" the reverse operation predefined by 'transport2' Use on your own risk. ''' from __future__ import absolute_import from __future__ import division from __future__ import print_function import argparse import os from random import randint from threading import Thread import time from edgetpu.basic.basic_engine import BasicEngine import model import pygame from pygame.locals import * import queue from random import randrange from adafruit_servokit import ServoKit import board import busio import adafruit_pca9685 import time i2c = busio.I2C(board.SCL, board.SDA) hat = adafruit_pca9685.PCA9685(i2c) hat.frequency = 60 kit = ServoKit(channels=16) # set number of channels #kit.servo[0].actuation_range = 160 #kit.servo[0].set_pulse_width_range(1000, 2000) # min, center and max settings up_l = 145 # servo up/down: up md_l = 95 dn_l = 45 up_r = 135 # servo forward/backward md_r = 90 dn_r = 50 ri_t = 30 # turning arm right or left: right position md_t = 90 # turning arm right or left: center position le_t = 150 op_g = 65 # gripper open md_g = 90 # gripper centered cl_g = 130 # gripper closed vert = 15 # number of servo port, servo up/down forw = 11 # number of servo port, forward/backward moving servo turn = 7 # servo port for turning servo grip = 3 # servo port for grip servo #list of arm settings for nine positions position = [(md_l,md_r,md_t,op_g), (up_l,md_r,ri_t,op_g),(up_l,md_r,md_t,cl_g),(up_l,md_r,le_t,cl_g), (md_l,md_r,md_t,op_g),(md_l,md_r,md_t,md_g),(md_l,md_r,md_t,cl_g), (dn_l,dn_r,ri_t,op_g),(dn_l,dn_r,md_t,md_g),(dn_l,dn_r,le_t,md_g)] # defines 10 base positions, indicated by integers 0-9 # transport procedures [vert/forward/turn/grip] transport1 = [(140,70,65,op_g),(110,50,65,op_g),(65,50,65,op_g),(65,70,65,cl_g),(120,70,65,cl_g), #get object (100,70,135,cl_g),(100,80,135,cl_g), (100,80,135,md_g),(100,80,135,op_g),(140,70,135,op_g),(140,70,90,op_g),(140,70,65,op_g)]</p><p>transport2 = [(140,70,65,op_g),(140,70,135,op_g),(95,70,135,op_g),(95,80,135,op_g), (95,80,135,cl_g),(110,70,135,cl_g),(110,70,65,cl_g),(70,70,65,cl_g), (70,70,65,op_g),(80,50,65,op_g)]</p><p>dance1 =(0,8,7,4,1,2,3,6,9,8,5,2,1,4,7,8,9,6,3,2,0) # a "dance" #moving MeArm to Zero position status =[md_l,md_r, md_t,md_g] kit.servo[vert].angle = status[0] kit.servo[forw].angle = status[1] kit.servo[turn].angle = status[2] kit.servo[grip].angle = status[3] print (status) class Controler(object): #Callback function def __init__(self, q): self._q = q def callback(self, command): self._q.put(command) class App: def __init__(self): self._running = True def on_init(self): pygame.init() self.game_started = True self._running = True return True def on_event(self, event): if event.type == pygame.QUIT: self._running = False def MeArmPos(self, keys): # drives MeArm to preset positions, keywords: "position x" key = int(keys) p = position[key] a = p[0] b = p[1] c = p[2] d = p[3] print ("Positions: ", key, " vert/forw/turn/grip: ",a,"/",b,"/",c,"/",d,"degrees") status = [a,b,c,d] # documents current status print (status) # sys.stdout.write("Position: ", key, " left/right: ",a,"/",b,"degree") kit.servo[vert].angle = a kit.servo[forw].angle = b kit.servo[turn].angle = c kit.servo[grip].angle = d time.sleep(0.5) def DancingMeArm(self): # controls MeArm dance, keyword: "start_video" dnce = dance1 sp=(len(dnce)) for r in range (sp): #dancing order of positions, sp steps dc = dnce[r] p = position[dc] a = p[0] b = p[1] c = p[2] d = p[3] kit.servo[vert].angle = a kit.servo[forw].angle = b kit.servo[turn].angle = c kit.servo[grip].angle = d time.sleep(1) # sets velocity of movements time.sleep(0.5) # break at the end of procedure def TransMeArm1(self): # controls MeArm transport 1, keyword: "launch game" tr1 = transport1 sp=(len(tr1)) #calculate number of steps for r in range (sp): #go to any step p = tr1[r] a = p[0] b = p[1] c = p[2] d = p[3] kit.servo[vert].angle = a kit.servo[forw].angle = b kit.servo[turn].angle = c kit.servo[grip].angle = d print (p) time.sleep(1) # sets velocity of movements time.sleep(0.5) def TransMeArm2(self): # controls MeArm dance, keyword: "next game" tr2 = transport2 sp=(len(tr2)) for r in range (sp): #dancing order of positions, sp steps p = tr2[r] a = p[0] b = p[1] c = p[2] d = p[3] kit.servo[vert].angle = a kit.servo[forw].angle = b kit.servo[turn].angle = c kit.servo[grip].angle = d print (p) time.sleep(1) # sets velocity of movements time.sleep(0.5) def RandomMoves(self): # jumps randomly between predefined positions, keyword: "random game" dr= randrange (9) #randomly selects a position p = position[dr] # reads position parameters a = p[0] b = p[1] c = p[2] d = p[3] kit.servo[vert].angle = a kit.servo[forw].angle = b kit.servo[turn].angle = c kit.servo[grip].angle = d time.sleep(1) # sets velocity of movements def MoveUp(self): # lifting gripper in small steps u0 = status[0] # read current status u1 = u0 + 5 # plus x degrees if (u1 > up_l): # tests if not exceeding min/max parameters u1 = up_l # otherwise set to min/max value kit.servo[vert].angle = u1 # move servo status[0] = u1 # adjust status value print ("up ", status) time.sleep (1) # sets velocity def MoveDown(self): d0 = status[0] d1 = d0 - 5 #minus x degrees if (d1 < dn_l): d1 = dn_l kit.servo[vert].angle = d1 # move servo status[0] = d1 print ("down ", status) time.sleep (1) def MoveForw(self): f0 = status[1] f1 = f0 + 5 #plus x degrees if (f1 > up_r): f1 = up_r kit.servo[forw].angle = f1 # move servo status[1] = f1 print ("forward ", status) time.sleep (1) def MoveBack(self): b0 = status[1] b1 = b0 - 5 #minus x degrees if (b1 < dn_r): b1 = dn_r kit.servo[forw].angle = b1 # move servo status[1] = b1 print ("back ", status) time.sleep (1) def MoveLeft(self): l0 = status[2] l1 = l0 + 2 #plus x degrees if (l1 > le_t): l1 = le_t kit.servo[turn].angle = l1 # move servo status[2] = l1 print ("left ", status) time.sleep (0.2) def MoveRight(self): r0 = status[2] r1 = r0 - 2 #minus x degrees if (r1 < ri_t): r1 = ri_t kit.servo[turn].angle = r1 # move servo status[2] = r1 print ("right ", status) time.sleep (0.2) def OpenGrip(self): kit.servo[grip].angle = op_g # set grip to "open" position: "open_tab" time.sleep(0.5) status[3] = op_g def CloseGrip(self): kit.servo[grip].angle = cl_g # set grip to "closed" position: "close_tab" time.sleep(0.5) status[3] = cl_g def StopMove(self): # does nothing, but stops movements print ("stop ", status) time.sleep(0.25) def spotter(self, args): engine = BasicEngine(args.model_file) mic = args.mic if args.mic is None else int(args.mic) model.classify_audio(mic, engine, labels_file="config/labels_gc2.raw.txt", commands_file="config/commands_v1_MeArm.txt", dectection_callback=self._controler.callback, sample_rate_hz=int(args.sample_rate_hz), num_frames_hop=int(args.num_frames_hop)) def on_execute(self, args): if not self.on_init(): self._running = False q = model.get_queue() self._controler = Controler(q) if not args.debug_keyboard: t = Thread(target=self.spotter, args=(args,)) t.daemon = True t.start() item = -1 while self._running: pygame.event.pump() if args.debug_keyboard: keys = pygame.key.get_pressed() else: try: new_item = q.get(True, 0.1) except queue.Empty: new_item = None if new_item is not None: item = new_item if (args.debug_keyboard and keys[pygame.K_ESCAPE]) or item == "stop": self._running = False # if (args.debug_keyboard and keys[pygame.K_SPACE]) or item == "go": # self.MeArmPos(7) # if (args.debug_keyboard and keys[pygame.K_RIGHT]) or item == "right": # turn right self.MoveRight() if (args.debug_keyboard and keys[pygame.K_LEFT]) or item == "left": # turn left self.MoveLeft() if (args.debug_keyboard and keys[pygame.K_UP]) or item == "up": self.MoveUp() if (args.debug_keyboard and keys[pygame.K_DOWN]) or item == "down": self.MoveDown() if (args.debug_keyboard and keys[pygame.K_B]) or item == "b": # backwards self.MoveBack() if (args.debug_keyboard and keys[pygame.K_F]) or item == "f": # forwards self.MoveForw() if (args.debug_keyboard and keys[pygame.K_O]) or item == "o": # open grip: self.OpenGrip() if (args.debug_keyboard and keys[pygame.K_C]) or item == "c": # close grip: self.CloseGrip() if (args.debug_keyboard and keys[pygame.K_S]) or item == "s": # stop movement: "start_game" self.StopMove() if (args.debug_keyboard and keys[pygame.K_0]) or item == "0": self.MeArmPos(0) if (args.debug_keyboard and keys[pygame.K_1]) or item == "1": self.MeArmPos(1) if (args.debug_keyboard and keys[pygame.K_2]) or item == "2": self.MeArmPos(2) if (args.debug_keyboard and keys[pygame.K_3]) or item == "3": self.MeArmPos(3) if (args.debug_keyboard and keys[pygame.K_4]) or item == "4": self.MeArmPos(4) if (args.debug_keyboard and keys[pygame.K_5]) or item == "5": self.MeArmPos(5) if (args.debug_keyboard and keys[pygame.K_6]) or item == "6": self.MeArmPos(6) if (args.debug_keyboard and keys[pygame.K_7]) or item == "7": self.MeArmPos(7) if (args.debug_keyboard and keys[pygame.K_8]) or item == "8": self.MeArmPos(8) if (args.debug_keyboard and keys[pygame.K_9]) or item == "9": self.MeArmPos(9) if (args.debug_keyboard and keys[pygame.K_a]) or item == "d": self.DancingMeArm() #dancing MeArm, on "next_game" if (args.debug_keyboard and keys[pygame.K_r]) or item == "r": self.RandomMoves() #random dance "random game" if (args.debug_keyboard and keys[pygame.K_j]) or item == "j": self.TransMeArm1() # transport object: "lunch_game" if (args.debug_keyboard and keys[pygame.K_k]) or item == "k": self.TransMeArm2() # transport object reverse direction: "next_game" ''' if (args.debug_keyboard and keys[pygame.K_l]) or item == "l": self.JumpingJack2(1) #LED blink "target" ''' time.sleep(0.05) self.on_cleanup() if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument( '--debug_keyboard', help='Use the keyboard to control the MeArm.', action='store_true', default=False) model.add_model_flags(parser) args = parser.parse_args() the_app = App() the_app.on_execute(args)
Participated in the
Make it Move
