Introduction: TJBot - Use Your Voice to Control a Light With Watson

This instructable guides you through connecting a Raspberry Pi to Watson Speech to Text and change the color of a LED through speech. For example, you may say "Turn the light green" to change the color of the LED to green.

Step 1: Parts

Step 2: Prepare Your Pi

If you have used Raspberry Pi before, install Node.js and the Alsa packages and go to the next step. Otherwise, follow the instructions below to set up your Pi:


Getting Started with Your Pi

Raspberry Pi is similar to a full computer, which means you need a monitor, mouse, and keyboard for it. If you have a TV around, you can connect your Pi to your TV via a HDMI cable. In most of the Pi kits, the SD card is already preloaded with an image of the Raspberry Pi Operating System. You need to put the SD card in the Pi, turn the Pi ON and follow the instructions on screen to complete the installation of the operating system. If you have problems setting up your Pi, you can troubleshoot here.

Here is an easy way to get your Pi connected to wifi.


Install Packages

Open a terminal application on the Pi and execute the following commands to install the latest version of Node.js and npm (Node Package Manager). You need these packages later to run your code.

sudo apt-get update

sudo apt-get dist-upgrade

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -

sudo apt-get install -y nodejs

sudo apt-get install alsa-base alsa-utils libasound2-dev

Step 3: Assemble TJBot

Here is a quick video of how to fold the kit. The detailed instructions are available on another instructable: Build TJBot out of Cardboard

Once your TJBot is ready, get ready to connect your LED to the Pi Pins.

Attention

It's very important to make sure you connect the LED to the right pins, otherwise you may damage your LED or the Pi.
ALWAYS start with the GND pin. GND is the longest leg of your LED. The LED has a flat side. The GND is the second leg from the flat side. Again if you are unsure, DO NOT connect the LED to the Pi.




Connect the GND to GND pin on the pi (see the picture). Then connect 'Data in', the 4th leg from the flat side, to the Pi's GPIO pin. Lastly, connect the third leg from the flat side to the 3.3V on the Pi.

Step 4: Download the Sample Code

The sample code is available at github. Download or clone the sample code and execute the following commands from a terminal to install its dependencies.

Here are the instructions for how to clone a repository from github if you have not done that before.

git clone https://github.com/ibmtjbot/tjbot.git
cd tjbot/recipes/speech_to_text
npm install

Step 5: Update Your Credentials

In this step, we help you get API access to the Watson Speech to Text. Let's start with creating a Speech to Text instance on Bluemix (If you don't have a Bluemix account, follow the instructions to create a free trial).

You may leave the default values and select 'Create'.


Almost done! You just need to copy your credentials to the clipboard.
Click on 'Sevice Credentials' on the left menu and copy your credentials into the clipboard.


Now go to the speech_to_text folder and make a copy of the default config file.

cp config.default.js config.js

Open config.js file using your favorite text editor and update it with your username and password.

Step 6: Run the Code

Once done with testing the LED, you can run the main module:

sudo node stt.js

Now talk to your microphone to change the color of the LED.
Say "Turn the light blue" to change the light to blue. You can try other colors: yellow, green, orange, purple, magenta, red, blue, aqua, white). You can either say "Turn the light on" or "Turn the light off"!


Troubleshoot

Is the LED not lighting up? No worries, we can fix it.

You may test your LED setup to make sure the connections are correct and the library is properly installed. When you run the test module, it should turn your LED on.

cd tjbot/bootstrap/tests 
npm install 
sudo node test.led.js

If the LED does not light up, you can try moving the power from 3.3 to 5 volts. If neither the 3.3v or 5v pins work, you will need a 1N4001 diode. The diode is inserted between the power pin of the LED (the shorter of the two middle pins) and the 5v pin on the Raspberry Pi. If you have problems with the setup, please refer to Adafruit's Neopixel on Raspbeery Pi guide to troubleshoot.

The LED library uses the PWM module (GPIO 18) to drive the data line of the LEDs. This conflicts with the built-in audio hardware, which uses the same pin to drive the audio output. Depending on your configuration of Raspbian, the sound drivers may be more aggressive in taking away control of GPIO 18 from other processes. If your LED shows random colors instead of the expected color, use this trick to fix it.

cd ..
sudo cp tjbot-blacklist-snd.conf /etc/modprobe.d/
sudo update-initramfs -u

Reboot and confirm no "snd" modules are running by executing the command "lsmod".

lsmod

Enjoy! Don't forget to share a picture/video of your setup #TJBot! :-)

Step 7: What's Next?

You can add new colors to your color palette in stt.js. TJBot uses a NeoPixel RGB LED, which means it can show any combination of red, green, and blue.

We have hidden a disco party for you. Find the code for disco party in stt.js and uncomment the code. Now you can ask TJBot to show you the disco lights by saying "Let's have a disco party"!

Try implementing your own TJBot party and share it with us #TJBot!

Once ready to move on, try the next recipe to make TJBot respond to emotions using Watson.

Comments

author
duyhat (author)2017-03-11

Hi,

Thanks for the interesting tutorial and nice recipe!

I have a Common Anode + LED instead of the one used in tutorial so I went ahead and made a quick change to the code to make it work with my LED, if someone wants to try, here is the separated branch in my fork:

https://github.com/dnguyenv/tjbot/tree/common-anod...

Changes:

- Removed rpi-ws281x-native library, use rpio npm lib
- Changed the main tss.js file to work with my LED
- Added my wire diagram (https://github.com/dnguyenv/tjbot/blob/common-anode/recipes/speech_to_text/tts.png)
- Changed README to be relevant to the change

Best - Duy

author
cmackin (author)duyhat2017-08-01

Hi Duy. I see your code for a multi color solution. It is so much easier to understand your code and I think it is closer to what I am trying to do. Can you help me to create an even a simpler code that will set a specified GPIO pin to be an output and then be able to set it HIGH or LOW based on voice commands? If possible, I would like to add this functionality to the Conversation TJbot recipe so that I can have a conversation as well as issue commands to switch GPIO output pins HIGH and LOW.

author
cmackin (author)2017-08-01

These instructions are awesome. I just have to follow instructions and they always work. I do want to ask however if someone can anyone help me to modify the TJbot LED Voice control project that controls the PWM GPIO pin? I would like to first understand how to modify the program to simply point to a specified GPIO pin and then change it to an output pin and then be able to set the pin HIGH and LOW in accordance with Speech-to-text voice control. To be honest, I can't figure out where the actual code is located that points to the PWM pin


Then, ideally, the voice control part of this recipe would be added to the conversation recipe so that I can have a conversation with the robot, but when a command is heard, a GPIO pin will be controlled to be set HIGH or LOW.

By way of example, I would say

User: Watson, Hello, how are you?
Watson: I am doing very well. Thank you for asking.
User: Turn on the red LED
Watson will then cause the GPIO 5 to be set HIGH, which will switch a circuit that will route power to a red LED.
User: Watson, thank you for turning the LED on
Watson; Your very welcome.
User: Watson, Turn on the White LED.
Watson will then cause the GPIO 4 to be set HIGH, which will switch a circuit that will route power to a white LED.

author
jman24saints (author)2017-05-06

Hi,

Our light will only stay blue. We tried doing the test and when the light should turn red it stays blue. we have checked that all of our connections are correct, and even tried switching from the 3.3V to 5V and that didn't work either. We also can't get past initializing microphone in the terminal.

author
geneLo (author)jman24saints2017-05-24

Having the same issue with the microphone. Did you ever get passed it?

author
MauricioM82 (author)2017-02-15

Hi.

When I run stt.js program it shows the message:
"TJBot is listening, you may speak now."
but don't give me time to speak and close the program right after showing the message. No errors, nothing, just show the message and close the program.

I am using an USB sound card with both input and output. Both arecord and aplay commands works just fine, I am able to record and listen to the audio I've recorded. I am also able to use the voice control to control bb-8.

Any suggestions?

Thank you.

author
victordibia (author)MauricioM822017-02-15

Hi MauricioM82,

You are correct, the issue is definitely related to the app having difficulty setting up audio. For arecord on command line, do you pass any parameters for your app to work correctly ? For example if your sound input might be listed with an unexpected hardware ID or might need several parameters to work.

To find out your hardware device ID run arecord -L ,and then specify that as a parameter when you instantiate the mic in stt.js.

In the sample apps we provide, we use the mic npm library (

https://www.npmjs.com/package/mic) to record audio. You can provide hardware options when you instantiate your mic to record audio e.g

var micInstance = mic({ 'rate': '44100', 'channels': '2', 'debug': false, 'exitO

nSilence': 6, 'device': 'plughw:1,0' });


A related issue (solved) on recording from a bluetooth and usb mic can be found here

https://github.com/ibmtjbot/tjbot/issues/12

author
geneLo (author)victordibia2017-05-24

Hi Victordibia

First off, thanks for this fun and easy to follow recipe!

I've followed all the steps on the tutorial however i'm getting an error after running the command "sudo node stt.js".

I believe the trouble is derived from raspbian jessie not seeing my usb mic.

The error reads "the speech_to_text service returned an error" and then prints lines of code. It appears right after the console logs "TjBot initializing microphone"

I've ran asplay -l and a usb sound card does not appear. However when i go to the raspberry pi settings > preferences > audio devices> the dropdown shows my usb mic, but when I attempt at making it the default audio, it never saves the change.

I've been stuck on this problem some time, so any help would be greatly appreciated.

I'm running Raspbian 8.0 Jessie.

Thanks for reading and I hope to hear from you soon!

author
MauricioM82 (author)victordibia2017-02-15

Great!

Thank you for your help. So, I specified the device when instantiating the mic and it worked, it now recognizes my voice.

The problem now is that the led don't change colors (it's green, and always green).

I did the suggestions on the Troubleshooting part of the article but after reboot I see several "snd" modules loaded, like

snd_usb_audio 118800 3
snd_hwdep 5727 1 snd_usb_audio
snd_usbmidi_lib 19801 1 snd_usb_audio
snd_rawmidi 18516 1 snd_usbmidi_lib
snd_seq_device 3621 1 snd_rawmidi
snd_pcm 75762 1 snd_usb_audio
snd_timer 19288 1 snd_pcm

Any suggestions?

Thank you in advance.

author
MauricioM82 (author)MauricioM822017-02-15

Maybe I found the answer.. I may be using a different type of led.

We have the ground pin and one of each other is for one color.

Does the led used on the article has a special name, or some way I can identify it to purchase in a store?

Thank you.

author
victordibia (author)MauricioM822017-02-15

Great!
We use the RGB Neopixel LED based on the ws2812 protocol.

https://www.adafruit.com/products/1734

Really look forward to seeing what you build! Pictures, videos ? You can share on twitter #tjbot

V.

author
MauricioM82 (author)victordibia2017-02-15

I live in Brazil and it's really hard to find the adafruit version or anything similar around here. And shipping from adafruit site to Brazil is way way way to expensive.

Maybe someone could ship a kit from the site to me via regular mail? :D

Thank you for all your help.

author
TJBot (author)MauricioM822017-02-15

We used a NeoPixel LED. Here is a link to the specs: https://www.adafruit.com/products/1734

author
DivyaB20 (author)2017-05-05

Hi,

I am using raspberry Pi 3 model B with Hardware : BCM2835.

All connections are made as per tutorial. My LED is shining blue all the time.

I installed rpi-ws281x-native, but when i connect to Watson Speech to text, it says connected to TJ bot.

next it gives error

[rpi-ws281x-native] Could not verify raspberry-pi version. If this is wrong and you are running this on a raspberry-pi, please file a bug-report at https://github.com/beyondscreen/node-rpi-ws281x-n...

Any help is welcome in this regard.

Thanks

Divya

author
PiotrJ16 (author)DivyaB202017-05-13

Hello,

I had the same problem and I fixed it by recompiling the library rpi-ws281x-native

To do it:

go in the catalog with your project

uninstall the old library

npm uninstall rpi-ws281x-native

install sources of the new library

sudo npm install -g node-gyp

git clone --recursive https://github.com/beyondscreen/node-rpi-ws281x-n...

cd node-rpi-ws281x-native/lib

there is a file in this folder ws281x-native.js and you need to edit it and remove the following code

if (raspberryVersion === 0) { console.warn('[rpi-ws281x-native] Could not verify raspberry-pi ' + 'version. If this is wrong and you are running this on a ' + 'raspberry-pi, please file a bug-report at ' + ' https://github.com/beyondscreen/node-rpi-ws281x-n... + 'A non-functional stub of this modules interface will be ' + 'returned.'); return stub;

save the file

cd ..

npm install

node-gyp rebuild

mv node-rpi-ws281x-native node_modules/rpi-ws281x-native

This is more or less what needed to be done :) I hope it helps you to find the right solution for you.

author
DivyaB20 (author)PiotrJ162017-05-17

Hi,

The node-rpi-ws281x-native has provided a fix. Now I am able to run the basic functionality test rainbow.js which is mentioned in https://github.com/beyondscreen/node-rpi-ws281x-na...

A directory node-rpi-ws281x-native is installed.

Issue is when I use the Tjbot example, I am still getting the same error.

I tried mv the node node-rpi-ws281x-nativeto rpi-ws281x-native.

But still getting error with Tjbot example

What change is required to work this with tjbot?

[rpi-ws281x-native] Could not verify raspberry-pi version. If this is wrong and you are running this on a raspberry-pi, please file a bug-report at https://gi thub.com/beyondscreen/node-rpi-ws281x-native/issues

Thanks

Divya

author
rewillen (author)DivyaB202017-05-09

I'm also having the problem that the light is always blue. The console is showing that TJBot is hearing the commands correctly but the light isn't changing. I purchased the light from Adafruit.

I was successful getting the conversation to work, just not the light.

author
DivyaB20 (author)rewillen2017-05-17

Please use the Strandtest by adafruit. This will ensure the LED is showing correct colours
https://learn.adafruit.com/neopixels-on-raspberry-pi/software

author
kifumi (author)DivyaB202017-05-09

Hi, I got the same problem with the latest Raspbian Jessie (2017-04-10). One month ago, my TJbot worked fine with same hardware with old version Jessie. So, I solved the problem by following;

1) Install the latest Raspbian Jessie (2017-04-10).

2) Do NOT execute "sudo apt-get update" and "sudo apt-get dist-upgrade".

3) Execute "curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -

", "sudo apt-get install -y nodejs", and "sudo apt-get install alsa-base alsa-utils libasound2-dev".

author
TolgaD6 (author)DivyaB202017-05-05

Having the exact same issue.

author
Johsto (author)2017-05-05

I can't find any directory tjbot/bootstrap/

for testing the LED.

Johann

author
chris-sim (author)2017-05-01

Hi,
I finished assembly my TJ and can run all services successfully. Only nasty points are that the Speech to Text is quite never translating correctly. Also in this case here, the LED color wont change. At the shell output the software does it quite fine. But eventhough it tells me the color has been changed... nothing happens.
Wiring is exactly as stated here, also the LED type.
It's just shining a beatiful blue all the time...

Any tips&tricks welcome.

author
rhendro (author)2017-04-30

I think my USB Mic is not recognized by speech_to_text although it works with arecord/aplay.

>>verbose: TJBot library version v1.2.1

>>I understand lots of colors. You can tell me to shine my light a different color by saying 'turn the light red' or 'change the light to green' or 'turn the light off'.

>>verbose: TJBot initializing microphone

>>error: the speech_to_text service returned an error. Error: Session timed out, no data received in the last 30 seconds.

I read below that the mic setting can be modified like this:

>> var micInstance = mic({ 'rate': '44100', 'channels': '2', 'debug': false, 'exitO

nSilence': 6, 'device': 'plughw:1,0' });

My question, which module has this var micInstance? I only find var micParams.

Thanks..

author
rhendro (author)rhendro2017-04-30

Well, I'm quite sure now that the mic is recognized. I managed to turn on the mic function debug: true and got this:

>>Found silence block: 481 of 6

>>Found silence block: 482 of 6

>>Found speech block

>>Found speech block

>>Found speech block

>>Found speech block

>>Found speech block

>>Found silence block: 1 of 6

>>Found silence block: 2 of 6

The timeout suspect is now with speech_to_text call. I have moved the region from US South to Sydney but same result. I tried this

curl -X POST -u {username}:{password}
--header "Content-Type: audio/flac"
--header "Transfer-Encoding: chunked"
--data-binary @{path}audio-file.flac
"https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true"

It responded back fast no timeout so it is not network issues.

Any hint?

author
jdpurce61DIY (author)2017-04-18

Just completed this recipe and can verify the workaround to blacklist sound drivers is necessary as of now. Wonder if this will be an issue with recipes requiring TJBot to "speak."

uname -a reports "4.4.50-v7+ #970 SMP Mon Feb 20 19:18:29 GMT 2017 armv7l"

Nice recipe. Thanks for creating.

--John.

author
OriginalDopey (author)2017-03-17

I am trying to merge this project with the conversation project. My goal is to have the LED turn green if the name was recognized before passing to conversation and text to speech. This will give the user a notification that the services are working. However, when I run the conversation app as sudo, I no longer get audio through the bluetooth speaker. Since sudo is required to run the LED, this is an issue. Has anyone worked through this before and gotten the audio to work through sudo or the light to work without sudo?

author
TJBot (author)OriginalDopey2017-03-23

Did you unblock the sound modules? Raspberry Pi has a known issue of PWD conflicting with audio jack. This means you cannot use the PWD pin (used for the LED) and the audio together. To solve this, we suggested on the last step to block all the sound modules https://www.instructables.com/id/Use-Your-Voice-to-Control-a-Light-With-Watson/step7/Run-the-Code/ so if you done that, you need to undo this first and then use an external sound card to get the audio. Here is the sound card we've been using https://learn.adafruit.com/usb-audio-cards-with-a-raspberry-pi/instructions

author
JamieC135 (author)2016-12-13

Can anyone suggest a cheap usb microphone I could buy because my budget isn't big enough for a $70 one

author
TJBot (author)JamieC1352016-12-13

Hi Jamie, I've been using https://www.amazon.com/gp/product/B00IR8R7WQ/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1 and it works well for the price.

author
AlexanderT78 (author)2016-11-26

can i use raspberry 2 instead of raspberry3

author
TJBot (author)AlexanderT782016-11-26

It should work on Raspberry 2. I haven't tested it, but victordibia confirmed he was able to run it on Pi2.

author
AlexanderT78 (author)2016-11-26

hi, yes right i have a talking robot with Watson Conversation. i fixed the credentials, but still my loudspeaker do not work, i tried it with an apple headset, but also no sound. i used sudo raspi-config and changed the audio settings, but no sound.

i also tried the code

sudo amixer cset numid=3 1, but no sound

Alexander

author
TJBot (author)AlexanderT782016-11-26

Alexander, I'm glad your payload quota error is resolved. If you direct the sound to the right channel and still don't get any output sound, I guess the problem is the blacklist. Try the troubleshoot on step 7 and make sure the Pi is not blocking the sound output: https://www.instructables.com/id/Build-a-Talking-Robot-With-Watson-and-Raspberry-Pi/step7/Run-the-Code/

author
AlexanderT78 (author)2016-11-24

hi,

i have the talking tjbot and also getting the error

TJ is listening, you may speak now.

===== An Error has occurred =====

You may have exceeded your payload quota.

[object Object]

Press <ctrl>+C to exit.







i checked all credentials and compared with my bluemix credentials, they are all ok. in conversation i do not have only export.username, export.password, also i have export.conworkspace, but i do not see it in the bluemix credentials! what i have to enter in export.conworkspace.

greetings

alexander

author
TJBot (author)AlexanderT782016-11-25

Hi Alexander, you are right, the payroll quota error is usually caused by incorrect credentials.

Which recipe are you trying to run? The "Use your voice to control a light with Watson" doesn't use Watson Conversation. I guess you are referring to another TJBot recipe ("Build a talking robot with Watson Conversation"). If so, please follow the instructions on step 6 to look up your conversation workspace ID. https://www.instructables.com/id/Build-a-Talking-R... I just added a screenshot of the config file. Hopefully it makes it easier to locate the part.

Please let me know if this solves your problem! ::robot-face::

author
victordibia made it! (author)2016-11-22

Cool stuff!

DSC_0005.JPG
author
MarkK236 (author)2016-11-17

Help? I get this:

TJ is listening, you may speak now.

===== An Error has occurred =====

You may have exceeded your payload quota.

[object Object]

Press <ctrl>+C to exit.

author
victordibia (author)MarkK2362016-11-18

Hi MarkK236,

Can you confirm you have set up your bluemix speech to text credentials correctly in config.js?
Usually if your speech-to-text credentials are incorrect (or missing), you will get this error. Please see step 5 for how to add your Bluemix credentials to config.js

https://www.instructables.com/id/Use-Your-Voice-to-...

Please let me know if this helps!

V.

author
MarkK236 (author)victordibia2016-11-19

Got it! I had erroneously tried to cut and paste credentials from a different Watson service that I had created (from the Tone Analyzer demo). That's what I get for not following instructions. :D

Cool demo! Thank you for your help!

author
victordibia (author)MarkK2362016-11-19

Awesome!

author
TJBot made it! (author)2016-11-18

Thanks, Victor. Hi Mark, I just added a screenshot of the config file for you on step #5. https://www.instructables.com/id/Use-Your-Voice-to-Control-a-Light-With-Watson/step5/Update-your-Bluemix-credentials/ Let me know if this solves your problem! ::robot-smile::

Screen Shot 2016-11-18 at 10.53.43 AM.png
author
DIY Hacks and How Tos (author)2016-11-08

That is really cool. I always wanted to have a voice controlled robot.

author
TJBot made it! (author)DIY Hacks and How Tos2016-11-09

We cannot wait to see what you build with TJ!

FPLEL0JIUSM0K7M.jpeg

About This Instructable

16,691views

55favorites

Bio: I'm an open source project designed to help you access Watson Services in a fun way. You can laser cut or 3D print me ... More »
More by TJBot:How to Run a TJBot WorkshopHow to 3D Print a TJBotBuild TJBot Out of Cardboard
Add instructable to: