Introduction: A Talking Color Sensor, Based on the AIY Voice Kit
Having learned a bit about Braille recently, I was wondering if I could build something using the AIY voice kit for the Raspberry Pi, that may have a real-live benefit for the visually impaired. So described in the following you will find an prototype of a simple color detection device that reads its findings aloud.
A more elaborate version of this system might be useful for persons with impaired sight or color blindness.
The system is using an Raspberry Pi with an AIY voice HAT attached. A TCS34725 RGB sensor breakout is connected to the I2C port of the HAT. The breakout contains a bright warm white LED to illuminate the object to be analyzed. The breakout was placed in a housing to optimize and standardize measurement conditions.
The three color sensor measures about the same three frequency ranges as the color sensors in your eyes. Then the red, green and blue (RGB) values are used to calculate the overall color impression.
The nice thing about this special system is that it now tells you the color verbally, using the AIY voice kits' "say" command. Please have a look on the accompanying video.
The device might also be useful as an example for an I2C sensor device connected to the AIY voice HAT.
Step 1: Materials Used
Raspberry Pi 3. ~ 35 US$ or EUR
AIY voice kit, with headers soldered to the HAT. ~ 25US$ or EUR
Adafruit TCS34725 breakout, with a header soldered. ~ 8 US$ or EUR
A breadboard (optional)
For the sensor housing:
- a used "Dolce Gusto" coffee capsule
- a small round piece of 2mm Forex (PVC foam plate), about 37mm diameter
- a non-reflecting black material to cover the inner walls of the housing. I used self-adhesive black rubber foam.
Optional: a small switch to evoke the measurements
A few drops of plastic glue and a cutter knife.
Step 2: Assembly and Usage
The Raspberry Pi with the AIY voice HAT was setup as described in the AIY manual. Before assembly, headers were soldered to the ports on the HAT. For the housing of the sensor, a "Dulce Gusto" coffee capsule was emptied, cleaned, and a part of the bottom carefully removed with a knife. You may use something else for this purpose, the coffee capsule just had the right size and shape. A round piece of 2mm Forex was cut from a plate, the breakout was then placed centrally on the Forex plate, the position marked with a felt pen, and a slot for the header on the breakout was cut at the appropriate position.
Now the Forex piece was glued onto the housing and the sensor breakout attached to the Forex plate, using a Velcro strip. Then the inner walls were covered with a light absorbing black material, I used a self-adhesive rubber foam. Black cardboard should work as well. Now, using jumper cables, the I2C "3.3V" port of the HAT was connected to "V in" on the sensor, Ground to Gnd, sda to sda and scl to scl. I had used a breadboard to connect both parts, but that's not neccessary.
Place the AIY_TCS34725 python script in the src folder and run the script from the dev terminal, entering "sec/AIY_TCS34752.py". You may have to make the python script executable first. When asked, place the sensor unit over the object to be measured, press the button in the AIY device and wait a second or two.
Then, based upon the measured RGB and white values, the device first calculates the corresponding hue value, then estimates the color based on this value and communicates them verbally via the AIY voice system, e. g. as "dark red", but also gives the hue value. RGB, hue and brightness (lightness, to be exact) values are also printed to screen.
To simplify the color annotation process, the RGB values are transformed into HSV (hue, saturation, value) format. This allows to annotate a color to a certain range of angles (i.e. a pie slice), and pick the color based on the calculated hue value.
You need to normalize your device against a white and a black reference. Just measure the whitest and blackest pieces of paper you have available, take a measurement each, and place these values as maximum and minimum values into the code. Only optimal reference values will give a good color recognition.
One basic problem is reflection. If you have an object with a glossy or polished surface it will reflect lot of the light emitted by the LED, appearing much lighter than it really is. You may use a sheet of membrane to scatter the light, but you may need to implement a correction factor.
In the case of translucent objects, it might be handy to place them on a white paper, otherwise the amount of reflected light will be to small and the object reported as "black".
If you want to measure the color of objects that emit light, you should switch off the LED on the breakout by connecting the "LED" port on the breakout to "Ground". Now set the normalization values accordingly.
Another general problem is the illumination of the object. The warm white LED on the breakout emits a non-continuous spectrum of light. Therefore certain colors might be over- or underrepresented in the RGB spectrum. For more information on this topic, please have a look on my previous instructables on a colorimeters/ photometers and spectrometers:
Step 3: The Code
The code is a combination of a modification of a code from the AIY voice manual, and the TCS34725 sensor code by Bradspi.
I had also tried to use the TCS34725 python code from Adafruit, but had problems run this and some other codes that are using external libraries in combination with the AIY HAT. Any help welcome.
As mentioned before, the color annotation is based on a transformation on the RGB to hue values. You must set normalization settings based on experimental measurements of white and black reverence materials. Fill in the absolute values for R, G and B min or max accordingly.
The script uses a new version of the "say" command that allows to regulate volume and pitch. In case, you may have to either update the audio.py and tty driver files or delete the "volume and pitch parts" from the script.
# This script is an adaption of the servo_demo.py script for the AIY voice HAT, # optimized for the color recognition uing the Afafruit TCS34725 breakout
import aiy.audio import aiy.cloudspeech import aiy.voicehat #from gpiozero import LED # could be helpful for an external LED on servo-port #from gpiozero import Button # could be helpful for an external button on servo-port import time import smbus bus = smbus.SMBus(1) import colorsys
def hue2color(hue): # color interpretation based on the calculated hue values if ((hue> 12) and (hue< 26)): # i.e. between 12° and 40°. All settings may require optimization color="orange" return color elif ((hue> 25) and (hue< 70)): color="yellow" return color elif ((hue> 69) and (hue< 165)): color="green" return color elif ((hue> 164) and (hue< 195)): # 180 +/- 15 color="cyan" return color elif ((hue> 194) and (hue< 270)): color="blue" return color elif ((hue> 269) and (hue< 320)): color="magenta" return color elif ((hue> 319) or (hue< 20)): color="red" return color else: print ("something went wrong")
def tcs34725(): # measurement and interpretation. # The measurement is performed by the Bradspi TCS34725 script: # https://bradsrpi.blogspot.com/2013/05/tcs34725-rg...
bus.write_byte(0x29,0x80|0x12) ver = bus.read_byte(0x29) # version # should be 0x44 if ver == 0x44: print ("Device found\n") bus.write_byte(0x29, 0x80|0x00) # 0x00 = ENABLE register bus.write_byte(0x29, 0x01|0x02) # 0x01 = Power on, 0x02 RGB sensors enabled bus.write_byte(0x29, 0x80|0x14) # Reading results start register 14, LSB then MSB data = bus.read_i2c_block_data(0x29, 0) clear = clear = data << 8 | data red = data << 8 | data green = data << 8 | data blue = data << 8 | data crgb = "Absolute counts: C: %s, R: %s, G: %s, B: %s\n" % (clear, red, green, blue) print (crgb) time.sleep(1) else: print ("Device not found\n")
# normalization and transformation of the measured RGBW values col=""
# Maximum values Normalization factors, must be defined experimentally # e.g. vs. a white sheet of paper. Check and correct from time to time. max_bright = 5750 max_red = 1930 max_green = 2095 max_blue = 1980
# Background/Minimum values normalization factors, must be defined experimentally # e.g. vs. black sheet of paper. Check and correct from time to time. min_bright = 750 min_red = 340 min_green = 245 min_blue = 225
# normalized values, between 0 and 1 rel_bright = ((clear - min_bright)/(max_bright - min_bright)) rel_red = ((red - min_red)/(max_red - min_red)) rel_green = ((green - min_green)/(max_green - min_green)) rel_blue = ((blue - min_blue)/(max_blue - min_blue)) hsv_col = colorsys.rgb_to_hsv(rel_red, rel_green, rel_blue) hue = hsv_col*359 if rel_bright > 0.9: col = "white" # if very bright -> white elif rel_bright < 0.1: col = "black" # if very dark -> black else: col = hue2color(hue) # color selection by hue values
# print("relative values bright, red, green, blue:") # print (rel_bright, rel_red, rel_green, rel_blue) # print("HSV values (hue, saturation, value):", hsv_col) # print ("hue in ° ",hue)
return [col, rel_bright, rel_red, rel_green, rel_blue, hue]
def main(): button = aiy.voicehat.get_button() # change Button status led = aiy.voicehat.get_led() # change Button-LED status aiy.audio.get_recorder().start() # buttoni= Button(5) # distance sensor or other external button, connected to servo3/GPIO 05
aiy.audio.say("Hello!", lang="en-GB", volume=50, pitch=100) # volume and pitch require November 2017 revision of audio.py and _tty.py driver! aiy.audio.say("To start, move the sensor above the object. Then press the blue button", lang="en-GB", volume=50, pitch = 100) print("To activate color measurement place sensor above object, then press the blue button")
while True: led.set_state(aiy.voicehat.LED.ON) button.wait_for_press() # for external button, replace button by buttoni led.set_state(aiy.voicehat.LED.BLINK) aiy.audio.say("Measuring", lang="en-GB", volume=50, pitch = 100) result = tcs34725() # evokes measurement and interpretation col = result # color, as text hue = str(int(result)) # hue in °, as text r_red = str(int(result*255)) # R value, as text r_green = str(int(result*255)) # G value, as text r_blue = str(int(result*255)) # B value, as text r_bright = str(int(result*100)) # W value, as text led.set_state(aiy.voicehat.LED.OFF)
if col == "white" or col=="black": bright = "" elif (result >0.69): #brightness/lightness of color bright ="light" elif (result <0.25): bright ="dark" else : bright ="medium" # communiating the results color_text =("The color of the object is " + bright + " " + col) print (color_text) aiy.audio.say(color_text, lang="en-GB", volume=75, pitch=100) hue_text = ("The hue value is "+ hue+ " degrees") print (hue_text) aiy.audio.say(hue_text, lang="en-GB", volume=75, pitch = 100)
if __name__ == '__main__': main()
Step 4: Some Links and Remarks
The TCS34725 sensor data sheet can be found here: https://cdn-shop.adafruit.com/datasheets/TCS34725.pdf
The code to read the sensor I have used was described here: http://bradsrpi.blogspot.de/2013/05/tcs34725-rgb-c...
You may find some additional information on color measurements with this and another sensor in my previous instructables: