A Talking Color Sensor, Based on the AIY Voice Kit

Having learned a bit about Braille recently, I was wondering if I could build something using the AIY voice kit for the Raspberry Pi, that may have a real-live benefit for the visually impaired. So described in the following you will find an prototype of a simple color detection device that reads its findings aloud.

A more elaborate version of this system might be useful for persons with impaired sight or color blindness.

The system is using an Raspberry Pi with an AIY voice HAT attached. A TCS34725 RGB sensor breakout is connected to the I2C port of the HAT. The breakout contains a bright warm white LED to illuminate the object to be analyzed. The breakout was placed in a housing to optimize and standardize measurement conditions.

The three color sensor measures about the same three frequency ranges as the color sensors in your eyes. Then the red, green and blue (RGB) values are used to calculate the overall color impression.

The nice thing about this special system is that it now tells you the color verbally, using the AIY voice kits' "say" command. Please have a look on the accompanying video.

The device might also be useful as an example for an I2C sensor device connected to the AIY voice HAT.

Step 1: Materials Used

Raspberry Pi 3. ~ 35 US$ or EUR

AIY voice kit, with headers soldered to the HAT. ~ 25US$ or EUR

Adafruit TCS34725 breakout, with a header soldered. ~ 8 US$ or EUR

Jumper cables.

A breadboard (optional)

For the sensor housing:
- a used "Dolce Gusto" coffee capsule
- a small round piece of 2mm Forex (PVC foam plate), about 37mm diameter
- a non-reflecting black material to cover the inner walls of the housing. I used self-adhesive black rubber foam.

Optional: a small switch to evoke the measurements

A few drops of plastic glue and a cutter knife.

Step 2: Assembly and Usage

The Raspberry Pi with the AIY voice HAT was setup as described in the AIY manual. Before assembly, headers were soldered to the ports on the HAT. For the housing of the sensor, a "Dulce Gusto" coffee capsule was emptied, cleaned, and a part of the bottom carefully removed with a knife. You may use something else for this purpose, the coffee capsule just had the right size and shape. A round piece of 2mm Forex was cut from a plate, the breakout was then placed centrally on the Forex plate, the position marked with a felt pen, and a slot for the header on the breakout was cut at the appropriate position.

Now the Forex piece was glued onto the housing and the sensor breakout attached to the Forex plate, using a Velcro strip. Then the inner walls were covered with a light absorbing black material, I used a self-adhesive rubber foam. Black cardboard should work as well. Now, using jumper cables, the I2C "3.3V" port of the HAT was connected to "V in" on the sensor, Ground to Gnd, sda to sda and scl to scl. I had used a breadboard to connect both parts, but that's not neccessary.

Place the AIY_TCS34725 python script in the src folder and run the script from the dev terminal, entering "sec/AIY_TCS34752.py". You may have to make the python script executable first. When asked, place the sensor unit over the object to be measured, press the button in the AIY device and wait a second or two.

Then, based upon the measured RGB and white values, the device first calculates the corresponding hue value, then estimates the color based on this value and communicates them verbally via the AIY voice system, e. g. as "dark red", but also gives the hue value. RGB, hue and brightness (lightness, to be exact) values are also printed to screen.

To simplify the color annotation process, the RGB values are transformed into HSV (hue, saturation, value) format. This allows to annotate a color to a certain range of angles (i.e. a pie slice), and pick the color based on the calculated hue value.

You need to normalize your device against a white and a black reference. Just measure the whitest and blackest pieces of paper you have available, take a measurement each, and place these values as maximum and minimum values into the code. Only optimal reference values will give a good color recognition.

One basic problem is reflection. If you have an object with a glossy or polished surface it will reflect lot of the light emitted by the LED, appearing much lighter than it really is. You may use a sheet of membrane to scatter the light, but you may need to implement a correction factor.

In the case of translucent objects, it might be handy to place them on a white paper, otherwise the amount of reflected light will be to small and the object reported as "black".

If you want to measure the color of objects that emit light, you should switch off the LED on the breakout by connecting the "LED" port on the breakout to "Ground". Now set the normalization values accordingly.

Another general problem is the illumination of the object. The warm white LED on the breakout emits a non-continuous spectrum of light. Therefore certain colors might be over- or underrepresented in the RGB spectrum. For more information on this topic, please have a look on my previous instructables on a colorimeters/ photometers and spectrometers:

https://www.instructables.com/id/An-Inexpensive-Ph...

https://www.instructables.com/id/A-Mimimal-Six-Col...

Step 3: The Code

The code is a combination of a modification of a code from the AIY voice manual, and the TCS34725 sensor code by Bradspi.

I had also tried to use the TCS34725 python code from Adafruit, but had problems run this and some other codes that are using external libraries in combination with the AIY HAT. Any help welcome.

As mentioned before, the color annotation is based on a transformation on the RGB to hue values. You must set normalization settings based on experimental measurements of white and black reverence materials. Fill in the absolute values for R, G and B min or max accordingly.

The script uses a new version of the "say" command that allows to regulate volume and pitch. In case, you may have to either update the audio.py and tty driver files or delete the "volume and pitch parts" from the script.

#!/usr/bin/env python3
# This script is an adaption of the servo_demo.py script for the AIY voice HAT,
# optimized for the color recognition uing the Afafruit TCS34725 breakout 

import aiy.audio
import aiy.cloudspeech
import aiy.voicehat
#from gpiozero import LED     # could be helpful for an external LED on servo-port
#from gpiozero import Button  # could be helpful for an external button on servo-port
import time
import smbus
bus = smbus.SMBus(1)
import colorsys

def hue2color(hue):   # color interpretation based on the calculated hue values
 if ((hue> 12) and (hue< 26)): # i.e. between 12° and 40°. All settings may require optimization
     color="orange"
     return color
    
 elif ((hue> 25) and (hue< 70)):
     color="yellow"
     return color
    
 elif ((hue> 69) and (hue< 165)):
     color="green"
     return color
     
 elif ((hue> 164) and (hue< 195)): # 180 +/- 15
     color="cyan" 
     return color
    
 elif ((hue> 194) and (hue< 270)):
     color="blue" 
     return color
    
 elif ((hue> 269) and (hue< 320)):
     color="magenta" 
     return color
    
 elif ((hue> 319) or (hue< 20)):
     color="red" 
     return color
    
 else: print ("something went wrong")
def tcs34725(): # measurement and interpretation. 
# The measurement is performed by the Bradspi TCS34725 script: 
#  https://bradsrpi.blogspot.com/2013/05/tcs34725-rg...

 bus.write_byte(0x29,0x80|0x12)
 ver = bus.read_byte(0x29)
 # version # should be 0x44
 if ver == 0x44:
  print ("Device found\n")
  bus.write_byte(0x29, 0x80|0x00) # 0x00 = ENABLE register
  bus.write_byte(0x29, 0x01|0x02) # 0x01 = Power on, 0x02 RGB sensors enabled
  bus.write_byte(0x29, 0x80|0x14) # Reading results start register 14, LSB then MSB
  data = bus.read_i2c_block_data(0x29, 0)
  clear = clear = data[1] << 8 | data[0]
  red = data[3] << 8 | data[2]
  green = data[5] << 8 | data[4]
  blue = data[7] << 8 | data[6]
  crgb = "Absolute counts: C: %s, R: %s, G: %s, B: %s\n" % (clear, red, green, blue)
  print (crgb)
  time.sleep(1)
 else: 
  print ("Device not found\n")

# normalization and transformation of the measured RGBW values
 col=""

# Maximum values Normalization factors, must be defined experimentally
# e.g. vs. a white sheet of paper. Check and correct from time to time.
 max_bright = 5750 
 max_red = 1930
 max_green = 2095
 max_blue = 1980

# Background/Minimum values normalization factors, must be defined experimentally 
# e.g. vs. black sheet of paper. Check and correct from time to time.
 min_bright = 750
 min_red = 340
 min_green = 245
 min_blue = 225

# normalized values, between 0 and 1
 rel_bright = ((clear - min_bright)/(max_bright - min_bright)) 
 rel_red = ((red - min_red)/(max_red - min_red))
 rel_green = ((green - min_green)/(max_green - min_green))
 rel_blue = ((blue - min_blue)/(max_blue - min_blue))
 hsv_col = colorsys.rgb_to_hsv(rel_red, rel_green, rel_blue)
 hue = hsv_col[0]*359
 if rel_bright > 0.9: col = "white"    # if very bright -> white
 elif rel_bright < 0.1: col = "black"  # if very dark -> black
 else: col = hue2color(hue)            # color selection by hue values
 
 # print("relative values bright, red, green, blue:") 
 # print (rel_bright, rel_red, rel_green, rel_blue) 
 # print("HSV values  (hue, saturation, value):", hsv_col)
 # print ("hue in ° ",hue)

 return [col, rel_bright, rel_red, rel_green, rel_blue, hue]
def main():
 button = aiy.voicehat.get_button()   # change Button status
 led = aiy.voicehat.get_led()         # change Button-LED status
 aiy.audio.get_recorder().start()
 
 # buttoni= Button(5)     # distance sensor or other external button, connected to servo3/GPIO 05
 aiy.audio.say("Hello!", lang="en-GB", volume=50, pitch=100)  # volume and pitch require November 2017 revision of audio.py and _tty.py driver!
 aiy.audio.say("To start, move the sensor above the object. Then press the blue button", lang="en-GB", volume=50, pitch = 100)
 print("To activate color measurement  place sensor above object, then press the blue button")
 
 while True:
    led.set_state(aiy.voicehat.LED.ON)
    button.wait_for_press() # for external button, replace button by buttoni
    led.set_state(aiy.voicehat.LED.BLINK)
    aiy.audio.say("Measuring", lang="en-GB", volume=50, pitch = 100) 
    result = tcs34725()  # evokes measurement and interpretation 
    col = result[0]                     # color, as text
    hue = str(int(result[5]))           # hue in °, as text
    r_red = str(int(result[2]*255))     # R value, as text
    r_green = str(int(result[3]*255))   # G value, as text
    r_blue = str(int(result[4]*255))    # B value, as text
    r_bright = str(int(result[1]*100))  # W value, as text
    led.set_state(aiy.voicehat.LED.OFF)

    if col == "white" or col=="black":
        bright = ""
    elif (result[1] >0.69): #brightness/lightness of color
        bright ="light"     
    elif (result[1] <0.25):
        bright ="dark"
    else :
        bright ="medium"
    # communiating the results    
    color_text =("The color of the object is " + bright + " " + col)
    print (color_text)
    aiy.audio.say(color_text, lang="en-GB", volume=75, pitch=100)
   
    hue_text = ("The hue value is "+ hue+ " degrees")
    print (hue_text)
    aiy.audio.say(hue_text, lang="en-GB", volume=75, pitch = 100)
             
if __name__ == '__main__':
 main()