A Voice-Controlled, Braille-Type Pattern Semaphore

"Lord Vetinari stood at his window watching the semaphore tower on the other side of the river. All eight of the big shutters facing him were blinking furiously - black, white, black, white, black, white…
Information was flying into the air. Twenty miles behind him, on another tower on Sto Lat, someone was looking through a telescope and shouting out numbers.
How quickly the future comes upon us, he thought."
T. Pratchett, The Fifth Elephant

After building a voice-controlled jumping jack based on the AIY voice kit, I had the idea to build a voice-controlled semaphore, bringing together the latest in IT technology with the beginnings of telecommunications and data transfer.

At first I had the idea to replicate the French semaphore system by Chappe, which was the first known system for nation-wide telecommunications using a semaphore system. But it turned out to be a bit too complicated to be realized using standard servos within a day. My next target was something similar to the clacks system described by Terry Pratchett, e.g. in "Going Postal", as a 2x4 shutter semaphore system (not as the 4x4 matrix depicted in the movie). Unfortunately I could not find too much technical details available on this system. So I ended up with the 2x3 matrix semaphore system developed by Lord Murray, that had been used by the British Navy for a while. In addition, a six shutter/bit system does fit well to the six servo connectors available on the AIY voice HAT. But, as I did not have six servos at hand between the years, I finally decided to build a simulator made of LEDs first.

Concerning the code displayed, one could use the Murray system, but again the information I had available on it was rather limited, not allowing to display any numbers and symbols. So I came to the point to use the Braille system instead, which also uses a 2x3 matrix to display letters, numbers and other signs. The Braille system is the international standard to print texts readable for the blind. It is also a markup language, which uses a number indicator to define that numbers will be displayed next, and indicators to define that one, or many, of the following letters are written as capitals. I therefore decided to set up a slightly simplified system, with numbers and some signs defined by the Nemeth extension of the Braille system instead, and just use upper case letters, at least for the beginning. This allows to have unique patterns for every letter, number or sign to be displayed in my special application, and to omit the text analysis required for real Braille.

The final device allows to speak a word or sentence into the AIY voice recognition system, then the voice pattern data is sent via WLAN and internet to some Google sever in the US, gets decoded there, and, at least in my case, the interpreted data is sent back to Europe, where finally I get the recognized sentence displayed as a text string. This text string is then broken by the Python script into the individual letters, and now, by comparison with a dictionary defining the corresponding patterns, the pattern information is retreived and the patterns are displayed on a 2x3 LED matrix. Please have a look on the accompanying video.

I have set the display rate to one character per second, which should be long enough for a trained person to identify and translate the pattern. A possible next step would be to use a pattern recognition device like the AIY vision HAT (so far not available in Europe) to read and interpret the patterns automatically, so to close the circle.

Further concepts for improvements, some with more real-world relevance, are discussed in the 'outlook' part of this instructable.

Step 1: Materials Used

Raspberry Pi 3

AIY voice HAT

Eight white LEDs, 5 mm diameter. These run on 3V, thus a resistor is needed.

100kOhm Resistor. Maybe not the perfect solution, but was at hand.

Jumper cables

A short piece of wire

Breadboard, optional to test the setup.

A plastic box for business cards.

Two pieces of 4 mm plastic foam, some left over rubbish.

Some piece of plastic membrane, as diffuser, as above.

Soldering iron and solder, a knife.

Step 2: Setup and Usage

Setup the Raspberry Pi and AIY HAT as indicated in the AIY voice HAT manual. I would recommend to solder headers at least to the servo ports before you assemble Pi and HAT, as this allows you easily connect servos, a breadboard or LEDs.

The display box had been build from the lid of a plastic box for business cards, two pieces of foam fitting into the box and a similar sized piece of a packaging membrane as diffuser. In one of the foam parts six holes were pushed and the LEDs placed into them. The shorter feet (ground side) of the LEDs were connected with each others by a piece of cable, then a resistor was added and a jumper cable was soldered to the latter. To the other feet (plus side) of the LEDs jumper cables were soldered.

These then were connected to the servo ports on the AIY voice HAT via elongation cables, the positive side to the (outer) "P in" pins, the negative connector to one of the (inner) ground/minus pins. Please have a look on the attached scheme.

I would strongly recommend to test the setup on a breadboard before soldering.

Now the membrane, the LED plate and the sealing layer were placed in the plastic box.

Place the Braille_LED_1.py script in the src folder. In case, you may have to make the script executable first.

Now using the Dev terminal (!) the Braille_LED_1.py program is started. Enter 'src/Braille_LED_1.py' and press 'Enter'.

You now will be asked to press the button of the AIY box and say your word or sentence. With some delay, the system will repeat what had been understood, and display it on screen as well as, letter by letter, on the six LED display.

If you give the keyword "Goodbye" instead of a sentence, the system will tell you Goodbye, and the program will be closed.

Step 3: The Code

Below you find the code that allows you to speak to the AIY voice device and have the recognized sentence displayed letter by letter on a small 2x3-LED 'semaphore' or Braille-type matrix.

The code is a derivative of the script I used for a previous project using the AIY voice HAT, being a derivative of the servo_demo.py example described in the AIY voice HAT manual.

You can also find a dictionary with the part of the Murray code I had found in the internet, as text file. It neither contains numbers and omits some letters, which would give a problem here.

One limitation of the program in its current state is that if a sign not included in the dictionary it would crash the program. In addition it is not a representation of the complete Braille Code as a Markup language. As you can see in the script below, the Nemeth code for numbers is redundant with some signs in standard Braille, but that should not give a problem for our specific application.

#!/usr/bin/env python3

# This script is an adaption of the servo_demo.py script for the AIY voice HAT,
# optimized for the AIY based Baille type symbol display

import aiy.audio
import aiy.cloudspeech
import aiy.voicehat
from gpiozero import LED
#from gpiozero import Button
from time import sleep


# Dictionary: an artificially modified Braille alphabet,
# numbers and some symbols taken from the Nemeth extension of Braille

Braille_6A = {
" " : "123456",  # space
"A" : "1",
"B" : "12",
"C" : "14",
"D" : "145",
"E" : "15",
"F" : "124",
"G" : "1245",
"H" : "125",
"I" : "24",
"J" : "245",
"K" : "13",
"L" : "123",
"M" : "134",
"N" : "1345",
"O" : "135",
"P" : "1234",
"Q" : "12345",
"R" : "1235",
"S" : "234",
"T" : "2345",
"U" : "136",
"V" : "1236",
"X" : "1346",
"Y" : "13456",
"Z" : "1356",
"W" : "2456",
"#" : "3456", # Number Prefix, i.e. next signs are numbers
"," : "2",
"." : "256",  # Full stop, end of sentence (GB)
"?" : "236",
"!" : "235",
"'" : "3",
"-" : "24",
";" : "23",
"Cap" : "6",  # Next letter is in Capitals; Number-Stop ?
"" : "",
# Nemeth Braille code is a mathematical expansion for 6-Point Braille
# see: <a href="https://en.wikipedia.org/wiki/Nemeth_Braille">  <a href="https://en.wikipedia.org/wiki/Nemeth_Braille"> https://en.wikipedia.org/wiki/Nemeth_Braille </a> </a>


"1" : "2",    # Nemeth Code '1', Braille 'comma'  
"2" : "23",
"3" : "25",
"4" : "256", 
"5" : "26",
"6" : "235",   # Nemeth '6', Braille '!'
"7" : "2356",
"8" : "236",  # Nemeth '8', Braille '?'
"9" : "35",
"0" : "356",
"+" : "346",
"-" : "36",
"/" : "34",
"(" : "12356",
")" : "23456",
"*" : "1346" # '*' is a two pattern symbol in Nemeth, here replaced by an 'x' to omit crashes

}


""" for reasons of simplicity, the standard Braille number patterns given below 
    were replaced by the corresponing Nemeth-Codes
"1" : "1",  
"2" : "12",
"3" : "14",
"4" : "145",
"5" : "15",
"6" : "124",
"8" : "1245",
"9" : "24",
"0" : "245",
"""
#Text = "rbhTZkl 9t64+34#!"  # Sampletext, for debugging purposes
def main():
    
 recognizer = aiy.cloudspeech.get_recognizer()
 recognizer.expect_phrase('goodbye') # keyword, ends the program
 button = aiy.voicehat.get_button()   # AIY Button status
 led = aiy.voicehat.get_led()         # AIY Button-LED status
 aiy.audio.get_recorder().start()
 
 led_1 = LED(26) # 1st connector, servo0, GPIO 26 # upper left
 led_2 = LED(6)  # 2nd connector, servo1, GPIO 06 # middle left
 led_3 = LED(13) # 3rd connector, servo2, GPIO 13 # lower left
 led_4 = LED(5)  # 4st connector, servo3, GPIO 05 # upper right
 led_5 = LED(12) # 5th connector, servo4, GPIO 12 # middle right  
 led_6 = LED(24) # 4st connector, servo3, GPIO 13 # lower right  
 #distance= Button(5)    # distance sensor connected to servo3/GPIO 05, not used here
 aiy.audio.say("Hello!", lang="en-GB")   
 aiy.audio.say("To start, please push the button", lang="en-GB")
 aiy.audio.say("If you tell me Goodbye, I will end the program", lang="en-GB")
 
 while True:             # starts loop
    led.set_state(aiy.voicehat.LED.BLINK)   
    print("To activate voice recognition, press the blue button, then speak")
    print()
    button.wait_for_press()
    print('Listening...')
    aiy.audio.say("I am listening", lang="en-GB")    
    led.set_state(aiy.voicehat.LED.BLINK_3)
    text = recognizer.recognize()               # text string of the reognized sentence
    led.set_state(aiy.voicehat.LED.OFF)
    if text is None:
       aiy.audio.say('Sorry, I did not hear you.', lang="en-GB")
    elif 'goodbye' in text:
             aiy.audio.say("Goodbye", lang="en-GB")              
             aiy.audio.say('Arrivederci', lang="it-IT")
             aiy.audio.say('Auf Wiedersehen', lang="de-DE") 
             sleep (3)
             print('bye!')
             break       # stops loop and ends program
    else:
        print('You said "', text, '"')      # Lets you check the systems interpretation, including captialization
        aiy.audio.say('I guess you said', lang="en-GB")
        aiy.audio.say(text, lang="en-GB")   # acustic verification
        Text_up = text.upper()              # transfers all to upper case  
        print (Text_up)
        Text_Len = len(Text_up)             
        print (Text_Len)
        for i in range (Text_Len):
           Lett = Text_up [i]         # Picks a single letter, starting with first, i.e. [0]
           print ("Letter=", Lett)
           Lett_B = Braille_6A[Lett]  # Picks the corresponding code from the dictionary. A missing sign will break the code!
           print (Lett_B)
           if ("1" in Lett_B):
             print ("LED 1")      
             led_1.on()      # activates LED at servo0"
           if "2" in Lett_B:
             print ("LED 2")
             led_2.on()
           if "3" in Lett_B:
             print ("LED 3")
             led_3.on()
           if "4" in Lett_B:
             print ("LED 4")
             led_4.on()
           if "5" in Lett_B:
             print ("LED 5")
             led_5.on()
           if "6" in Lett_B:
             print ("LED 6")
             led_6.on() 
           sleep (1) # display pattern for a second
           print()
           led_1.off() # inactivate all LEDs on servos0-5n
           led_2.off()
           led_3.off()
           led_4.off()
           led_5.off()
           led_6.off()
           sleep(0.3)  # a short dark break, to indicate the end of the letter 
             
if __name__ == '__main__':
 main()</p>

Step 4: Outlook and Remarks

So what could be next?

Beside that IT meta-joke with the AIY video kit or another automated image recognition system mentioned in the introduction, there might be other options to expand the concept described in this instructable. Some of them even might be of real-world use. These could be:

- improved programming, so that the text is transferred into standard Braille code, with all the markups and compressions. That should not be very much efforts for a skilled python programmer.Which I am not, so any help would be welcome.

- expand the system to a 2x4 matrix. Should be possible as well, and would help to use the corresponding 8-dot Braille codes, as are used by electronic Braille displays. Above this, it would be a bit closer to the Dearheart clacks system.

- construct a real 2x3 or 2x4 Braille display. Should be possible using either an array of servos or an array of 5V mini solenoids. The primary challenge would be that the standard distance between tactile points is 2.45 mm, or 1/10 inch, on electronic displays, so some gears and mechanics might be required. The other challenge could be to precisely regulate the push length.

Such a simple and cheap solution might be of interest for a wider public, as commercial Braille displays are quite expensive. There might be a benefit for Braille learners using such a voice-controlled system. As they might verbally enter an (English) sentence of their choice, and get the text, letters and signs, displayed on their finger tip(s).

- construct a mechanical shutter system resembling the Murray or the Dearheart systems. Using servos, that should not be too complicated and might be described in another instructable. Or could be a nice school project. Anyone interested?

–――――

Some remarks and links of interest:

- There is an instructable describing a "DIY semaphore lantern", a 2x4 LED matrix, based on some Propeller demo board. I like the layout, but the programming looks a bit complicated to me. Please have look yourself.

- I now found a python program to generate Fully Contracted (Grade 2) Braille. Unfortunately its limited to python 2 and the 2002 American English version: https://github.com/jpaugh/braille-converter .

- a more complete program seems to be liblouis, https://github.com/liblouis/liblouis , but I have no idea how to integrate this into this solution.

- an interesting python solution seems come from Greece, https://github.com/ant0nisk/pybrl can integrate several languages and generate grade 2 Braille .

- I am neither a programmer, nor an electronics guy, nor had I much knowledge on Braille a few days ago.

So if you see any errors, omissions, or you have some ideas for the project, please let me know.

- If you like this instructable, please vote for it!