In a world where communication barriers persist for individuals unable to speak, sign language stands as a vital means of expression. However, the necessity of a third-party interpreter often impedes spontaneous and natural communication, posing challenges for seamless interaction. To address this prevalent issue, we've developed an innovative electronic device equipped with AI technology capable of detecting hand symbols and translating them into audible speech in real-time. Powered by the VIAM app and the Raspberry Pi Zero, our aim is to empower individuals who are deaf or hard of hearing to communicate more freely and independently, eliminating the constant reliance on interpretation assistance.

The electronic device we've created takes the form of wearable spectacles, providing a practical and discreet solution for communication barriers. Designed to be worn by individuals with disabilities, the spectacles seamlessly integrate into everyday life, allowing users to communicate effortlessly with those who do not know sign language. This wearable design not only enhances accessibility but also promotes a sense of independence and autonomy, empowering users to express themselves confidently in various social and professional settings.

In addition to promoting inclusivity and independence, our device represents a significant advancement in assistive technology. By leveraging AI-driven sign language detection and translation capabilities, we're not only breaking down communication barriers but also fostering greater understanding and empathy among individuals of diverse abilities. Ultimately, our goal is to create a more inclusive society where communication is seamless and accessible to all, regardless of linguistic or physical barriers.

Supplies

1 x Raspberry Pi Zero 2 W

1 x Raspberry Pi Camera Module V3

1 x USB sound card

1 x PAM8043 Amplifier Module

1 x Speaker: 3W, 4 ohms

Step 1: Hardware

In a world where communication barriers persist for individuals unable to speak, sign language stands as a vital means of expression. However, the necessity of a third-party interpreter often impedes spontaneous and natural communication, posing challenges for seamless interaction. To address this prevalent issue, we've developed an innovative electronic device equipped with AI technology capable of detecting hand symbols and translating them into audible speech in real-time. Powered by the VIAM app and the Raspberry Pi Zero, our aim is to empower individuals who are deaf or hard of hearing to communicate more freely and independently, eliminating the constant reliance on interpretation assistance.

The electronic device we've created takes the form of wearable spectacles, providing a practical and discreet solution for communication barriers. Designed to be worn by individuals with disabilities, the spectacles seamlessly integrate into everyday life, allowing users to communicate effortlessly with those who do not know sign language. This wearable design not only enhances accessibility but also promotes a sense of independence and autonomy, empowering users to express themselves confidently in various social and professional settings.

In addition to promoting inclusivity and independence, our device represents a significant advancement in assistive technology. By leveraging AI-driven sign language detection and translation capabilities, we're not only breaking down communication barriers but also fostering greater understanding and empathy among individuals of diverse abilities. Ultimately, our goal is to create a more inclusive society where communication is seamless and accessible to all, regardless of linguistic or physical barriers.

To install the OS on the Raspberry Pi just follow this guide.

These are the main hardware used. Then we can discuss the software.

Step 2: VIAM

VIAM is a software platform designed to streamline the development and management of smart machines. These machines are essentially robots or devices equipped with sensors and computing power, allowing them to interact with their environment. VIAM simplifies tasks for developers by providing a consistent interface for controlling these machines, collecting data, and even incorporating machine learning capabilities.

VIAM's open-source and modular design gives developers flexibility. They can choose the functionalities they need and integrate VIAM with various hardware components. Additionally, VIAM offers cloud-based solutions for managing and scaling large numbers of smart machines.

These are the main terminologies used in the VIAM app.

smart machine in Viam consists of at least one computer, typically a single-board computer, running viam-server and communicating with any hardware connected to it by signalling through a digital data pin.
The term component describes a piece of hardware that a computer controls, like an arm or a motor.
The term Services are built-in software packages that make it easier to add complex capabilities such as motion planning or object detection to your machine.

Step 3: Creating the Smart Machine With VIAM

So we can start creating our smart machine. The process is pretty simple. First, we logged onto the VIAM app and created a new machine. Then we installed the VIAM server by running these commands which are given in the image above. Just follow this guide if you have any doubts.

Once the Viam server successfully starts running on the Pi Zero 2W, you can see the device's online status on your dashboard.

Step 4: Adding Components to Machine

So we first added the camera component. csi-pi module is used here to get that job done. This module easily integrates CSI Cameras into your Viam projects using this camera module.

Now's the moment to integrate computer vision into the system by setting up the vision service for the machine.

While Viam's default mlmodel vision service is compatible with Tensor Flow Lite models, this project employs a YOLOv8 model. To achieve this, we'll utilize a module from the modular resource registry designed to enhance Viam with YOLOv8 integration.

With the YOLOv8 module, we have gain the capability to leverage any YOLOv8 model seamlessly to the Viam machines.

We have used a yolov8 model from the hugging face that is capable of reading the 26 letters of American sign language alphabets.

Next, we need to add a text-to-speech service to speak these letters.

For that, we used the speech service from VIAM. It is a modular service that provides text-to-speech (TTS) and speech-to-text (STT) capabilities for machines running on the Viam platform.

We have used google as the speech provider.

So we added all the required services and components to the Smart machine.

Step 5: Python SDK

To run this project we are using their Python SDK and they also provide a 70% completed code snippet to get started with. To install the Python SDK just go to their connect tab.

You can find the entire code for this project here.

Step 6: Some Additional Hardware

The Raspberry pi zero 2W has no audio output jack, so either we need to use a shield or a sound card with the micro USB adapter.

So we decided to go with the USB sound card. The main problem is the size of the sound card. In this project the space is premium so we decided to remove the case and the audio jack and decided to solder directly to the sound card.

We used a 16-ohm 0.25-watt speaker to play the output audio. The audio performance is not that much good. This is just a prototype so that's enough for it.

The audio is very feeble so we used a PAM8043 amplifier module.

Step 7: Case

The case for this project is designed on the Fusion 360 and 3D printed with the PLA filament.

Step 8: Setting Up the Front Rim

So let's start assembly. First, we attached the camera module to the front rim using the glue.

Step 9: Setting Up the Raspberry Pi

In this step, we soldered the connections between the sound card and the Raspberry Pi. Then we attached the PAM8043 amplifier module to the front side of the Raspberry Pi by making the necessary connections. You can find the schematics in the above image.

Step 10: Remaining Assembly

Finally, we inserted the Pi zero into the Temple portion and connected the camera cable. Then we attached the left temple portion to the RIM part. Finally attached the speaker to the right RIM part and also glued it to the RIM portion.

Step 11: Demo Video

Here is the demo video of the Sign Language Translator. American Sign Language letters are given for the reference.

Step 12: Future Updates

The motivation behind the project stems from the desire to foster greater inclusivity and accessibility in communication. By providing real-time translation of sign language into a universally understandable format, the device aims to bridge the communication gap between individuals who use sign language and those who do not. Future iterations of the project aim to expand the device's capabilities to interpret more complex elements of sign language, such as hand movements and orientations, to provide a more nuanced translation.