BotGPT: a Physical ChatGPT Robot Assistant

TLDR: Let’s build an animatronic robot that runs on ChatGPT!

**Note: I have linked a demo video of the robot but this is unfinished, I will post a new final video soon**

Goal

With ChatGPT, I can have an unconstrained conversation with a robot for hours at a level that easily passes the Turing test! It is quite a remarkable piece of technology that opens up many new doors...especially in robotics. In this Instructable, I will explore making an animatronic robot assistant that is powered with ChatGPT. The purpose of the robot is to create a more human UX for interfacing with ChatGPT. To achieve this goal, the robot will have a microphone/speaker system that enables a user to communicate entirely verbally. The robot will also have multiple anthropomorphic features like blinking/moving eyes and a mouth.

Achieved

The finished robot is a ROS-based device that runs off of opencv, assemblyai, pygame, and ofcourse ChatGPT. In additional to the original goal of having the robot blink/move its eyes, the robot can also rotate and physically track a user's face via a basic webcam. I am currently visiting Taiwan, so I have specifically programmed the robot to act as an interpreter between Anglophones (like me) and Sinophones. However, due to the ChatGPT platform, the robot can be relatively easily configured to perform other tasks. I have published all CAD/code for this project opensource in this Instructable so that other hackers can use this project as a launching point for building animatronic robots.

Code/CAD Repo: https://github.com/CallMeSwal/BotGPT

Supplies

General Components

1 x Spool of Grey 3D Printer Filament
1 x Spool of White 3D Printer Filament
1 x PC Speakers
1 x High Torque Stepper Motor and Driver
1 x Microphone
1 x Webcam
1 x Arduino Uno
1 x Ubuntu Computer that can run ROS Noetic
Assorted M2, M3, M4, M5 hardware, I recommend buying a box like this: https://www.amazon.ca/1760pcs-Metric-Assortment-Upgrade-Wrenches/dp/B0C38YFL3D/ref=sr_1_1_sspa?crid=ASUDCFLRST8&keywords=m2+m3+m4+m5+screw+kit&qid=1701166871&sprefix=m2%2C+m3%2C+m4%2C%2Caps%2C332&sr=8-1-spons&sp_csd=d2lkZ2V0TmFtZT1zcF9hdGY&psc=1

Eye Mechanism Components

1 x 1000ml bottle of Crystal Clear Liquid Resin
1 x 0.5m of Red Yarn
1 x A4 Printer Label Sticker
1 x Bottle Clear Krazy Glue
5 x TowerPro SG90 Servos (preferable all metal)
1 x Futaba S3004 Servo
1 x HCPA9685 Servo Driver Board
1 x 1Kg Kit of Moulding Silicone

Tools

3D printer
Soldering Iron
Heat Gun

Step 1: Inspiration

I was inspired to build this robot while visiting Taiwan. As a non-Chinese speaker, I am almost entirely reliant on Google Translate to converse with locals here. This results in me constantly staring at a screen while talking to someone which creates extremely non-human experiences. Originally, I wanted to build an animatronic robot that made API calls to Google translate. But after playing with ChatGPT, I realized it would allow me to create a much more powerful/diverse robot.

Step 2: Disclaimer and Opensource

Building a robot is hard, building an animatronic robot doubly so, and building a drone triply so (but that's my December project :) ). At the beginning of this Instructable, I would like to warn you there will most likely be a lot of frustration and exasperation during this build. I strongly recommend that you read the entire Instructable thoroughly before you begin your project. You will need a prerequisite understanding of ROS, python, and robotics in general for this project. I want to make it easy for people to adjust and tailor this project for specific robot solutions that they are trying to build. For that reason, I have released the raw fusion files in addition to the STLs so that users can easily modify the robot. Also for this reason, I chose to build the robot with ROS so that the codebase is highly modular. I have divided the core functions of the robot into four small Python scripts which you can Frankenstein as you wish.

Step 3: Creating the Eye Mechanism

You speak with your eyes….or something cheesy like that! Whether or not the robot looks realistic is primarily dependent upon the eye mechanism. The specific eye mechanism I designed for this robot is a poor imitation of Will Cogley’s simplified eye mechanism. For those of you not aware, Will Cogley is the GOAT creator of hyperrealistic eye mechanisms. My project would not have been possible without all of the open-source work done by Will. In the subsequent steps, I am going to largely regurgitate Will’s process. However, since I am more artistically challenged than Will, I have made some slight modifications to his process that make it easier for non-art gods to follow.

Like Will’s design, my eye mechanism operates on the concept of having the X and Y movements completely independent from each other. Since my eye mechanism is almost three times the size of Will’s, I have added some bracing to make it more sturdy.

Step 4: Printing Eye Mould Positives

To create a hyper-realistic finish, our 3D-printed eyes will be cast in a clear liquid resin. In order to do this, we first need to create a matching silicone mould for the eyes.

The first step to creating the mould is printing out the mould positive. I recommend printing several copies so that we can create multiple moulds simultaneously.

Step 5: Sanding Eye Mould Positives

We need our mould positive to have a extremely smooth finish to ensure no artifacts / blemishes are found in the mould. I started by sanding the mould positive at 100grit and worked my way up to 2500 grit. I then used a polishing compound to create an ultra shiny service. I never got it perfect but it was close enough.

In order to make it easier to sand the eyes / eye mould positives, I have created a tool that allows you to hold the parts in a drill chuck. I strongly recommend using this option when sanding the eyes.

Step 6: Casting Mould With Silicone

Mix together the moulding silicone solution and then pour a generous amount of the mixture into the container you want to use for your mould. Place the mould positive into container so that the flat of the positive is level with the top of the mould. Buoyancy will force the mould positive up, so use some sort of paperweight to keep the positive down. Ideally, the positive should be completely orthogonal to the container, but this doesn’t matter that much since the part is spherically and not oblong-shaped.

Ideally, you want to prevent bubbles from impacting the surface finish of your mould. You can pop bubbles by tapping against the mould or inducing heat from a heat gun on the mixture.

Depending on the moulding solution you use, the mould should be fully cured between 1-2 hours. After you are certain the mould is cured, slip the positive out of the mould. You will probably need to rip the disposable container in order to do this.

Step 7: 3D Printing Eyes

Next 3D print the eyes. I recommend printing the parts in a matte white filament.

Step 8: Pre Processing Eyes

Once again sand the parts from 100grit all the way to 2500grit and then use a polishing compound. This may sound like a lot of work, but it’s quite easy with a drill press. Once the eyes are polished, you can add on the iris. Will Cogley, manually painted the iris on the part but I found this way too difficult. Instead, I just printed irises on a piece of label paper and cutout/stuck the stickers on my eyes. If you want to choose this option, I have attached a PDF of correctly sized eyes for my mechanism to this instructables.

I would recommend taking a crack at actually painting the irises. While I was working on this project, a random Taiwanese girl noticed me struggling to paint the eyes and offered to help me. In 10 minutes, she did a much better job than I could in 2 hours, so maybe it’s just me who struggles to paint irises 🙂

Another option to make the eyes seem more realistic is to add veins protruding from the sides. This can be done by cutting extremely thin pieces of red yarn and super-gluing them erratically to the eye. Ensure you use a lot of super glue so that the yarn does not move during the casting process.

Step 9: Casting Eyes

Once you are happy with your eyes, casting them in the moulds with clear resin. Depending on your resin product, the liquid should cure within 24 hours.

Step 10: Post Processing Eyes

Once the resin is fully cured, remove the finished eyes from mould. They may be some artifacts/blemishes attached to the eye. DO NOT USE SANDPAPER TO TRY TO GET RID OF THESE BLEMISHES! It will not work. I did have some success once again using a polishing compound and precisely applying an exacto knife to cut off any knubs.

Step 11: Assembling Eye Mechanism

Print out the remaining eye mechanism components and then assemble the device as seen in the CAD files. The principle is that the lateral movement and horizontal movement servo motors are completely independent of each other. I noticed that the weight of my eyes could make my bracing sag. For this reason, I created a large amount of clearance between the eyelids and eyes.

Step 12: Mounting Eye Mechanism on Monitor

Next mount your eye mechanism on the computer monitor using the MONITOR_CONNECTOR component. For my robot, I am using an HP 1502 monitor because I found one of these lying around in my workshop. But any monitor would do the job. For this reason, I have attached the raw fusion files so you can edit the CAD as you see fit.

Step 13: Mounting Monitor on Turntable

I want the robot to have an immersive/interactive experience. So in addition to the moving eyes, I created a turntable for the monitor to “slowly” rotate upon. The turntable is driven by high torque stepper motor.

Step 14: Wiring Robot

Wire up all of the servos to the HCPA9685 driver board and then connect the driver board to your Arduino. Connect SCL/SDA pins on the driver board directly to the SCL/SDA pins on the Arduino.

Connect a 5V power source to the HCPA9685 driver board for powering the servos.

Connect the stepper motor to the stepper motor driver and then wire the stepper motor driver to the arduino. Connect an appropriate external PSU to the driver for powering the stepper motor driver.

Step 15: Code Base

Next, I will go through the code base that is running upon the robot. As stated before, the purpose of this documentation is to act as a launching pad for someone to build a ChatGPT robot. My goal is that you understand the scripts well enough so that you can repurpose them for your application.

The robot itself is running on ROS Noetic so it’s highly modular. The first step to get the code base working is to install ROS Noetic on your machine.

Step 16: Arduino Firmware

The Arduino firmware controls the movement of the servos and stepper driver. It forces the eyelids to blink after a random n number of seconds. The eyelids will blink once/twice if n is an even/odd number respectively. The firmware also takes serial input that controls the movement of the eyes. This serial input is sent by a python program that tracks the face of the user. I also have examples on Github that has the eyes just move randomly scanning around a room.

Step 17: Tracking Face

The script FINAL_TRACKING.py uses OpenCV and Haarcascades to locate a human face within the webcam’s FOV. The coordinates of the face are then forwarded over serial to the Arduino.

Step 18: Get Robot to Hear

There are many free SpeechToText online tools. They all suck. Or atleast I had poor experience with them. I am using AssemblyAIs pay to use API to transcribe text in realtime. I have used the API all month and only spent $20 in credits so I am pretty happy with the experience.

The script FINAL_STT.py here transcribes speech to text in realitime using AssemblyAI and then published the finished transcriptions to a ROS topic.

Step 19: Connecting Robot to ChatGPT

ChatGPT has official Python APIs for accessing data. However, I was disappointed by the fact these APIs can’t maintain an existing conversation autonomously. You have to resend all previous prompts to maintain context. Yuck! Instead of using the official apis, I have used this Pyhton script written by for scraping a chatgpt client with Selenium. The response from GPT is again published to a ROS topic.

This Python script itself was original made by Michaelango27: https://github.com/Michelangelo27/chatgpt_selenium_automation. Thanks for creating such a useful resource Mike!

Step 20: Getting Robot to Speak

Since the rest of the project is in Python, I used Pygame to create the UI. Looking back, I probably could have made something better looking with Electron. But this is good enough! It’s pretty simple, the Pygame app is subbed to some ROS topics and blitsthe user input and gpt response on a screen. Using Google Translation API, the text is translated (somewhat consistently) into Chinese (traditional since I am in Taiwan). I then use python tools to convert the text to speech.