ESP32: Cellular Voice Recognition

Have you thought about using speech recognition with an ESP32? Today I will show you how to use this system through an Android smartphone application. That is, you will see how to connect an ESP32 with a mobile phone and how to use Google's voice recognition. It will also understand the .ino code of ESP32 and better know this voice recognition program that was done by AppInventor.

Step 1: Assembly

Our assembly is simple and composed by the ESP, a led and a 220ohm resistor that was connected on port 23. Just this for our example today.

I emphasize that, in place of the led, you can connect a relay or a triac, which will enable you to connect here an air conditioner, a light bulb, among other residential automation devices.

Step 2: Demonstration

In the video you can see that, in the demo part, we have our assembly in which the ESP32 is powered by USB. The first thing to do to work with voice recognition is, on your smartphone, open your browser and access your local IP address from the microcontroller. At this time, the led will flash, which confirms that the cell phone is connected to the same network as the ESP32. Next, we open the AppESP32 application and issue voice commands that will be recognized and executed by the program.

Step 3: IDEs Used

I always use the arduino IDE to program the ESP32. Already in the case of AppInventor, developed by MIT, this one generates the application for Android.

Step 4: Resources Used

ESP32

Smartphone

Led

220ohm Resistor

Protoboard

Step 5: ESP32 Code

ESP32 [Code Organization]

I always like to leave a skeleton to facilitate its visualization in relation to the source code. An important function is the ReadIncomingRequest, which appears next to a Setup function body and a Loop function body.

Step 6: ESP32 [Definitions and Variables]

In this part I give birth to several definitions and treatment of variables.

//lib necessária para conectar o wifi
#include  <WiFi.h>
 
//led conectado no pino 23
#define ledVerde 23
 
//mensagem enviada pelo client (aplicativo)
String ClientRequest;
 
//ip estático, o mesmo deve ser usado no app do smartphone
IPAddress staticIP(192,168,3,120);
//gateway, deixe aqui o gateway da rede em que está conectado
IPAddress gateway(192,168,3,255);
//máscara, deixe aqui a máscara da rede em que está conectado
IPAddress subnet(255,255,255,0);
 
//objeto do servidor
WiFiServer server(80);
 
//objeto do cliente
WiFiClient client;
 
//variável usada para obter o request do client 
String myresultat;

Step 7: ESP32 [ReadIncomingRequest - Request Read]

Here we work with the function used to read the request without any line break characters such as "\ n" or "\ r", we assign to the String variable the command sent by the client and we verify an HTTP request.

//função usada para a leitura do request sem caracteres de quebra de linha como "\n" ou "\r"
String ReadIncomingRequest()
{
  //enquanto houver bytes enviados pelo client 
  while(client.available()) 
  {
    //atribui para a variável String o comando enviado pelo cliente sem "\r"
    ClientRequest = (client.readStringUntil('\r'));
 
    //se existir "HTTP/1.1" na String então recebe comando, senão o comando não é aceito
    //isso verifica que a solicitação seja HTTP/1.1
    if ((ClientRequest.indexOf("HTTP/1.1")>0))
      myresultat = ClientRequest;
  }
  //retorna variável
  return myresultat;
}

Step 8: ESP32 [Setup]

In the Setup we will initialize variable and set the led pin as output. Also we will initialize the serial as 115200 bits per second to be able to give a print on the screen and give a delay of 10 milliseconds. I display connection information, configure static IP, gateway and mask, and initialize the server.

void setup()
{
  //inicializa varíavel como vazia
  ClientRequest = "";
   
  //define pino do led como saída
  pinMode(ledVerde,OUTPUT);
 
  //inicializa serial com 115200 bits por segundo
  Serial.begin(115200);
  //aguarda 10ms
  delay(10);
 
  //A partir daqui conecta wifi
  Serial.println("START");
 
  //configura ssid e senha da rede
  WiFi.begin("robotica", "XXXXXXXX");
 
  //enquanto não conectar exibe "."
  while (WiFi.status() != WL_CONNECTED)
  {
      delay(500);
      Serial.print(".");
  }
 
  //exibe "conectado"
  Serial.println("Connected");
 
  //configura ip estático, gateway e máscara (definidos globais no início do código)
  WiFi.config(staticIP, gateway, subnet);
 
  //exibe ip utilizado pelo ESP
  Serial.println("Your IP is");
  Serial.println((WiFi.localIP()));
 
  //inicializa servidor
  server.begin();
}

Step 9: ESP32 [Loop]

In this step we get the client and configure the LED control according to the received command.

void loop()
{
    //obtém cliente
    client = server.available();
 
    //se ele for nulo, retorna até que ele realmente exista
    if (!client) 
      return; 
 
    //enquanto não existir request aguarda
    while(!client.available())
      delay(1); 
 
    //obtém request utilizando a função local ReadIncomingRequest
    ClientRequest = (ReadIncomingRequest());
     
    //retira dados da página e obtém apenas o comando enviado
    ClientRequest.remove(0, 5);
    ClientRequest.remove(ClientRequest.length()-9,9);
 
    //controla led conforme o comando recebido
    if (ClientRequest == "acender") 
      digitalWrite(ledVerde,HIGH);
    if (ClientRequest == "apagar")
      digitalWrite(ledVerde,LOW);
    if (ClientRequest == "piscar") 
    {      
      digitalWrite(ledVerde,HIGH);
      delay(500);
      digitalWrite(ledVerde,LOW);
      delay(500);
      digitalWrite(ledVerde,HIGH);
      delay(500);
      digitalWrite(ledVerde,LOW);
      delay(500);
    }
 
    //exibe na página a palavra "OK", caso acessado por um navegador
    //se estiver no aplicativo esta exibição não será feita
    client.println("HTTP/1.1 200 OK");
    client.println("Content-Type: text/html");
    client.println("");
    client.println("");
    client.println("");
    client.println("OK");
    client.println("");
    client.flush();
    client.stop();
    delay(1);
}

Step 10: AppInventor Programming

Import file [AppESP32.aia]

Download the .aia file available on my blog and import it into your AppInventor account by following these steps.

Step 11: Smartphone Communication Test With ESP32

To verify that your cell phone can communicate with ESP32, open the browser and type as shown, remembering that the IP must be the same as defined in "staticIP792_10" in the .ino code of ESP32.

Step 12: AppInventor Program [Designer]

We will use two Labels and three non-visible components: Web, Clock and SpeechReconigzer

Note: Label2 is located just below Label1, with no text inserted

Step 13: AppInventor [Blocks]

1. Initialize global variable with "0". This variable is used so that the speech recognition function is called only once, otherwise, every second, this function will be called and will be in a "uncontrolled" loop.

2. This loop runs as soon as the application opens and it runs from time to time (milliseconds). It's like the Arduino IDE loop, only according to the clock of this component.

3. If the command was not received yet ...

4. Then call the "SpeechRecognizer1" function, which is Google's voice recognition function.

5. Assign variable "1" to the variable, which prevents the function from being called again in this loop. (Until it is set to "0" again).

Step 14: AppInventor [Blocks]

1. After the voice is recognized and the text has already been retrieved ...

2. Inserts the text that has been recognized on the label.

3. If the word was "on," then ...

4. Insert in the label the text "Command sent" plus the command in parentheses.

5. Send the command "ignite" (the ip must be the same as the ESP).

6. Repeat for the other commands.

7. Reset variable so that recognition is made again.

Step 15: Installing the AppInventor Application [Installation Permission]

Before installing the application, enable the installation permission of applications from unknown sources.

On your smartphone go to: Settings -> Security -> Device Administration

Check the option according to the figure.

Step 16: Installing the AppInventor Application [PlayStore]

Open the PlayStore and search for AppInventor. The application is called "MIT AI2 Companion".

Step 17: ​Use and Installation of the Developed Program

Open the downloaded application and see the main screen, as in the image above. It is possible to connect with code or scan with QR code.

Now, in the image below, you see, on the left, the option used to only use the application without it being installed. Detail: This option allows the application to be updated in real time as soon as the "designer" or "blocks" changes.

On the right side you'll see the option used to install the application on your smartphone.