Controlling Applications Using Brain Signals

OVERVIEW OF OUR LITTLE SCIENCE EXPERIMENT

As the title of this instructable says, we would literally be controlling applications using brain signals. We will look at the design of the hardware, as well as the software, and see how we can design the entire pipeline so that we can control anything we want using our Brain Signals.

Motivation for this project?

As we will see in this instrucatable, there are many different applications in which such a project can be utilized. However, one of the main motivation behind this project is to give people with various physical disabilities an equal opportunity in the society and to elevate them to a point where they can compete on a level parallel to the rest of the society.

Another main highlight of this project was to create a cost effective amplification system that was easily in the affordability range of the general consumer. The results that we obtained from our amplification system were quite satisfactory and gave reliable and accurate results that were comparable to the expensive equipment available in the market.

How does this work?

Essentially what happens is that our brain generates frequencies of different wavelengths depending on the type of activity that a person is performing. The table above shows this. So, lets suppose if you are in deep sleep, that would mean that on your Frontal Lobe (front of your head) you will be generating Delta Waves that have a frequency range from 0-4 Hz. We make use of this information by extracting it using electrodes, amplifiers and filters. Apply some Digital Signal Processing to it, and VOILA! you're a genius. So without any further wait, lets jump into it.

Step 1: A Look at the Hardware Used

Hardware for capturing the Brain Signals

The type of hardware that we use would depend on the application that we would be building. However, with that said, the general pipeline of hardware required for processing the brain signals would be the same. So you need the following.

1. 1x EEG cap (10-20 electrode system)
2. 1x EEG Gel
3. 1x EEG Gel Syringe
4. 4x OPA2337
5. 2x INA2126UA
6. Resistors and Capacitors
7. 1x Microcontroller (I used TM4C123GH6PM Tiva C series microcontroller)
8. 1x UART Cable

Step 2: Design of the Hardware

The diagram shows the High Level Overview of our system. Brain Signals are directed from the brain and captured via the EEG cap. From the EEG cap the signals are sent to the Amplification System, this is a combination of both amplifiers and filters. From here we then send the signals to our microcontroller.

Step 3: Design of Instrumentation Amplifier

Instrumentation amplifier is a precision instrumentation amplifier for accurate, low noise differential signal acquisition. It has a very high common mode rejection ratio (CMRR) and very high input resistance. These characteristics made it suitable for EEG applications. Its gain can be varied by just changing the value of external resistor.

Step 4: Design of High Pass Filter

Some DC offset (about 400-600mV) is always present at the output of the amplifier due to the high gain. So, in order to remove this offset, a high pass RC filter of cut off 0.20 Hz is used which blocks this DC offset and stops the shifting of the signal.

Step 5: Design of Operational Amplifier

This is the second stage of our signal amplification. We are operating the operational amplifier in non-inverting mode. The gain from second stage is also 100. So, the total gain we get is about 10000.

Step 6: Design of the Low Pass Filter

The region of interest of our EEG signal is in the range 0-30 Hz. So, we have used a simple RC low pass filter of cut off 27 Hz. This will attenuate the unwanted frequencies from the signal

Step 7: Putting It All Together

Once that we have designed the Amplification System it is now time to design the PCB layout for it. The diagram shows the end product that got developed using the Eagle Software.

Step 8: And the Amplifier Is Ready

After that the PCB is ready, it was then time to make it a reality and have it ready with all the components added on to it. As it can be seen in the diagram above.

Step 9: Tiva C Series Microcontroller

We are using Tiva microcontroller from TI for ADC sampling and UART. It has two 16 bit timers, fast 12 bit ADC converter, eight 8 bit UART modules DMA, two PWM modules, 48 GPIO pins, I2C.

Step 10: The Expansion Board

We have used expansion board for transmitting data to the COM port for Matlab. The board has RS232 protocol jack and we have used RS232 to USB converter for transferring our data to the Matlab. The board also have 7 segments displays, LDR and LCD space to incorporate it in the board

Step 11: Relay Circuit Module

We have used relay circuit module for controlling various applications. It has optocouplers which isolates the supplies of controller and the relay circuit module.

Step 12: Putting It All Together in a Box

Once that we have everything we need, its time to neatly place it inside of a box, which would prevent the hardware designed from scattering all over the place.

Step 13: The MatLab GUI

For the visualization of data a GUI was created in MATLAB. The GUI played the role of a building bridge between the data coming from the microcontroller and the user. The user could not only see the data coming in real time but also see its real time Fast Fourier Transform along with the results of the correlation, if being taken. The way the GUI communicated with the microcontroller was through the UART interface using the RS 232 protocol. For this communication to take place the GUI was incorporated with the option of selecting the port to which the UART was connected to, along with ability to select the baud rate. Furthermore, the GUI gave the option of closing and opening the selected port.

Step 14: The Control Panel

For the visualization of data, the user was given the option to either start or stop the display of the real time data. The user could also see the real time values of the real time data being plotted, its FFT and correlation. Corresponding to this, a small display panel was also created that showed the detected signal. A major feature of this GUI was the ability to select the application to be controlled. A drop down menu was created for this purpose, which upon selection of an option ran the back end code corresponding to the selection

Step 15: Lets Do Some Science!

The main motivation behind this project was the development of applications that we are able to be controlled through brain signals reliably and with such an accuracy so that person with disabilities could use them with ease. In this project we have controlled different hardware and software applications with the help of the following signals acquired from the brain

• Eye Blinks
• Alpha Waves
• Theta Waves

Step 16: An Introduction to Eye Blinks

Eye blink signals are emitted from the frontal part of human brain. Eye blinks produce an upward detection by means of changing electric field as the cornea gets nearer to the frontal lobe during eye blink. It can be used to control different machines which can help disable people to do their routine tasks

Step 17: Recording of Eye Blinks

To record eye blinks we have placed electrodes at Fp1 and Fp2 according to 10-20 electrode system. The ground electrode is placed on the ear lobe. We are obtaining the differential signal from these two points. Due to the differential amplifier being used, the blinks from the two eyes are opposite in polarity. One will be called as left-blink and the other as right-blink. The plots for real time right and left blink is shown below.

Step 18: Detection of Eye Blinks

To detect eye blinks we have used two methods.

Threshold Detection

This is quite a simple and easy method to detect eye blinks. Noting the maximum value of left eye blink and minimum value of right eye blink we can distinguish the two blinks using if-else statements in the software. However, there is certain weakness of accuracy associated with this method. If for example due to some reason (e.g. dryness of gel, improper electrode placement) the signal strength weakens, then the previously defined threshold may not work and hence, a new threshold has to be set. Moreover, due to the plethora of artifacts present, both of high frequency and amplitude, they can also be mistakenly be detected as an eye blink. The signature of the eye blink, as shown above, was made using Microsoft Excel. Ten different samples of an eye blink were obtained and then averaged out, the result of this averaging gave us the final signature. Once this signature has been obtained, we then correlate the signature with the real time data. If it matches, then the eye blink is detected, otherwise it is discarded as an artifact. The advantage of this method is that it is more reliable because

Correlation Detection

A more reliable method is detection through correlation. In this method we take multiple samples of an eye blink and then average them out to make a signature eye blink. The signature of the eye blink was made using Microsoft Excel. Ten different samples of an eye blink were obtained and then averaged out, the result of this averaging gave us the final signature. Once this signature has been obtained, we then correlate the signature with the real time data. If it matches, then the eye blink is detected, otherwise it is discarded as an artifact. The advantage of this method is that it is more reliable because even if the signal strength weakens, its shape remains more or less same and hence, we are still able to extract the eye blink. An interesting thing to note here is that, since the left and right eye blinks are simply the reciprocal of each other, we need the signature of either one of the two eye blinks.

Step 19: Snake Game Using Eye Blinks

The idea of creating such an application was to give a demonstration of some form of cursor control. This is an imperative application for the disabled as it will give them the capability of using devices such as a laptop or even a mobile phone. The signals that we use for this application are eye blinks. Each blink corresponds to a different action being performed. In the case of the snake game, the snake changes its direction corresponding to the blink detected. Hence, using a combination of blinks, the user can guide the snake towards its goal. Since we were using MATLAB, we had to import a JAVA Robot class into MATLAB to give it the capability to move the cursor. A snake game developed in MATLAB was used for this purpose. The game had four separate buttons each corresponding to a different movement of the snake. The positioning of these buttons were mapped to eye blinks using the JAVA Robot class. Using trial and error, the positioning of the cursor were declared into MATLAB.

Step 20: PDF Reader Using Eye Blinks

Another software application created using MATLAB integrated with the JAVA Robot Class was a PDF reader. The PDF reader allowed the user the move the page up and down with the help of eye blinks. The left blink moved the page up, while the right blink moved the page down. The cursor coordinations used for this are universal. This means that they can be used for almost any windows application such as an internet browser, a word file or as in this case, a PDF reader

Step 21: Toy Car Using Eye Blinks

Apart from the software applications built. A few hardware based applications were also developed to show the power of Application Control through Brain Computer Interface. This again is extremely imperative in the eyes of someone who is disabled as it will allow them the control of any home appliance or something similar to a wheel chair. To demonstrate how the control of a wheelchair would work, we controlled a small toy car. The car was controlled using a transmitter and a receiver. The transmitter of the car was attached to the microcontroller via an optocoupler. The optocoupler was used for the protection of the microcontroller. Finally, Eye blinks were mapped to different movements of the car, hence, corresponding to a particular eye blink the car was able to be controlled in a particular direction.

Step 22: Move on to THE ALPHA WAVES

Alpha waves are associate with the relaxed state of a person, with closed eyes. Their frequency range lies in the band of 8-13 Hz and are dominant when the person’s eyes are closed. When the eyes are opened, alpha waves fades away rapidly.

Step 23: Recording of Alpha Waves

Alpha waves are recorded from the occipital region of the brain (back side). For the measurement, recording electrodes were placed at sites O1 and O2 according to 10-20 system. Ground electrode was placed on the ear lobe. Differential signal was taken from the two recording electrodes. Below is the real time plot of alpha waves.

Step 24: Detection of Alpha Waves

Real time Fourier Transform of the data was taken in MATLAB. Alpha waves were detected when the 10 Hz frequency showed values much greater than the other frequency values. The figure below shows FFT of alpha waves.

Step 25: Car Alert Using Alpha Waves

Once that we had detected Alpha Waves, we decided to make an application which tried to solve an existing problem of drowsy driving. Drowsy driving is becoming a serious problem. In this, the driver of the car because of being stressed or some other reason closes his eyes, and hence, this distraction results in an accident to take place. In order to prevent such a thing from occurring, we came up with a system that we call Car Alert. In Car Alert when the user closes his eyes, the car begins to slow down, an alarm sounds in order to wake the driver up and an emergency light is turned on to alert the other drivers. For demonstration purposes, we controlled a small toy car using the same transmitter receiver method described above. When the subject closed his eyes, alpha waves were generated. On the detection of these alpha waves, the transmitter and the led light were turned on and off using the relay circuit. The alarm sound was produced using MATLAB by importing a .wav file.

Step 26: Stress Detection Using Alpha Waves

Eyes closed in a relaxed state results in the alpha waves of higher magnitude being produced. Under this condition if the subject came under some sort of stress the magnitude of these alpha waves decreased. In order to show this in a more tangible form a small red led was connected. This led was associated with the magnitude of the alpha waves. When the magnitude of the alpha waves was above a certain threshold the led lit up, and otherwise it remained off. The subject was asked to close his eyes, which resulted in the led light to be turned on. Now, in order to put the subject under stress he was asked a series of difficult questions. It was observed that while the subject was in the thought process of answering these questions the led light turned . off, meaning that the magnitude of the alpha waves had decreased and the subject was under stress. And the moment the subject gave the answer and was relieved of this task, the led light turned on again.

Step 27: Finally, Some Theeta Waves!

Theta waves are rightly known as the bridge between the conscious and subconscious mind. They are one of the slower waves and hence are found in the lower frequency band of about 4-7 Hz. These waves are associated with meditation, intuition and memory. The way we acquired these waves were that we asked the subject to relax and close his eyes. After a while, the Alpha waves started to fade away and theta waves began to appear. The FFT shown below clearly shows the frequency lying in the theta range. All the applications that were controlled through Alpha waves can be controlled through theta waves as well.