Intangible Surface is a gesture and IoT based touchless interface to interact with the digital world.

Step 1: What Problem I Am Trying to Solve?

A button is the most common interface to interact with the digital world. It could be as simple as a light switch or at a pedestrian crossing or it could be as complex as in the elevator or a kiosk. These buttons can act as a catalyst in the spread of viruses. The problem I am trying to solve is to reduce the touch-based interfaces a.k.a the catalysts by replacing it with a gesture and IoT based interface that is intangible, simple, and intuitive to operate.

Step 2: How I Am Trying to Solve It?

My solution is divided into two parts

Method 1: Replace the tangible buttons with an intangible gesture-based interface. A person will then interact with the digital world (light, fan, elevator, etc.) using gesture(s) instead of touch.

Method 2: No matter how much you try you will always touch your smartphone. So why not enable your smartphone to complement the gesture-based interface. It will make some complex interfaces like an elevator easy to use (to see the elevator control using a smartphone please go to the elevator application section i.e. end of the page).

A person will swipe right to turn ON a light/fan and swipe left to turn it OFF. Rotate hand clockwise or anticlockwise to increase or decrease a fan's speed or change a light's intensity. The same device can also be used for some complex interfaces like an elevator. Swipe up or down to call the elevator based on the direction you are going. Swipe or rotate your hand to select the floor. And the magical part is that all of these actions are possible using a single interface. No need for a separate knob to control the fan speed. No need for multiple buttons to select the floor numbers. Selecting 2 digit floor on an elevator using gestures could be a multi-step process, therefore, you also have the option of controlling an elevator using your smartphone. Not only the elevator but you can control almost anything using your smartphone.

Step 3: Circuit Diagram

The main elements of my design are

Gesture sensor
Arduino Wi-Fi 1010
LCD
Smartphone

The electronic modules I used for prototyping are grove based. Connecting them is a child's play. The sensor and LCD both use I2C protocol and the Arduino I am using has only one I2C port so I have used an I2C hub to connect all three modules. Please refer to the below schematics for connecting the modules.

Step 4: Lets Assemble It

I have used a product box of a charger to assemble my project. Ideally, I would love to design and print a custom enclosure but I don't have access to a 3D printer. I have slightly modified the box and it does a fantastic job of housing the circuit. It also hides the cluttered wiring and makes it look clean.

Code for this project is available here.

Step 5: How Does It Work?

A person can control any device using intangible surface via two methods

Gesture
IoT (I don't want to call it IoT, please suggest me a better name)

5.1 Gesture

The gesture sensor I am using can detect 9 different gestures

Left
Right
Up
Down
Clockwise
Anti-clockwise
Forward
Backward
Wave

The sensor detects the gesture and sends it to the Arduino. Arduino then translates the gesture into a specific meaning based on the device we want to control. Also, the number of gestures used depends on the device we want to control. The interface at pedestrian crossing needs only 1 gesture and light needs 2 whereas a fan needs 4 gestures.

5.2 IoT (Someone Else's Computer)

In this method, your smartphone acts as a remote control. The app on your phone uses the MQTT protocol over WiFi to communicate with the intangible surface. When you click a button on your mobile app, it publishes a message to a topic. The Intangible Surface reads that message by subscribing to that topic. After processing the message it sends a control signal to the device it is controlling. Intangible Surface then publishes the feedback message to the topic which is then reflected by the mobile app.

To know more about MQTT protocol please click here. I used the Mosquitto broker installed on a RaspberryPi (acting as a server) to test this design. I have also tested it with CloudMQTT. You can use any MQTT broker of your choice.

A clear benefit of this method is that multiple people can control the same device at the same time. So you and your friend can select different floor numbers in an elevator at the same time using your smartphones.

The smartphone app I am using to talk to the Intangible Surface is called MQTT Dashboard. I have designed various control interfaces in it according to the different applications.

Step 6: Applications - Pedestrian Crossing

I have developed some application interface demonstrating the capability of the Intangible Surface.

You might have seen and used a switch at a pedestrian crossing which looks like the one in the above image.

It's a very simple interface. You press the button and wait for a while until the signal turns green. Once the signal is green, it tells you to walk. This functionality can be replicated using intangible surface. In the new interface just swipe LEFT or RIGHT to activate the signal. Voila!

Step 7: Applications - Light Switch

You must have used an on/off switch to control a light or any other electrical appliance. It can also be replaced with our touch-less interface. Simply swipe RIGHT to turn ON the light and swipe LEFT to turn it OFF.

Step 8: Applications - Fan Control

You might have seen a fan switch and a knob to control its speed(Sorry! I can't find a royalty-free image). Lets replicate that interface using Intangible Surface.

Swipe RIGHT to turn ON and swipe LEFT to turn OFF. Rotate your hand CLOCKWISE to increase the speed and ANTICLOCKWISE to decrease the speed.

Step 9: Applications - Elevator Summon

A common elevator has two interfaces, one to summon it (outside), and another to select a floor (inside). The elevator summon interface look like the one in above image.

To call the elevator using Intangible Surface, simply swipe UP or DOWN based on the direction you want to go.

Step 10: Applications - Elevator Floor Selection Interface

The floor selection interface inside the elevator is generally quite complex. You will find lots of buttons like the one shown in the image above.

These buttons are divided into 3 main category

Floor selection buttons.
Door control buttons.
Alarm/Emergency button.

We can replace these 3 functionalities using our Intangible Surface.

Using the touch-less interface we can select the floor by rotating our hand CLOCKWISE or ANTICLOCKWISE. If a building has too many floors then you can swipe LEFT or RIGHT to decrease or increase the floor number by a factor of 5, respectively.
For door control, simply swipe UP to open the door and swipe DOWN to close.
To activate the alarm, wave your hand.

Step 11: Applications - Mobile As a Remote (IoT)

I have implemented the Elevator control using IoT also. Using this method, one can interact with the elevator using a smartphone.

Step 12: Future Work

I want to prototype one more feature and that is to "auto-detect Intangible Surface devices around you in the mobile App".

If Intangible Surface is deployed indoors then use BLE's advertise feature to inform users around it about the device's identity.
If Intangible Surface is deployed outdoors then use GPS in your smartphone to complement the above method.

Unfortunately in Arduino MKR 1010, you can use either WiFi or BLE. I am using WiFi to communicate with the MQTT server therefore I will need a separate BLE radio which I don't have at the moment.

Step 13: Summary

Intangible Surface is an extremely versatile and cost-effective design that can be used to replace a number of touch-based interfaces, simple as well as complex. This device makes you touch less stuff and do more things. Replace the touch-based interface using this and Protect yourself and the people around you from COVID19.

Which interface you would like to replace?

Open Source is LOVE. Happy Making and Take Care:-)