What Is Human Augmentation - Sensory Substitution

This instructable is a high-level overview of a type of human augmentation called sensory substitution. typically my work is considered as sensory augmentation rather than substitution, but a lot of my work - especially my early work as an undergrad - is heavily inspired by the field of sensory substitution.

So that leads to the question of what is the difference between augmentation and substitution? The formal definition is that sensory substitution is used to circumvent the loss of a sense by transducing its information through another sensory modality. I’m not the biggest fan of that definition as it can be deceiving to newcomers. You probably read “loss of a sense” and might think that sensory substitution is only about accessibility. That’s not always true, sensory substitution is broadly about altering our perception through sensory modalities we already have.

The three photos in the introduction are the Fingertip Extension Glove and the Ultrasonic Sensory Extension Wearable.

If you have any questions, want to keep up with my work, or just toss around ideas, please do so on my Twitter: @4Eyes6Senses. Thanks!

Step 1: How We Experience the World

The way we experience the world can be divided into two categories. First is sensation, this is physical stimuli that interact with our receptors to create signals. An example of this is light being reflected on the retina of your eye or a physical object interacting with the nerves in your skin.

The second is perception, and this is the point at which meaning is extracted from neural signals, this affects the way we interpret the input from the world around us and how we categorize the signals that are generated from our sensations. For example, you're walking in a park and there's light that's reflecting off an object into your retina. Almost immediately, your perception categorizes that object as a ladybug.

Step 2: Experiment: Tactile Illusion

So now we will have a quick experiment where we will trick your perception using a tactile illusion. The definition of a tactile illusion is the misinterpretation of a real sensory experience / a misinterpretation of an object's properties.

First, have an LED or a circular object nearby.

Second, cross your middle finger over your index finger.

Third, place the LED between the two crossed fingers.

Bonus points if you are able to stroke your nose with the crossed fingers!

This tactile illusion is called the Aristotle illusion, and if done successfully, should feel like you are touching two separate objects or that you have two noses.

Step 3: What Does This Experiment Tell Us About the Brain?

This experiment gives a lot of insight into the brain and our perception.

One is that our perception can be "tricked". Now tricked isn’t the most accurate word here, really our perception is working exactly how it’s been trained to. Every second of every day our 5 senses are sensing the world and we have trained our perception to label those inputs accurately. So when we place something into our crossed fingers our perception is recalling previous interactions from those same locations (and this is highlighted with the red on the figure).

When your fingers are uncrossed, you would need two objects to touch the red highlighted area. When the fingers are crossed and something touches that same area, our perception tells us that we should be sensing more than one object, funny stuff.

Step 4: So Let's Say...

I duct tape my twisted fingers together for several months. What would happen to my perception? Would the illusion still work?

So, no one has ever done this exact experiment to my knowledge. But there have been several similar studies that show the illusion would not still work. My perception would adapt and the items that go between my twisted index and middle finger would feel like a single object.

This concept is called neuroplasticity, and is the ability for our perception/cognition/neural network to recognize and adapt to new external stimuli. I won't go further into neuroplasticity here, but I highly suggest you research this topic further as an HCI or augmentation maker, the topic has given me so many valuable insights as a researcher.

Step 5: Other Sensory Illusions

These types of sensation illusions have been around for decades (hence the name, Aristotle illusion), and there are visual illusions, auditory illusions, taste, and smell illusions. Every couple of years one of these types of illusions become really popular, for example; the dress that either looks black and blue or white and gold; or the audio file that either sounded like Yanny or Laurel. The one photo on the right is a visual illusion that originates from the 1800s and is a dog and its owner (if you can find it).

An interesting difference between tactile illusions and visual illusions is that tactile illusions take longer to adapt. For example, you can twist your fingers again and get the same tactile illusion, yet you can’t close your eyes for several minutes and look back at the dog illusion and not identify the man (that’s assuming that you can see him).

Step 6: How Does This Relate to Sensory Substitution?

So how does perception and sensation relate to sensory substitution? If someone has a defect in one of their 5 senses, for example being deaf or blind, is it still possible for them to perceive this information by substituting it with one of their other four senses? In other words, would it be possible to “see” through touch or “hear” through taste and have the brain perceive those inputs accurately?

The answer is yes! And we have been doing it for centuries. Figure 2 is someone using braille to read through touch, and figure 3 is someone using a white cane to navigate through touch and sound. If you are unfamiliar, braille has been around since the early 1800s and the white cane dates back to biblical times.

Step 7: Sensory Substitution Mid-1900s

The mid-1900s (1930s - 1970) gave birth to the field and term sensory substitution. The man on the left is Theodor Erismann, a Russian researcher who did several goggle/glasses experiments that proved the brain had neuroplasticity. His famous study, the upside-down goggles, turned the wearer's vision upside down. After wearing the goggles for several weeks the brain would flip the wearer's vision right-side-up, showing that the brain's perception can adapt to external change (he did several studies in the same vein, such as prismatic goggles). The man in the middle, who is a major inspiration to me, is the father of computational sensory substitution, named Paul Bach-y-Rita. Bach-y-Rita created several devices that studied the idea of neuroplasticity and introduced the idea as a way to treat people with neurological disorders. His most famous substitution is the ability to see through touch, where a camera sensor would generate a low-resolution image that would be displayed by an electrotactile tongue display unit (an earlier device can be seen on the right that used blunted needles on the user's back to display the output). An example of how his systems work can be seen below:

Step 8: Modern Day Sensory Substitution

Modern-day sensory substitution has extended from strict accessibility to creative domains. Figure 1 is the FeelSpace belt and Neosensory vest which use haptics to represent a sense. The FeelSpace belt gives the user a sense of direction, after wearing the belt for several days the wearer has an increased sense of the direction north. The Neosensory Vest transposes the sense of hearing with touch (haptics), where users can build an understanding and perception of language and sounds that are sensed by the vest.

Step 9: Modern Day Sensory Substitution

Neil Harbisson is an artist with a sensor that is connected to the occipital bone. Neil was born with achromatopsia (vision only black and white), the sensor implanted in his head enables him to gain color perception that he wasn't born with. The implant transduces the sense of colors as sounds, which he uses to perceive and create art.

The second example is a synesthesia mask from Instructables, this is an interesting example of substitution because synesthesia is an example of sensory perception being crossed with another sensation from birth. This project turns color perception into smells that the wearer can perceive, and is still a valuable example of how humans can substitute senses.

Step 10: End

While I could make several more "steps" of sensory substitution examples, I'll let you search further what the field of sensory substitution has contributed to HCI, cognitive science, and augmentation.