Project D: Sensory substitution
Implement the vOICe image-to-sound mapping and wear it for a week. Build the feelSpace compass belt. Build an ultrasonic bat-sense wearable that is real assistive tech for the blind. Three neuroplasticity experiments on yourself, with no risk.
You are skimming: the title, the first figure, and the short version. Switch to Read in the header for the full page, or Deep to open every deep dive.
The brain does not care where information comes from; it cares whether the information is useful and consistent. Sensory substitutionSensory substitutionDelivering information from one sense through another, such as an image as a soundscape, and letting the brain learn to use it. Glossary entry exploits this: deliver what one sense would tell you through another, and after days or weeks of wearing the device, people report that the new signal stops feeling like a code and starts feeling like a sense. Three classic devices, each buildable in a weekend, each a legitimate experiment on your own plasticity, and one of them is a real navigation aid for blind users.
A blind person uses a camera-to-sound device for several weeks. What happens in their visual cortex?
It becomes active. Brain imaging of long-term users shows visual cortex responding to the soundscapes, and even shape-selective regions responding to the shapes encoded in them. The cortex is not “visual” so much as “good at spatial shape,” and it takes whatever input carries that. This is the deepest thing sensory substitution teaches, and you can read the studies while you wear the device.
The vOICe
Peter Meijer’s 1992 mapping: a camera image is scanned left to right once per second; each column becomes a moment of sound in which height maps to pitch and brightness to loudness. A bright horizontal line is a steady tone; a diagonal is a sweep; a bright blob is a burst. Implement it in Python with a webcam: grayscale, downsample to 64 by 64, and for each column synthesize a chord of 64 sinusoids with amplitudes from the pixel brightnesses, then play the columns in sequence over one second, panned left to right.
Then wear it. An hour a day for a week, with a blindfold for part of it, doing simple tasks: find the light switch, find the doorway, find the cup on the table. Keep a log each day of what you can do and how it feels. By the end of the week most people can find a door; the interesting entry is the day the sound stops sounding like sound.
The feelSpace belt
A belt with eight to sixteen vibration motors around the waist and a compass module. The motor that points north always vibrates. That is the entire device. People who wore it for six weeks reported a new sense of direction, altered spatial perception, and a feeling of loss when they took it off. Build it with a magnetometer breakout, an Arduino, a motor driver, and coin vibration motors sewn into a belt. Wear it for as long as you can stand, every day, and log.
Bat sense
An ultrasonic rangefinder (or a small time-of-flight lidar) on a headband or a wristband, mapped to vibration intensity or to a tone whose pitch rises as an obstacle gets closer. Sweep your head and you feel the room. This is a real class of assistive device (the Sunu band, the various “smart canes”), and it is the one of the three that a blind user might actually want. If you can, build it with a blind user’s input from the start; the Utah Assistive Technology Program or a local blindness organization is the way to find one.
What to measure
Task performance over days: time to find the door, success rate at locating an object. Your subjective log. For the belt, a pointing test: eyes closed, point north, measure the error, before and after two weeks. For bat sense, an obstacle course with and without. And read: Bach-y-Rita’s original work, the vOICe imaging studies, the feelSpace papers, Eagleman’s Livewired for the popular account.
- Implement the vOICe. Verify a bright vertical bar is a click and a horizontal bar is a tone.
- Wear it an hour a day for a week, blindfolded for tasks. Log daily.
- Build the compass belt. Wear it daily for two weeks. Pointing test before and after.
- Build bat sense. Run an obstacle course with and without. If possible, get a blind user’s feedback.
- Write up all three with performance curves and the subjective log, and what the literature says you should have expected.