Project A: A speller for someone with ALS
Hybrid EOG plus SSVEP, designed with a real partner found through the ALS Association or the rehabilitation hospital. Start with the eyes, add EEG for when the eyes fail. That is the clinical progression, and it is the project that writes your internship essays.
On this pageA semester
Amyotrophic lateral sclerosisAmyotrophic lateral sclerosis (ALS) A progressive disease of motor neurons that removes movement and eventually speech while leaving thought intact; the population many BCIs are built for. Related: Locked-in syndromeLocked-in syndrome Full awareness with almost no voluntary movement, sometimes only the eyes; communication depends entirely on assistive technology., Augmentative and alternative communication (AAC)Augmentative and alternative communication (AAC) Any tool that helps someone who cannot speak communicate, from picture boards to eye-gaze spellers to BCIs.. Glossary entry takes movement and eventually speech while leaving thought intact. Communication devices for people with ALSAmyotrophic lateral sclerosis (ALS) A progressive disease of motor neurons that removes movement and eventually speech while leaving thought intact; the population many BCIs are built for. Related: Locked-in syndromeLocked-in syndrome Full awareness with almost no voluntary movement, sometimes only the eyes; communication depends entirely on assistive technology. , Augmentative and alternative communication (AAC)Augmentative and alternative communication (AAC) Any tool that helps someone who cannot speak communicate, from picture boards to eye-gaze spellers to BCIs. . Glossary entry start with switches, move to eye-gaze cameras, and, when the eyes fail too, have almost nothing left; that is where BCIsBrain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. Related: DecoderDecoder The algorithm that maps neural signals to intended actions, such as a classifier or a Kalman filter. , Closed loopClosed loop A system in which the output feeds back to influence the input, such as a user adjusting their brain activity while watching the cursor it controls. . Glossary entry are aimed. In this project you find one person, through an organization, learn what they actually need, and build a speller that starts with the EOGElectrooculography (EOG) Recording eye position from the standing voltage across the eyeball, which moves with the eyes; also the main source of blink artifacts in EEG. Related: Blink artifactBlink artifact The large, slow deflection an eye blink produces in frontal channels, often ten times the size of the brain signal. , Electroencephalography (EEG)Electroencephalography (EEG) Recording the brain's electrical activity from electrodes on the scalp, a few tens of microvolts of blurred, summed cortical activity. . Glossary entry joystick from Phase 1 and can fall back to the SSVEPSteady-state visual evoked potential (SSVEP) When you stare at something flickering at a fixed rate, visual cortex oscillates at that rate, and the frequency can be read straight out of the EEG spectrum. Related: Canonical correlation analysis (CCA)Canonical correlation analysis (CCA) Finding the combination of channels that correlates best with reference sinusoids at each candidate frequency; the standard SSVEP decoder. , Brain-computer interface (BCI)Brain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. . Glossary entry decoderDecoder The algorithm that maps neural signals to intended actions, such as a classifier or a Kalman filter. Related: ClassifierClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples. , Kalman filterKalman filter An estimator that blends a prediction from a motion model with a noisy measurement, weighting each by how much it trusts it; the decoder behind most intracortical cursor control. , Brain-computer interface (BCI)Brain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. . Glossary entry from Phase 3. The person, not the technology, defines success.
Commercial eye-gaze communication systems cost several thousand dollars and are very good. Why would a person with ALS want anything else?
Eye-gaze systems fail in exactly the situations that matter: bright light, ptosis, lying in bed, and the late-stage slowing of eye movements. Insurance approval can take months that the person does not have. The need is not “a better camera”; it is communication that keeps working as the disease progresses and in the bedroom at 3 a.m. Ask your partner and you will hear a version of this before you propose anything.
Finding the person
Through the ALSAmyotrophic lateral sclerosis (ALS) A progressive disease of motor neurons that removes movement and eventually speech while leaving thought intact; the population many BCIs are built for. Related: Locked-in syndromeLocked-in syndrome Full awareness with almost no voluntary movement, sometimes only the eyes; communication depends entirely on assistive technology. , Augmentative and alternative communication (AAC)Augmentative and alternative communication (AAC) Any tool that helps someone who cannot speak communicate, from picture boards to eye-gaze spellers to BCIs. . Glossary entry Association’s local chapter, a multidisciplinary ALSAmyotrophic lateral sclerosis (ALS) A progressive disease of motor neurons that removes movement and eventually speech while leaving thought intact; the population many BCIs are built for. Related: Locked-in syndromeLocked-in syndrome Full awareness with almost no voluntary movement, sometimes only the eyes; communication depends entirely on assistive technology. , Augmentative and alternative communication (AAC)Augmentative and alternative communication (AAC) Any tool that helps someone who cannot speak communicate, from picture boards to eye-gaze spellers to BCIs. . Glossary entry clinic (the University of Utah has one), or the rehabilitation hospital’s speech-language pathologists. Never by contacting patients directly. Explain that you are a student, that you want to learn what is needed, and that you can offer time and engineering. The speech-language pathologist is your most important ally; they know every device and every failure.
Listen first
Two or three visits before you build anything. Watch how the person communicates now. Time it. Notice what breaks: a caregiver has to recalibrate the camera every hour; the letter board is useless in the dark; the person has one message they need to send fast (“I need suction”) and it takes a minute. Write need statementsNeed statement One sentence, "a way to [do X] for [whom] that [outcome]," with no solution in it. Related: Needs findingNeeds finding Observing clinical work to catch the moments that are slow, awkward, or unsafe, before proposing any solution.. Glossary entry with no solution in them. Bring them back and ask which one matters most. The needs-finding explainer is the method.
The device
Stage one: EOGElectrooculography (EOG) Recording eye position from the standing voltage across the eyeball, which moves with the eyes; also the main source of blink artifacts in EEG. Related: Blink artifactBlink artifact The large, slow deflection an eye blink produces in frontal channels, often ten times the size of the brain signal. , Electroencephalography (EEG)Electroencephalography (EEG) Recording the brain's electrical activity from electrodes on the scalp, a few tens of microvolts of blurred, summed cortical activity. . Glossary entry. Two channels of electrooculography, an amplifier with a very low high-pass, a small board on the headboard, Bluetooth to a tablet. The tablet shows a grid; horizontal and vertical gaze moves the highlight; a long dwell or a deliberate double-blink selects. Drift is the enemy: baseline tracking that re-centres whenever the eyes have been still, and a recentre gesture. Include a one-gesture emergency message that always works.
Stage two: SSVEPSteady-state visual evoked potential (SSVEP) When you stare at something flickering at a fixed rate, visual cortex oscillates at that rate, and the frequency can be read straight out of the EEG spectrum. Related: Canonical correlation analysis (CCA)Canonical correlation analysis (CCA) Finding the combination of channels that correlates best with reference sinusoids at each candidate frequency; the standard SSVEP decoder. , Brain-computer interface (BCI)Brain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. . Glossary entry fallback. When gaze control fails, the same electrodesElectrode The conductor that turns ion currents in tissue into electron currents in a wire; its chemistry and contact quality set the noise you live with. Related: Silver/silver-chloride (Ag/AgCl)Silver/silver-chloride (Ag/AgCl) The standard electrode material for biopotentials, because its chemistry produces a stable, low-noise, low-drift interface with salty gel. , Electrode impedanceElectrode impedance How hard it is to push current through the skin-electrode contact, typically 5 to 50 kilohms with gel and megohms dry, higher at low frequencies. , Half-cell potentialHalf-cell potential The DC voltage that appears wherever a metal meets an electrolyte; two electrodes with different half-cell potentials produce an offset that can swamp the signal. . Glossary entry plus two occipital ones give SSVEPSteady-state visual evoked potential (SSVEP) When you stare at something flickering at a fixed rate, visual cortex oscillates at that rate, and the frequency can be read straight out of the EEG spectrum. Related: Canonical correlation analysis (CCA)Canonical correlation analysis (CCA) Finding the combination of channels that correlates best with reference sinusoids at each candidate frequency; the standard SSVEP decoder. , Brain-computer interface (BCI)Brain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. . Glossary entry: four to eight flickering targets, a two-level letter tree, three to four seconds per selection. It is slower than gaze and it works when gaze does not. Photosensitivity screening is mandatory and is on the intake form.
Stage three: word prediction. Both stages become several times faster with a predictive text model that offers whole words after two letters. Any open-source predictive keyboard library works. Speed is measured in words per minute with the person, not characters per minute on you.
What makes it clinical
The design controls page has you write requirementsRequirements Testable statements of what the device must do, such as "input-referred noise below 1 microvolt RMS from 0.5 to 40 Hz." Related: VerificationVerification Testing that the device meets its requirements; did you build what you specified. . Glossary entry (“selection error rate below 5 percent in the person’s bedroom lighting”), the FMEA page has you list every way it fails and what happens to the person when it does, and the human factors page has you watch a caregiver set it up without help. Do all three. The device is not done when it works for you.
- Contact the organization. Meet the speech-language pathologist. Get onto your lab’s protocol.
- Three listening visits. Ten need statementsNeed statement One sentence, "a way to [do X] for [whom] that [outcome]," with no solution in it. Related: Needs findingNeeds finding Observing clinical work to catch the moments that are slow, awkward, or unsafe, before proposing any solution. . Glossary entry. The person picks.
- Build stage one on yourself. Then with the person, in their bed, in their light, for twenty minutes.
- Iterate on drift handling and the selection gesture until the person can spell a sentence unaided.
- Add the SSVEPSteady-state visual evoked potential (SSVEP) When you stare at something flickering at a fixed rate, visual cortex oscillates at that rate, and the frequency can be read straight out of the EEG spectrum. Related: Canonical correlation analysis (CCA)Canonical correlation analysis (CCA) Finding the combination of channels that correlates best with reference sinusoids at each candidate frequency; the standard SSVEP decoder. , Brain-computer interface (BCI)Brain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. . Glossary entry fallback and test it with the person (screened) for a short session.
- Add word prediction. Measure words per minute and error rate over three sessions.
- Write the requirementsRequirements Testable statements of what the device must do, such as "input-referred noise below 1 microvolt RMS from 0.5 to 40 Hz." Related: VerificationVerification Testing that the device meets its requirements; did you build what you specified. . Glossary entry, the FMEAFMEA Failure mode and effects analysis: a table of every way the device could fail, how bad and how likely each is, and what you did about it. Related: ISO 14971ISO 14971 The international standard for medical device risk management that every company follows. . Glossary entry, and the caregiver-setup observation. Hand over a device, a one-page guide, and your phone number.