Deep learning for EEG

Deep learning transformed vision and language because those fields have millions of examples. A motor imageryMotor imagery Imagining a movement without making it, which changes rhythms over motor cortex in a way a classifier can sometimes detect. Related: Sensorimotor rhythm (mu)Sensorimotor rhythm (mu) An 8 to 13 Hz rhythm over motor cortex that weakens when you move, or imagine moving, the corresponding body part. , Event-related desynchronization (ERD)Event-related desynchronization (ERD) The drop in rhythmic power over a cortical area when it becomes active, such as mu weakening when you imagine moving your hand. , BCI inefficiencyBCI inefficiency The finding that around one in five people cannot control a motor-imagery BCI even after training; once called "BCI illiteracy," a label the field is retiring. . Glossary entry dataset has a few hundred trials per person. That single fact explains most of what you need to know: the networks that work on EEGElectroencephalography (EEG) Recording the brain's electrical activity from electrodes on the scalp, a few tens of microvolts of blurred, summed cortical activity. Related: Electrocorticography (ECoG)Electrocorticography (ECoG) Recording from electrodes placed on the surface of the brain under the skull, usually during epilepsy surgery evaluation; cleaner and sharper than EEG. , Magnetoencephalography (MEG)Magnetoencephalography (MEG) Recording the tiny magnetic fields produced by cortical currents with superconducting sensors in a shielded room; less blurred by the skull than EEG, and far more expensive. , Volume conductionVolume conduction The spreading of currents from a source through the conductive tissues of the head, which is why one source shows up on many electrodes and why scalp EEG is blurry. , Microvolt (µV)Microvolt (µV) One millionth of a volt. Scalp EEG is tens of microvolts; a AA battery is 1.5 million of them. . Glossary entry are tiny, they are designed to bake in what we already know about the signal, and on fair benchmarks they usually tie the best classical methods. They win clearly in two places: when data are large (sleep staging, seizure detectionSeizure detection Automatically flagging seizures in EEG, a real clinical need and a good first machine-learning project on the TUH corpus. Related: TUH EEG CorpusTUH EEG Corpus Temple University Hospital's archive of tens of thousands of clinical EEGs, with a labelled seizure subset. , EpilepsyEpilepsy A tendency to recurrent seizures; its monitoring units are where seizure-detection algorithms get validated and where ECoG data comes from. . Glossary entry, intracortical speech), and when the features are unknown.

Why EEGNet is shaped the way it is

EEGNetEEGNet A compact convolutional neural network for EEG, published in 2018, that became the standard deep-learning baseline because it is small enough to train on a laptop. Related: BraindecodeBraindecode A PyTorch library of deep-learning models for EEG, including EEGNet and ShallowConvNet, with dataset loaders.. Glossary entry (Lawhern et al., 2018EEGNet, a compact convolutional neural network for EEG-based brain-computer interfacesLawhern, Solon, Waytowich, et al. · J. Neural Eng. · 2018A two-thousand-parameter network that mirrors the classical pipeline, and the first deep model to try on your data.) has about two thousand parameters, which is smaller than a single layer of most image networks. Its layers mirror the classical pipeline. First a temporal convolution: a set of learned band-pass filtersBand-pass filter A high-pass and a low-pass together, keeping only a band such as 0.5 to 40 Hz. Related: FilterFilter An operation that passes some frequencies and attenuates others; every filter also delays or distorts the timing of what it passes. . Glossary entry, one per feature map. Then a depthwise spatial convolution: for each learned filterFilter An operation that passes some frequencies and attenuates others; every filter also delays or distorts the timing of what it passes. Related: High-pass filterHigh-pass filter Removes slow drifts and DC offsets, passing everything above a cutoff; set it too high and you erase slow brain activity. , Low-pass filterLow-pass filter Removes fast content above a cutoff; the anti-alias filter is one, and so is the smoothing you apply to a noisy trace. , Notch filterNotch filter A very narrow filter that removes one frequency, usually 60 Hz mains hum; it also rings and distorts nearby content, so fix the hum at the source first. , Phase distortionPhase distortion Different frequencies being delayed by different amounts, which smears and shifts waveforms such as an ERP peak. . Glossary entry, a learned spatial filterFilter An operation that passes some frequencies and attenuates others; every filter also delays or distorts the timing of what it passes. Related: High-pass filterHigh-pass filter Removes slow drifts and DC offsets, passing everything above a cutoff; set it too high and you erase slow brain activity. , Low-pass filterLow-pass filter Removes fast content above a cutoff; the anti-alias filter is one, and so is the smoothing you apply to a noisy trace. , Notch filterNotch filter A very narrow filter that removes one frequency, usually 60 Hz mains hum; it also rings and distorts nearby content, so fix the hum at the source first. , Phase distortionPhase distortion Different frequencies being delayed by different amounts, which smears and shifts waveforms such as an ERP peak. . Glossary entry across channels, which is exactly what CSPCommon spatial patterns (CSP) Spatial filters that maximize the variance of one class while minimizing the other, the classic feature extractor for motor imagery. Related: Motor imageryMotor imagery Imagining a movement without making it, which changes rhythms over motor cortex in a way a classifier can sometimes detect. , ClassifierClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples. . Glossary entry produces. Then a separable convolution to combine and summarize over time, and a small classifierClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples. Related: Linear discriminant analysis (LDA)Linear discriminant analysis (LDA) A simple, robust classifier that draws a straight boundary between classes; the workhorse of EEG decoding for two decades. , Support vector machine (SVM)Support vector machine (SVM) A classifier that finds the boundary with the widest margin between classes; strong on small datasets. , Cross-validationCross-validation Testing a model on data it never saw during training, repeatedly, so the reported accuracy means something. . Glossary entry. It is CSPCommon spatial patterns (CSP) Spatial filters that maximize the variance of one class while minimizing the other, the classic feature extractor for motor imagery. Related: Motor imageryMotor imagery Imagining a movement without making it, which changes rhythms over motor cortex in a way a classifier can sometimes detect. , ClassifierClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples. . Glossary entry plus band-pass plus LDALinear discriminant analysis (LDA) A simple, robust classifier that draws a straight boundary between classes; the workhorse of EEG decoding for two decades. Related: ClassifierClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples. . Glossary entry, made learnable end to end. That is why it works with a few hundred trials and why its filtersFilter An operation that passes some frequencies and attenuates others; every filter also delays or distorts the timing of what it passes. Related: High-pass filterHigh-pass filter Removes slow drifts and DC offsets, passing everything above a cutoff; set it too high and you erase slow brain activity. , Low-pass filterLow-pass filter Removes fast content above a cutoff; the anti-alias filter is one, and so is the smoothing you apply to a noisy trace. , Notch filterNotch filter A very narrow filter that removes one frequency, usually 60 Hz mains hum; it also rings and distorts nearby content, so fix the hum at the source first. , Phase distortionPhase distortion Different frequencies being delayed by different amounts, which smears and shifts waveforms such as an ERP peak. . Glossary entry can be plotted and interpreted.

ShallowConvNet and DeepConvNet (Schirrmeister et al., 2017Deep learning with convolutional neural networks for EEG decoding and visualizationSchirrmeister, Springenberg, Fiederer, et al. · Hum. Brain Mapp. · 2017ShallowConvNet and DeepConvNet, the standard baselines next to EEGNet, with filter visualizations.) are the other standard baselines, similar in spirit. All three are in BraindecodeBraindecode A PyTorch library of deep-learning models for EEG, including EEGNet and ShallowConvNet, with dataset loaders. Related: EEGNetEEGNet A compact convolutional neural network for EEG, published in 2018, that became the standard deep-learning baseline because it is small enough to train on a laptop., MOABBMOABB Mother of All BCI Benchmarks: a library that runs classifiers across many public EEG datasets the same way so results are comparable.. Glossary entry with the datasets and training loops built in.

What the benchmarks say

MOABBMOABB Mother of All BCI Benchmarks: a library that runs classifiers across many public EEG datasets the same way so results are comparable. Related: BraindecodeBraindecode A PyTorch library of deep-learning models for EEG, including EEGNet and ShallowConvNet, with dataset loaders., BCI Competition IVBCI Competition IV A 2008 set of benchmark EEG datasets, especially dataset 2a (four-class motor imagery), still the most-cited test bed.. Glossary entry runs classifiersClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples. Related: Linear discriminant analysis (LDA)Linear discriminant analysis (LDA) A simple, robust classifier that draws a straight boundary between classes; the workhorse of EEG decoding for two decades. , Support vector machine (SVM)Support vector machine (SVM) A classifier that finds the boundary with the widest margin between classes; strong on small datasets. , Cross-validationCross-validation Testing a model on data it never saw during training, repeatedly, so the reported accuracy means something. . Glossary entry across dozens of public datasets with fixed, fair splits. The recurring result: within-subject, the Riemannian tangent-space classifierClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples. Related: Linear discriminant analysis (LDA)Linear discriminant analysis (LDA) A simple, robust classifier that draws a straight boundary between classes; the workhorse of EEG decoding for two decades. , Support vector machine (SVM)Support vector machine (SVM) A classifier that finds the boundary with the widest margin between classes; strong on small datasets. , Cross-validationCross-validation Testing a model on data it never saw during training, repeatedly, so the reported accuracy means something. . Glossary entry and EEGNetEEGNet A compact convolutional neural network for EEG, published in 2018, that became the standard deep-learning baseline because it is small enough to train on a laptop. Related: BraindecodeBraindecode A PyTorch library of deep-learning models for EEG, including EEGNet and ShallowConvNet, with dataset loaders. . Glossary entry-class networks are within a few points of each other on motor imageryMotor imagery Imagining a movement without making it, which changes rhythms over motor cortex in a way a classifier can sometimes detect. Related: Sensorimotor rhythm (mu)Sensorimotor rhythm (mu) An 8 to 13 Hz rhythm over motor cortex that weakens when you move, or imagine moving, the corresponding body part. , Event-related desynchronization (ERD)Event-related desynchronization (ERD) The drop in rhythmic power over a cortical area when it becomes active, such as mu weakening when you imagine moving your hand. , BCI inefficiencyBCI inefficiency The finding that around one in five people cannot control a motor-imagery BCI even after training; once called "BCI illiteracy," a label the field is retiring. . Glossary entry, P300P300 A positive bump about 300 milliseconds after a rare or meaningful stimulus, the basis of the classic BCI speller. Related: Event-related potential (ERP)Event-related potential (ERP) The small, stereotyped voltage waveform the brain produces in response to an event, invisible in a single trial and revealed by averaging many. , Oddball paradigmOddball paradigm An experiment in which a rare target is mixed in among frequent standards; the rare one produces a P300. . Glossary entry, and SSVEPSteady-state visual evoked potential (SSVEP) When you stare at something flickering at a fixed rate, visual cortex oscillates at that rate, and the frequency can be read straight out of the EEG spectrum. Related: Canonical correlation analysis (CCA)Canonical correlation analysis (CCA) Finding the combination of channels that correlates best with reference sinusoids at each candidate frequency; the standard SSVEP decoder. , Brain-computer interface (BCI)Brain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. . Glossary entry, and the network’s small advantage often disappears when hyperparameters are tuned honestly. Cross-subject, both are poor and the network sometimes does a little better because it can be trained on everyone at once. If a paper claims a large deep-learning gainGain How many times an amplifier multiplies its input; 1000× turns 50 microvolts into 50 millivolts, enough for an ADC to see. Related: Instrumentation amplifierInstrumentation amplifier A precision differential amplifier with very high input impedance and high CMRR, built from three op-amps and sold as a single chip such as the INA128 or AD8232. . Glossary entry on a small EEGElectroencephalography (EEG) Recording the brain's electrical activity from electrodes on the scalp, a few tens of microvolts of blurred, summed cortical activity. Related: Electrocorticography (ECoG)Electrocorticography (ECoG) Recording from electrodes placed on the surface of the brain under the skull, usually during epilepsy surgery evaluation; cleaner and sharper than EEG. , Magnetoencephalography (MEG)Magnetoencephalography (MEG) Recording the tiny magnetic fields produced by cortical currents with superconducting sensors in a shielded room; less blurred by the skull than EEG, and far more expensive. , Volume conductionVolume conduction The spreading of currents from a source through the conductive tissues of the head, which is why one source shows up on many electrodes and why scalp EEG is blurry. , Microvolt (µV)Microvolt (µV) One millionth of a volt. Scalp EEG is tens of microvolts; a AA battery is 1.5 million of them. . Glossary entry dataset, the first question is the split.

Where deep learning does win

Large data. Sleep staging has hundreds of thousands of labelled epochsEpoch A fixed-length slice of the recording locked to an event, such as one second starting 200 milliseconds before each stimulus. Related: Trial averagingTrial averaging Adding many time-locked epochs together so the consistent response grows and the random noise shrinks by the square root of the number of trials. , Baseline correctionBaseline correction Subtracting the mean of the pre-stimulus interval from each epoch so trials line up. . Glossary entry in public datasets; small CNNs reach expert-level agreement and beat feature engineering. Seizure detectionSeizure detection Automatically flagging seizures in EEG, a real clinical need and a good first machine-learning project on the TUH corpus. Related: TUH EEG CorpusTUH EEG Corpus Temple University Hospital's archive of tens of thousands of clinical EEGs, with a labelled seizure subset. , EpilepsyEpilepsy A tendency to recurrent seizures; its monitoring units are where seizure-detection algorithms get validated and where ECoG data comes from. . Glossary entry on TUHTUH EEG Corpus Temple University Hospital's archive of tens of thousands of clinical EEGs, with a labelled seizure subset. Related: Seizure detectionSeizure detection Automatically flagging seizures in EEG, a real clinical need and a good first machine-learning project on the TUH corpus. . Glossary entry, similarly. Intracortical speech decoding, where a recurrent network maps hundreds of channels of neural features to phonemes, is the most dramatic recent result in the field and is only possible with deep models trained on hours of data from one person.

Unknown features. Where nobody knows what to compute, a network that learns its own filtersFilter An operation that passes some frequencies and attenuates others; every filter also delays or distorts the timing of what it passes. Related: High-pass filterHigh-pass filter Removes slow drifts and DC offsets, passing everything above a cutoff; set it too high and you erase slow brain activity. , Low-pass filterLow-pass filter Removes fast content above a cutoff; the anti-alias filter is one, and so is the smoothing you apply to a noisy trace. , Notch filterNotch filter A very narrow filter that removes one frequency, usually 60 Hz mains hum; it also rings and distorts nearby content, so fix the hum at the source first. , Phase distortionPhase distortion Different frequencies being delayed by different amounts, which smears and shifts waveforms such as an ERP peak. . Glossary entry can find it. The price is interpretability, which for a medical device is not optional.

Transfer. Pretrain on many people or many datasets, fine-tune on the new person with a few minutes of data. This is where EEGElectroencephalography (EEG) Recording the brain's electrical activity from electrodes on the scalp, a few tens of microvolts of blurred, summed cortical activity. Related: Electrocorticography (ECoG)Electrocorticography (ECoG) Recording from electrodes placed on the surface of the brain under the skull, usually during epilepsy surgery evaluation; cleaner and sharper than EEG. , Magnetoencephalography (MEG)Magnetoencephalography (MEG) Recording the tiny magnetic fields produced by cortical currents with superconducting sensors in a shielded room; less blurred by the skull than EEG, and far more expensive. , Volume conductionVolume conduction The spreading of currents from a source through the conductive tissues of the head, which is why one source shows up on many electrodes and why scalp EEG is blurry. , Microvolt (µV)Microvolt (µV) One millionth of a volt. Scalp EEG is tens of microvolts; a AA battery is 1.5 million of them. . Glossary entry “foundation models” aim: large self-supervised models trained on thousands of hours of heterogeneous EEGElectroencephalography (EEG) Recording the brain's electrical activity from electrodes on the scalp, a few tens of microvolts of blurred, summed cortical activity. Related: Electrocorticography (ECoG)Electrocorticography (ECoG) Recording from electrodes placed on the surface of the brain under the skull, usually during epilepsy surgery evaluation; cleaner and sharper than EEG. , Magnetoencephalography (MEG)Magnetoencephalography (MEG) Recording the tiny magnetic fields produced by cortical currents with superconducting sensors in a shielded room; less blurred by the skull than EEG, and far more expensive. , Volume conductionVolume conduction The spreading of currents from a source through the conductive tissues of the head, which is why one source shows up on many electrodes and why scalp EEG is blurry. , Microvolt (µV)Microvolt (µV) One millionth of a volt. Scalp EEG is tens of microvolts; a AA battery is 1.5 million of them. . Glossary entry, fine-tuned per task. Early results are promising for clinical tasks and modest for BCIBrain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. Related: DecoderDecoder The algorithm that maps neural signals to intended actions, such as a classifier or a Kalman filter. , Closed loopClosed loop A system in which the output feeds back to influence the input, such as a user adjusting their brain activity while watching the cursor it controls. . Glossary entry paradigms. Watch this space; do not assume it has arrived.

Practical rules

Use BraindecodeBraindecode A PyTorch library of deep-learning models for EEG, including EEGNet and ShallowConvNet, with dataset loaders. Related: EEGNetEEGNet A compact convolutional neural network for EEG, published in 2018, that became the standard deep-learning baseline because it is small enough to train on a laptop. , MOABBMOABB Mother of All BCI Benchmarks: a library that runs classifiers across many public EEG datasets the same way so results are comparable. . Glossary entry; do not write your own training loop for a first project. Compare every network against the Riemannian baseline on the identical split, in the same script, or the comparison is not real. Augment: time shifts, channel dropout, amplitude scaling, and mixing trials all help small EEGElectroencephalography (EEG) Recording the brain's electrical activity from electrodes on the scalp, a few tens of microvolts of blurred, summed cortical activity. Related: Electrocorticography (ECoG)Electrocorticography (ECoG) Recording from electrodes placed on the surface of the brain under the skull, usually during epilepsy surgery evaluation; cleaner and sharper than EEG. , Magnetoencephalography (MEG)Magnetoencephalography (MEG) Recording the tiny magnetic fields produced by cortical currents with superconducting sensors in a shielded room; less blurred by the skull than EEG, and far more expensive. , Volume conductionVolume conduction The spreading of currents from a source through the conductive tissues of the head, which is why one source shows up on many electrodes and why scalp EEG is blurry. , Microvolt (µV)Microvolt (µV) One millionth of a volt. Scalp EEG is tens of microvolts; a AA battery is 1.5 million of them. . Glossary entry datasets. Report cross-session and cross-subject numbers, because networks overfit within-session more than linear models do. Plot the learned temporal and spatial filtersFilter An operation that passes some frequencies and attenuates others; every filter also delays or distorts the timing of what it passes. Related: High-pass filterHigh-pass filter Removes slow drifts and DC offsets, passing everything above a cutoff; set it too high and you erase slow brain activity. , Low-pass filterLow-pass filter Removes fast content above a cutoff; the anti-alias filter is one, and so is the smoothing you apply to a noisy trace. , Notch filterNotch filter A very narrow filter that removes one frequency, usually 60 Hz mains hum; it also rings and distorts nearby content, so fix the hum at the source first. , Phase distortionPhase distortion Different frequencies being delayed by different amounts, which smears and shifts waveforms such as an ERP peak. . Glossary entry; if they do not look like band-passes over motor cortex, ask what the network learned instead (often eye movements).