Project F: Reproduce a paper

Most published decoding accuracies are a little too good. Not from dishonesty, usually, but from choices: a favourable cross-validationCross-validation Testing a model on data it never saw during training, repeatedly, so the reported accuracy means something. Related: Data leakageData leakage Information from the test set sneaking into training, for example by shuffling trials from one session across both; it inflates accuracy and is the most common flaw in student BCI papers. , OverfittingOverfitting A model that has memorized its training data, including its noise, and fails on anything new. . Glossary entry split, a tuned hyperparameter, a subject dropped. The only way to know what a number means is to try to get it yourself. Pick one of the four targets below, download the public data, implement the method from the paper’s description, and report your number next to theirs with your best explanation of the gap. Then post it. A reproduction with an honest discrepancy is worth more to a lab than a novel result with none.

Four targets, in order of difficulty

EEGNetEEGNet A compact convolutional neural network for EEG, published in 2018, that became the standard deep-learning baseline because it is small enough to train on a laptop. Related: BraindecodeBraindecode A PyTorch library of deep-learning models for EEG, including EEGNet and ShallowConvNet, with dataset loaders. . Glossary entry on BCI Competition IVBCI Competition IV A 2008 set of benchmark EEG datasets, especially dataset 2a (four-class motor imagery), still the most-cited test bed. Related: Motor imageryMotor imagery Imagining a movement without making it, which changes rhythms over motor cortex in a way a classifier can sometimes detect. , MOABBMOABB Mother of All BCI Benchmarks: a library that runs classifiers across many public EEG datasets the same way so results are comparable. . Glossary entry 2a. The 2018 paperEEGNet, a compact convolutional neural network for EEG-based brain-computer interfacesLawhern, Solon, Waytowich, et al. · J. Neural Eng. · 2018A two-thousand-parameter network that mirrors the classical pipeline, and the first deep model to try on your data. reports four-class motor imageryMotor imagery Imagining a movement without making it, which changes rhythms over motor cortex in a way a classifier can sometimes detect. Related: Sensorimotor rhythm (mu)Sensorimotor rhythm (mu) An 8 to 13 Hz rhythm over motor cortex that weakens when you move, or imagine moving, the corresponding body part. , Event-related desynchronization (ERD)Event-related desynchronization (ERD) The drop in rhythmic power over a cortical area when it becomes active, such as mu weakening when you imagine moving your hand. , BCI inefficiencyBCI inefficiency The finding that around one in five people cannot control a motor-imagery BCI even after training; once called "BCI illiteracy," a label the field is retiring. . Glossary entry accuracy on nine subjects. MOABBMOABB Mother of All BCI Benchmarks: a library that runs classifiers across many public EEG datasets the same way so results are comparable. Related: BraindecodeBraindecode A PyTorch library of deep-learning models for EEG, including EEGNet and ShallowConvNet, with dataset loaders., BCI Competition IVBCI Competition IV A 2008 set of benchmark EEG datasets, especially dataset 2a (four-class motor imagery), still the most-cited test bed.. Glossary entry downloads the data and BraindecodeBraindecode A PyTorch library of deep-learning models for EEG, including EEGNet and ShallowConvNet, with dataset loaders. Related: EEGNetEEGNet A compact convolutional neural network for EEG, published in 2018, that became the standard deep-learning baseline because it is small enough to train on a laptop., MOABBMOABB Mother of All BCI Benchmarks: a library that runs classifiers across many public EEG datasets the same way so results are comparable.. Glossary entry has the model. The exercise is matching their preprocessing and their split (within-subject, session 1 train, session 2 test) and seeing how close you land. Then run the RiemannianRiemannian geometry classifiers Methods that treat each trial's covariance matrix as a point on a curved space and classify by distance there; the strongest simple baseline for EEG today. Related: ClassifierClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples., Common spatial patterns (CSP)Common spatial patterns (CSP) Spatial filters that maximize the variance of one class while minimizing the other, the classic feature extractor for motor imagery.. Glossary entry baseline on the same split and note that it is about as good. Two weeks.

A Kalman filterKalman filter An estimator that blends a prediction from a motion model with a noisy measurement, weighting each by how much it trusts it; the decoder behind most intracortical cursor control. Related: DecoderDecoder The algorithm that maps neural signals to intended actions, such as a classifier or a Kalman filter. , Bayesian estimationBayesian estimation Updating a belief about a hidden quantity by combining what you expected with what you measured, in proportion to their reliabilities. . Glossary entry cursor decoderDecoder The algorithm that maps neural signals to intended actions, such as a classifier or a Kalman filter. Related: ClassifierClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples. , Kalman filterKalman filter An estimator that blends a prediction from a motion model with a noisy measurement, weighting each by how much it trusts it; the decoder behind most intracortical cursor control. , Brain-computer interface (BCI)Brain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. . Glossary entry on intracortical data. The DANDIDANDI archive A public repository of neurophysiology data in NWB format, including human intracortical BCI recordings from published papers. Related: Neurodata Without Borders (NWB)Neurodata Without Borders (NWB) The standard file format for neurophysiology data, so recordings from different labs can be read the same way.. Glossary entry archive and the Neural Latents Benchmark host monkey and human motor cortex recordings with hand or cursor kinematics. Bin spikesSpike Shorthand for an action potential as seen by an electrode, a brief blip about a millisecond wide. Related: Action potentialAction potential The brief, all-or-nothing voltage spike a neuron fires, about a millisecond long and about 100 millivolts tall across its membrane. , Spike sortingSpike sorting Deciding which spikes on an electrode came from which neuron, by clustering their waveform shapes. , Extracellular recordingExtracellular recording Recording with an electrode near neurons rather than inside one; you see small, fast blips from nearby cells rather than the full membrane swing. . Glossary entry at 20 ms, fit a linear observation model from velocity to firing rates, and run the Kalman filterKalman filter An estimator that blends a prediction from a motion model with a noisy measurement, weighting each by how much it trusts it; the decoder behind most intracortical cursor control. Related: DecoderDecoder The algorithm that maps neural signals to intended actions, such as a classifier or a Kalman filter., Bayesian estimationBayesian estimation Updating a belief about a hidden quantity by combining what you expected with what you measured, in proportion to their reliabilities.. Glossary entry exactly as the BrainGate papers describe. Report the correlation between decoded and true velocity. Three weeks; the Kalman explainer is the preparation.

A seizure detector on the TUHTUH EEG Corpus Temple University Hospital's archive of tens of thousands of clinical EEGs, with a labelled seizure subset. Related: Seizure detectionSeizure detection Automatically flagging seizures in EEG, a real clinical need and a good first machine-learning project on the TUH corpus. . Glossary entry corpus. Request access to the Temple University seizure corpus. Band-power features per two-second window, a gradient-boosted classifierClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples. Related: Linear discriminant analysis (LDA)Linear discriminant analysis (LDA) A simple, robust classifier that draws a straight boundary between classes; the workhorse of EEG decoding for two decades. , Support vector machine (SVM)Support vector machine (SVM) A classifier that finds the boundary with the widest margin between classes; strong on small datasets. , Cross-validationCross-validation Testing a model on data it never saw during training, repeatedly, so the reported accuracy means something. . Glossary entry, and evaluation the way clinicians care about: sensitivity to seizures and false alarms per 24 hours, not accuracy per window. Compare to the published baselines on the corpus. Three to four weeks; the data are large.

A speech decoderDecoder The algorithm that maps neural signals to intended actions, such as a classifier or a Kalman filter. Related: ClassifierClassifier An algorithm that assigns each trial a label, such as left hand or right hand, after being trained on labelled examples. , Kalman filterKalman filter An estimator that blends a prediction from a motion model with a noisy measurement, weighting each by how much it trusts it; the decoder behind most intracortical cursor control. , Brain-computer interface (BCI)Brain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. . Glossary entry. The data behind the 2023 speech-BCIBrain-computer interface (BCI) A system that turns measured brain activity into control of something outside the body, from a cursor to a wheelchair to a speech synthesizer. Related: DecoderDecoder The algorithm that maps neural signals to intended actions, such as a classifier or a Kalman filter. , Closed loopClosed loop A system in which the output feeds back to influence the input, such as a user adjusting their brain activity while watching the cursor it controls. . Glossary entry papers (Willett et al.A high-performance speech neuroprosthesisWillett, Kunz, Fan, et al. · Nature · 2023Speech decoded from motor cortex at conversational rates, with data public for your own reimplementation.; Metzger et al.A high-performance neuroprosthesis for speech decoding and avatar controlMetzger, Littlejohn, Silva, et al. · Nature · 2023The second 2023 speech BCI, decoding speech and facial movement together, with public data.) are public. Reimplementing the recurrent network that maps neural features to phonemes is a real project, a month or more, and the result is a portfolio piece that few graduate students have. Start here only after the first two.

The method

Read the paper the way the Phase 0 page says. Then read its methods three times, writing down every choice: filterFilter An operation that passes some frequencies and attenuates others; every filter also delays or distorts the timing of what it passes. Related: High-pass filterHigh-pass filter Removes slow drifts and DC offsets, passing everything above a cutoff; set it too high and you erase slow brain activity. , Low-pass filterLow-pass filter Removes fast content above a cutoff; the anti-alias filter is one, and so is the smoothing you apply to a noisy trace. , Notch filterNotch filter A very narrow filter that removes one frequency, usually 60 Hz mains hum; it also rings and distorts nearby content, so fix the hum at the source first. , Phase distortionPhase distortion Different frequencies being delayed by different amounts, which smears and shifts waveforms such as an ERP peak. . Glossary entry settings, epochEpoch A fixed-length slice of the recording locked to an event, such as one second starting 200 milliseconds before each stimulus. Related: Trial averagingTrial averaging Adding many time-locked epochs together so the consistent response grows and the random noise shrinks by the square root of the number of trials. , Baseline correctionBaseline correction Subtracting the mean of the pre-stimulus interval from each epoch so trials line up. . Glossary entry windows, normalization, split, hyperparameters, and what was done with subjects or trials that were excluded. Where the paper is silent, you must choose, and every such choice goes in your report as “not specified; I assumed.”

Implement in a clean repository with a README that states the target number, your number, and the gap. Use the same evaluation the paper used first, exactly, even if you think it is flawed; then add the evaluation you think is right and report both.

The report

One page. The paper’s claim. Their number. Your number, with the same split. Your number with an honest split, if different. A list of every place you had to assume. Your best explanation of the gap, in order of likelihood. What you learned about the method that the paper did not say. Post it on your site and, if the gap is interesting, send it to the authors; they are usually glad.