Explainer18 min

Classification done honestly

Features, LDA, CSP, and Riemannian classifiers, then the part that matters more than the algorithm, which is how you split the data. Leakage, chance levels, calibration drift, and information transfer rate.

AssumesProject E: The eight-channel EMG armband Project A: The SSVEP speller SpineDecoding / signal processing / ML

On this page18 min

Features
Three classifiers that cover the field
Now the part that matters
Information transfer rate

A classifier learns a rule from labelled examples and applies it to new ones. Every BCI is one. The algorithms are the easy part and this page covers them in a few paragraphs. The hard part, the part where most student projects and a distressing number of papers go wrong, is evaluation: making sure the number you report is the number a new person on a new day would get. That takes most of the page, because it deserves it.

Features

A classifier wants a short vector of numbers per trial. For EMG and oscillatory EEG: band powers or log-variances per channel, sometimes after a spatial filter. For ERPs: the downsampled voltage waveform, channels concatenated. For SSVEP: correlations with reference frequencies. Good features make simple classifiers work; no classifier rescues bad features. The most reliable way to improve a BCI is still to improve the signal.

Three classifiers that cover the field

Linear discriminant analysis fits a Gaussian to each class with a shared covariance and draws the boundary where they are equally likely. Few parameters, hard to overfit, fast, and the default in BCI for two decades. With shrinkage regularization it handles more features than trials, which is the usual EEG situation.

Each class gets a score from its mean, the shared covariance, and its prior:

\delta_k(x) = x^\top\Sigma^{-1}\mu_k - \tfrac{1}{2}\mu_k^\top\Sigma^{-1}\mu_k + \log \pi_k

Here $\mu_k$ is the mean feature vector of class $k$ , $\Sigma$ the covariance shared across classes, and $\pi_k$ the prior probability of class $k$ . A new trial goes to the class with the largest score, so the boundary sits where two scores tie.

Common spatial patterns is a feature extractor for two-class oscillatory problems: it finds spatial filters whose output variance is large for one class and small for the other. The log-variance of a few CSP components, then LDA, is the classic motor imagery pipeline.

The objective is the ratio of the two class variances along the filter:

\max_w \frac{w^\top \Sigma_1 w}{w^\top \Sigma_2 w}

Here $\Sigma_1$ and $\Sigma_2$ are the two classes’ channel covariance matrices and $w$ is the spatial filter. The optimum is a generalized eigenvector, computed in one call.

Riemannian geometry classifiers skip feature engineering: each trial is represented by its channel covariance matrix, which lives on a curved space where the natural distance is not Euclidean. Map the matrices to the tangent space at their mean and run logistic regression. This is the strongest simple baseline on nearly every public EEG dataset and it is three lines in pyRiemann. Learn it after LDA, and use it by default.

The distance between two trial covariance matrices on their curved space is:

d_R(C_1, C_2) = \lVert \log(C_1^{-1/2} C_2 C_1^{-1/2}) \rVert_F

Here $C_1$ and $C_2$ are the two trials’ covariance matrices, $\log$ is the matrix logarithm, and $F$ is the Frobenius norm. Mapping every trial to the tangent space at the mean turns these distances into ordinary vectors for logistic regression.

from pyriemann.estimation import Covariances
from pyriemann.tangentspace import TangentSpace
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
clf = make_pipeline(Covariances('oas'), TangentSpace(), LogisticRegression())

Support vector machines, random forests, and neural networks exist too. On EEG-sized datasets they rarely beat the three above by more than noise, and they overfit more. The deep learning page is about when they do and do not help.

Now the part that matters

Random splittrials shuffled, then cut. Neighbouring trials land on both sidesmeaningless

Trial-wise CVwhole trials held out, but all from one sessionoptimistic

Session-held-outtrain on one session, test on another the same daycloser

Day-held-outtest on a different day, electrodes reappliedthe real number

Anything you fit on the test data, including a filter band, a channel selection, or an ICA decomposition, is leakage. Fit inside the training fold or not at all.

Figure 1. The same data, the same classifier, four ways of splitting it. The accuracy falls at every step to the right and only the rightmost one predicts what happens when you put the thing on a person tomorrow.

Predict before you look

You have 200 trials from one session. You shuffle them, train on 160, test on 40, and get 88 percent. Tomorrow, with the same classifier, what do you expect?

Substantially lower, and today’s data cannot tell you how much. Shuffled trials from one session share everything that drifts slowly: electrode impedances, the participant’s state, the exact cap position. The test set is not a new day; it is the same day with a few trials hidden. Only a second session tells you the cross-session number, and it is typically five to twenty points lower.

Leakage. Any information from the test set that reaches training is leakage, and it inflates the number. Shuffling trials across sessions. Fitting the CSP filters, the normalization, or the ICA on all the data before splitting. Choosing hyperparameters on the test set. Choosing which subjects to include after seeing results. Each of these is common, each is easy to do by accident, and scikit-learn’s pipelines exist to prevent the second one: put every fitted step inside the pipeline so cross-validation fits it on training folds only.

The right splits, in increasing honesty. Shuffled trials within a session: the wrong way, useful only to check the code runs. Grouped by run or block within a session: honest within-session. Train on session one, test on session two: the cross-session number, which is what a user experiences. Leave one subject out: the cross-subject number, which is what a calibration-free product would need. Report the one that matches the claim you are making, and say which.

Chance is not 50 percent. With two classes and forty trials, a coin-flip classifier scores above 62 percent one time in twenty. The binomial distribution gives the upper bound for any trial count; the statistics page has it. Always report the chance bound next to the accuracy. With unbalanced classes, use balanced accuracy or report the confusion matrix, because 80 percent on a task with 80 percent standards is chance.

Balanced accuracy averages the two per-class rates instead:

\text{BA} = \frac{\text{sensitivity} + \text{specificity}}{2}

A majority-class guesser scores 50 percent on this measure no matter how unbalanced the task is.

Calibration and drift. Signals change within a session (electrodes settle, attention fades) and between sessions (cap position, mood, sleep). A deployed BCI recalibrates at the start of each session and often adapts during it. If your evaluation does not include the cost of that, it is not evaluating the system.

Information transfer rate

Accuracy alone misleads: 90 percent on two targets in ten seconds is worse than 70 percent on eight targets in three. Wolpaw’s information transfer rateBrain-computer interfaces for communication and controlWolpaw, Birbaumer, McFarland, et al. · Clin. Neurophysiol. · 2002The review that defined information transfer rate, the one number that compares spellers honestly. combines them. Bits per selection:

B = \log_2 N + P\log_2 P + (1-P)\log_2\frac{1-P}{N-1}

for N targets and accuracy P, and ITR is B times selections per minute. A good SSVEP speller reaches 40 to 60 bits per minute; a P300 speller 10 to 25; motor imagery under 10. Report it, and report the time per selection it assumes.

As a rate, with $T$ seconds per selection:

\text{ITR} = B \times \frac{60}{T}

Stop here if you like. Features, three classifiers, honest splits, chance bounds, ITR. If every BCI paper did this, the field would be smaller and more trusted.

Deep dive Why LDA works so well on EEG 3 min

EEG features are roughly Gaussian, classes differ mostly in their means, and trials are few. LDA’s assumptions match, and its parameter count is small (a mean per class and one shared covariance), so it does not overfit. Shrinkage LDA regularizes the covariance toward the identity, which is essential when features outnumber trials, and the shrinkage amount can be set analytically (Ledoit-Wolf) with no cross-validation. It is hard to beat, and any method that claims to should be compared against it with the same split.

Deep dive Nested cross-validation for hyperparameters 3 min

If you tune anything (the number of CSP components, the regularization strength) you must tune it inside each training fold, using an inner cross-validation, and evaluate on the outer test fold that never saw the tuning. Tuning on the same folds you report is leakage and adds several points. scikit-learn’s GridSearchCV inside cross_val_score does this correctly.

Deep dive Transfer learning and calibration-free BCIs 3 min

Aligning each session’s covariance to a common reference (Riemannian alignment) before classification lets data from other sessions and other people help, and cuts calibration time. It works better for SSVEP and P300 than for motor imagery. The dream of a BCI that works for a new person with no calibration is the field’s current frontier, and it is measured by leave-one-subject-out accuracy, which is currently far below within-subject for everything but SSVEP.

Recall

Name four ways data leakage happens in a BCI evaluation.

Shuffling trials across sessions into train and test; fitting preprocessing (CSP, normalization, ICA) on all data before splitting; tuning hyperparameters on the test set; selecting subjects or trials after seeing results.

Recall

Which split answers 'what will a user experience tomorrow', and which answers 'could this work without calibration'?

Train on session one, test on session two (cross-session); leave one subject out (cross-subject).

Recall

Why report information transfer rate rather than accuracy alone?

Accuracy ignores the number of targets and the time per selection; ITR combines all three into bits per minute so systems with different designs can be compared.

Explain it to your roommate

Explain what this page was about to your roommate in three sentences. No jargon they would not know.