Explainer14 min

Statistics for small experiments

The binomial bound that says when a classifier is above chance, permutation tests that need no assumptions, cluster correction for waveforms, effect sizes, and the multiple-comparisons trap. Enough to not fool yourself.

AssumesClassification done honestly SpineDecoding / signal processing / ML

On this page14 min

When is a classifier above chance?
Permutation tests
Cluster-based permutation for waveforms
Effect sizes
The traps, named

Your classifier scored 62 percent. Your ERP has a bump. Your feedback nights looked better than sham. Each of these is a number, and the question is always the same: how often would noise alone produce a number this big? The tools on this page answer that for the small, messy datasets you actually have. None of them need more than arithmetic and a loop.

When is a classifier above chance?

Two classes, N trials, a classifier that is really guessing. The number it gets right follows a binomial distribution with probability one half. The 95th percentile of that distribution, divided by N, is the accuracy a guessing classifier beats one time in twenty. For N = 40 it is about 62 percent. For 80, about 59. For 200, about 56. For 1000, about 52. Only above that bound does your accuracy mean anything, and the bound should sit next to the accuracy in every report.

The bound is the inverse binomial tail, per trial:

a_{chance} = B^{-1}(1 - \alpha; N, 1/2) / N

Here $B^{-1}$ is the inverse cumulative binomial distribution, $\alpha = 0.05$ , $N$ the trial count, and $1/2$ the guessing rate for two classes.

from scipy.stats import binom
def chance_bound(n_trials, n_classes=2, alpha=0.05):
    return binom.ppf(1 - alpha, n_trials, 1 / n_classes) / n_trials

The bound assumes trials are independent, which shuffled trials from one session are not. So it is the least you need, not the most. For unbalanced classes, use the majority-class rate as the null.

Permutation tests

You want to know whether two conditions differ in some measure: mean amplitude, accuracy, alpha power. Compute the difference. Then shuffle the condition labels, recompute the difference, and repeat ten thousand times. The fraction of shuffles whose difference is at least as large as yours is the p-value. No assumption about distributions, no formula to misremember, and it works for any statistic you can compute. For within-subject designs, shuffle within each participant (swap the two conditions’ labels per person).

The p-value is the tail probability under shuffling:

p = P(|D^*| \ge |d_{obs}|)

Here $d_{obs}$ is the difference measured with the true labels and $D^*$ the differences from shuffled labels.

import numpy as np
def perm_test(a, b, n=10000, rng=np.random.default_rng(0)):
    # a, b: paired measurements per participant
    d = a - b; obs = d.mean()
    signs = rng.choice([-1, 1], size=(n, len(d)))
    null = (signs * d).mean(axis=1)
    return (np.abs(null) >= abs(obs)).mean()

Predict before you look

You test whether an ERP differs between conditions at every one of 200 time points and 32 channels. At a threshold of p below 0.05, how many significant points do you expect if there is no effect at all?

About 320. Five percent of 6400 tests are false positives by construction ( $E = 6400 \times 0.05 = 320$ ). Testing many points and reporting the ones that cross the line is the multiple-comparisons problem, and it produces a “significant” cluster somewhere in nearly every dataset. Cluster-based permutation, below, is the standard fix for waveforms.

Cluster-based permutation for waveforms

Compute a t-statistic at every time point (and channel). Threshold it. Group neighbouring above-threshold points into clusters and give each cluster a mass (the sum of its t-values). Shuffle labels, repeat, and record the largest cluster mass in each shuffle. Your cluster’s p-value is the fraction of shuffles whose biggest cluster was bigger. This controls the false-positive rate across the whole waveform while exploiting the fact that real effects are extended in time and space. MNE implements it in one function. Report the cluster’s extent honestly: the test says a difference exists somewhere in the cluster, not that it starts or ends at the cluster’s edges.

Each cluster gets one number, its mass, the sum of its thresholded t-values:

m = \sum_{cluster} t

Only the largest mass per shuffle is kept, which is what controls the error rate across the whole waveform.

Effect sizes

A p-value says whether an effect is distinguishable from zero. It does not say how big it is. Report the difference in the measure’s own units (2.3 µV) and a standardized effect size (Cohen’s d, the difference divided by the standard deviation across participants). A d of 0.2 is small, 0.5 medium, 0.8 large. Classic ERP effects are large; most “cognitive enhancement” claims are small at best, and a small effect in twelve people is noise.

Cohen’s d standardizes the difference by the spread across participants:

d = \frac{\bar{x}_1 - \bar{x}_2}{s}

Here $\bar{x}_1 - \bar{x}_2$ is the mean difference in the measure’s own units and $s$ is the standard deviation across participants.

The traps, named

What it looks like

The fix

Peeking

Checking significance as data come in and stopping when it appears

Fix N in advance, or use a sequential design that accounts for looking

Forking paths

Twenty defensible analyses, one of which reached 0.05

Pre-register the analysis, or report all twenty

Double dipping

Choosing the time window or channel from the same data you then test

Choose from an independent contrast, or a held-out half

Wrong chance level

Calling 55 percent above chance in a two-class task with 40 trials

Use the binomial bound for your N, not 50 percent

Significance for size

A p-value quoted as though it measured how big the effect is

Report an effect size and a confidence interval next to it

Every one of these gets more likely as the dataset gets smaller, which is exactly the situation you are in.

Figure 1. Five ways to get a number that is not there, none of which involve lying. Each one has a fix that costs less than the experiment did.

Peeking and stopping. Collecting data until the p-value crosses 0.05 guarantees it will. Set N in advance.

Garden of forking paths. Trying several windows, channels, filters, and rejection thresholds and reporting the one that worked. Pre-register the analysis.

Double dipping. Choosing the channel or window where the effect looks biggest, then testing it there. The selection already used the data.

Correlation across trials treated as independence. Trials from one person are not independent samples of the population. Test across participants.

Confusing significance with size. A p of 0.001 with a d of 0.1 is a real, tiny effect, probably from a large N. Say so.

Stop here if you like. Binomial bound, permutation, cluster correction, effect size, five traps. That is the statistics this phase needs.

Deep dive Bayesian alternatives 2 min

A Bayes factor compares how well the data are predicted by “there is an effect” versus “there is none,” and can express evidence for the null, which a p-value cannot. For BCI accuracy, a Bayesian estimate of the accuracy with a credible interval is often more useful than a bound. The tools (PyMC, brms) are more work to learn; do it when a reviewer asks or when you need to argue that something does not work, which the neurofeedback project may require.

Deep dive Information transfer rate has error bars too 2 min

ITR is computed from an accuracy, and the accuracy has uncertainty. Bootstrap: resample selections with replacement, recompute ITR, repeat, report the 2.5 and 97.5 percentiles. A speller reported at 42 bits per minute with an interval from 30 to 50 is a different claim from one at 42 exactly.

For $N$ targets at accuracy $P$ , the bits per selection are:

B = \log_2 N + P\log_2 P + (1-P)\log_2\frac{1-P}{N-1}

Resample the selections, recompute $B$ each time, and the percentiles of the results are the interval.

Recall

What accuracy must a two-class classifier exceed on 40 trials to be above chance at the 5 percent level, roughly, and why is that still not enough?

About 62 percent. The bound assumes independent trials; correlated trials from one session make it optimistic, so cross-session or cross-subject testing is still needed.

Recall

How does a permutation test work and what is its main advantage?

Shuffle the condition labels many times and recompute the statistic; the p-value is the fraction of shuffles at least as extreme as the observed value. It assumes nothing about distributions and works for any statistic.

Recall

What problem does cluster-based permutation solve, and what does it not tell you?

It controls false positives across thousands of time-channel tests by testing cluster mass against a shuffled maximum. It does not establish where within the cluster the effect begins or ends.

Explain it to your roommate

Explain what this page was about to your roommate in three sentences. No jargon they would not know.