Statistics for small experiments

You are skimming: the title, the first figure, and the short version. Switch to Read in the header for the full page, or Deep to open every deep dive.

When is a classifier above chance?

Two classes, N trials, a classifier that is really guessing. The number it gets right follows a binomial distribution with probability one half. The 95th percentile of that distribution, divided by N, is the accuracy a guessing classifier beats one time in twenty. For N = 40 it is about 62 percent. For 80, about 59. For 200, about 56. For 1000, about 52. Only above that bound does your accuracy mean anything, and the bound should sit next to the accuracy in every report.

from scipy.stats import binom
def chance_bound(n_trials, n_classes=2, alpha=0.05):
    return binom.ppf(1 - alpha, n_trials, 1 / n_classes) / n_trials

The bound assumes trials are independent, which shuffled trials from one session are not. So it is the least you need, not the most. For unbalanced classes, use the majority-class rate as the null.

Permutation tests

You want to know whether two conditions differ in some measure: mean amplitude, accuracy, alpha power. Compute the difference. Then shuffle the condition labels, recompute the difference, and repeat ten thousand times. The fraction of shuffles whose difference is at least as large as yours is the p-value. No assumption about distributions, no formula to misremember, and it works for any statistic you can compute. For within-subject designs, shuffle within each participant (swap the two conditions’ labels per person).

import numpy as np
def perm_test(a, b, n=10000, rng=np.random.default_rng(0)):
    # a, b: paired measurements per participant
    d = a - b; obs = d.mean()
    signs = rng.choice([-1, 1], size=(n, len(d)))
    null = (signs * d).mean(axis=1)
    return (np.abs(null) >= abs(obs)).mean()

Cluster-based permutation for waveforms

Compute a t-statistic at every time point (and channel). Threshold it. Group neighbouring above-threshold points into clusters and give each cluster a mass (the sum of its t-values). Shuffle labels, repeat, and record the largest cluster mass in each shuffle. Your cluster’s p-value is the fraction of shuffles whose biggest cluster was bigger. This controls the false-positive rate across the whole waveform while exploiting the fact that real effects are extended in time and space. MNE implements it in one function. Report the cluster’s extent honestly: the test says a difference exists somewhere in the cluster, not that it starts or ends at the cluster’s edges.

Effect sizes

A p-value says whether an effect is distinguishable from zero. It does not say how big it is. Report the difference in the measure’s own units (2.3 µV) and a standardized effect size (Cohen’s d, the difference divided by the standard deviation across participants). A d of 0.2 is small, 0.5 medium, 0.8 large. Classic ERP effects are large; most “cognitive enhancement” claims are small at best, and a small effect in twelve people is noise.

The traps, named

Peeking and stopping. Collecting data until the p-value crosses 0.05 guarantees it will. Set N in advance.

Garden of forking paths. Trying several windows, channels, filters, and rejection thresholds and reporting the one that worked. Pre-register the analysis.

Double dipping. Choosing the channel or window where the effect looks biggest, then testing it there. The selection already used the data.

Correlation across trials treated as independence. Trials from one person are not independent samples of the population. Test across participants.

Confusing significance with size. A p of 0.001 with a d of 0.1 is a real, tiny effect, probably from a large N. Say so.