Designing an experiment that can fail
Hypotheses, counterbalancing, jitter, triggers, sample size, pre-registration, consent, and the pilot. The parts of a human experiment that decide whether the result means anything, before any data exist.
On this page12 min
An experiment is a question arranged so that the world can answer no. Most of what goes wrong in student experiments goes wrong before the first participant arrives: the question was vague, the conditions differed in more than one way, the timing was periodic, the sample was too small, and the analysis was chosen after seeing the data. Each of those has a fix that costs an hour. Here they are.
- Week beforeOne question, stated so it can be wrongIf no outcome would change your mind, it is not a question yet.
- Week beforeConditions that differ in one thingAnything else that differs between them is an alternative explanation you handed to your reviewer.
- Week beforeRandomize, counterbalance, jitterOrder effects and periodic timing both produce results that look real and are not.
- Week beforeSample size, from an expected effectChosen before, not after. Underpowered studies find effects only when they exaggerate them.
- Week beforePre-register the analysisA dated file with the question, the conditions, N, and how you will analyse it. Even a Git commit counts.
- Day beforePilot on yourself and one friendTriggers, timing, comfort, and the length of the session. Never on a real participant first.
- Then, and only thenCollectAnd run the analysis you wrote down, first, before any other.
One question, stated so it can be wrong
“Does face processing produce an N170N170 A negative deflection about 170 milliseconds after seeing a face, over the back of the head on the right. Related: Event-related potential (ERP)Event-related potential (ERP) The small, stereotyped voltage waveform the brain produces in response to an event, invisible in a single trial and revealed by averaging many. . Glossary entry” is not a question. “Mean amplitude at P8 from 150 to 190 ms is more negative for faces than for houses, by at least 1 µV, in 16 participants” is. It names the measure, the window, the channel, the direction, the size, and the sample. Write it down before collecting anything. If the result comes out the other way, you have learned something; if you wrote nothing down, you will find a way to explain whatever happened.
Conditions that differ in one thing
Faces versus houses differ in what they depict. They also differ, unless you work at it, in brightness, contrast, spatial frequency, size, and how interesting they are. Any of those could produce a difference at 170 ms. Match the images on everything you can measure, and randomize what you cannot. This is the whole art of stimulus design and it is where most of the time goes.
Randomize and counterbalance
Trial order random, so that nothing about position in the session (fatigue, learning, electrodeElectrode The conductor that turns ion currents in tissue into electron currents in a wire; its chemistry and contact quality set the noise you live with. Related: Silver/silver-chloride (Ag/AgCl)Silver/silver-chloride (Ag/AgCl) The standard electrode material for biopotentials, because its chemistry produces a stable, low-noise, low-drift interface with salty gel. , Electrode impedanceElectrode impedance How hard it is to push current through the skin-electrode contact, typically 5 to 50 kilohms with gel and megohms dry, higher at low frequencies. , Half-cell potentialHalf-cell potential The DC voltage that appears wherever a metal meets an electrolyte; two electrodes with different half-cell potentials produce an offset that can swamp the signal. . Glossary entry drying) lines up with a condition. Response mappings (left key for faces) swapped across participants, so a motor difference does not masquerade as a perceptual one. Block order counterbalanced when blocks are unavoidable.
Jitter the timing
A fixed inter-stimulus interval lets the participant predict the next stimulus, which produces its own ERPEvent-related potential (ERP) The small, stereotyped voltage waveform the brain produces in response to an event, invisible in a single trial and revealed by averaging many. Related: P300P300 A positive bump about 300 milliseconds after a rare or meaningful stimulus, the basis of the classic BCI speller. , N170N170 A negative deflection about 170 milliseconds after seeing a face, over the back of the head on the right. , Error-related negativity (ERN)Error-related negativity (ERN) A sharp negative wave about 50 to 100 milliseconds after you make a mistake, before you consciously notice. , Trial averagingTrial averaging Adding many time-locked epochs together so the consistent response grows and the random noise shrinks by the square root of the number of trials. . Glossary entry (the contingent negative variation) and lets rhythms entrain. It also lets any periodic artifactArtifact Anything in the recording that did not come from the brain: blinks, muscle, heartbeat, movement, sweat, mains hum, electrode pops. Related: Blink artifactBlink artifact The large, slow deflection an eye blink produces in frontal channels, often ten times the size of the brain signal. , Independent component analysis (ICA)Independent component analysis (ICA) A method that unmixes multichannel data into statistically independent sources, so blinks and heartbeat can be identified and removed. , Artifact subspace reconstruction (ASR)Artifact subspace reconstruction (ASR) An automatic method that detects short high-variance bursts and reconstructs them from clean data; common in real-time pipelines. . Glossary entry (heartbeat, alpha, mains) align with the stimulus and survive averaging. Draw each interval from a range: 1000 to 1500 ms, uniformly.
Triggers you can trust
Markers pushed at the flip, a photodiode check once, LSLLab Streaming Layer (LSL) A protocol that timestamps and synchronizes streams from many devices and stimulus programs on a network; every BCI lab uses it. Related: PsychoPyPsychoPy Free Python software for presenting stimuli with precise timing and logging responses. , BrainFlowBrainFlow A library that talks to OpenBCI and most consumer EEG headsets with one API, in Python and other languages. . Glossary entry for everything. The infrastructure page covers it. A 30 ms systematic offset is invisible in a single study and a disaster when you compare latencies to the literature.
Your pilot participant shows a beautiful effect. Should you run the full study with the same analysis?
Use the pilot to fix the procedure and fix the analysis in writing, then run the study without the pilot’s data. The pilot’s job is to find broken timing, confusing instructions, and an analysis window that makes sense. Once you have looked at pilot data and chosen windows, that data cannot be part of the test, or you have chosen the analysis to fit it.
Sample size
Effect sizes for classic ERPsEvent-related potential (ERP) The small, stereotyped voltage waveform the brain produces in response to an event, invisible in a single trial and revealed by averaging many. Related: P300P300 A positive bump about 300 milliseconds after a rare or meaningful stimulus, the basis of the classic BCI speller. , N170N170 A negative deflection about 170 milliseconds after seeing a face, over the back of the head on the right. , Error-related negativity (ERN)Error-related negativity (ERN) A sharp negative wave about 50 to 100 milliseconds after you make a mistake, before you consciously notice. , Trial averagingTrial averaging Adding many time-locked epochs together so the consistent response grows and the random noise shrinks by the square root of the number of trials. . Glossary entry are known: the N170N170 A negative deflection about 170 milliseconds after seeing a face, over the back of the head on the right. Related: Event-related potential (ERP)Event-related potential (ERP) The small, stereotyped voltage waveform the brain produces in response to an event, invisible in a single trial and revealed by averaging many. . Glossary entry face effect is large (a Cohen’s d around 1), so twelve to sixteen participants give good power. The P300P300 A positive bump about 300 milliseconds after a rare or meaningful stimulus, the basis of the classic BCI speller. Related: Event-related potential (ERP)Event-related potential (ERP) The small, stereotyped voltage waveform the brain produces in response to an event, invisible in a single trial and revealed by averaging many. , Oddball paradigmOddball paradigm An experiment in which a rare target is mixed in among frequent standards; the rare one produces a P300. . Glossary entry oddball effect is larger still. New effects have unknown sizes, and the honest move is a pilot to estimate it and a power calculation to set N. A study with six participants that finds nothing has found nothing; a study with six that finds something has probably found noise.
Pre-register
Write the hypothesis, conditions, exclusion rules, measure, window, channel, and test, in a document with a date, before collecting the real data. Put it in your repository. It costs an hour and it converts “we found” into “we predicted and found,” which is a different kind of claim. Labs increasingly require it; do it before they ask.
Consent, done as a conversation
The IRBInstitutional review board (IRB) The committee that must approve any research on people before it starts, to protect the participants. Related: Informed consentInformed consent A participant's agreement to take part after understanding the risks, in language they can follow; a process, not a signature., CITI trainingCITI training The online human-subjects research course most universities require before you can touch participant data.. Glossary entry approves a consent form; the consent is the conversation. Explain what the electrodesElectrode The conductor that turns ion currents in tissue into electron currents in a wire; its chemistry and contact quality set the noise you live with. Related: Silver/silver-chloride (Ag/AgCl)Silver/silver-chloride (Ag/AgCl) The standard electrode material for biopotentials, because its chemistry produces a stable, low-noise, low-drift interface with salty gel. , Electrode impedanceElectrode impedance How hard it is to push current through the skin-electrode contact, typically 5 to 50 kilohms with gel and megohms dry, higher at low frequencies. , Half-cell potentialHalf-cell potential The DC voltage that appears wherever a metal meets an electrolyte; two electrodes with different half-cell potentials produce an offset that can swamp the signal. . Glossary entry do and do not do (record, never stimulate), how long it takes, that they can stop any time without explanation, what happens to the data, and who to contact. Ask if they have questions. Watch for the person who is uneasy and has not said so; offer the exit again. Then the signature. A participant who understood is a participant whose data you can use with a clear conscience.
The pilot
Run yourself, then one friend, through the whole thing, timing included. Check the marker offsets, the epochEpoch A fixed-length slice of the recording locked to an event, such as one second starting 200 milliseconds before each stimulus. Related: Trial averagingTrial averaging Adding many time-locked epochs together so the consistent response grows and the random noise shrinks by the square root of the number of trials. , Baseline correctionBaseline correction Subtracting the mean of the pre-stimulus interval from each epoch so trials line up. . Glossary entry counts, the number of rejected trials, and whether the instructions were understood. Fix everything. Then freeze the protocol and begin.
Deep dive Exclusion rules, written first 2 min
Which participants will you exclude (too few clean trials, did not follow instructions, technical failure) and which trials (amplitude threshold, response too fast or too slow)? Decide the thresholds before the data exist. Report how many were excluded and why. An exclusion rule invented after looking at results is a way of choosing the result.
Deep dive Within-subject versus between-subject designs 2 min
Every participant seeing every condition (within-subject) is far more powerful, because the huge differences between people cancel in the comparison. Almost every EEGElectroencephalography (EEG) Recording the brain's electrical activity from electrodes on the scalp, a few tens of microvolts of blurred, summed cortical activity. Related: Electrocorticography (ECoG)Electrocorticography (ECoG) Recording from electrodes placed on the surface of the brain under the skull, usually during epilepsy surgery evaluation; cleaner and sharper than EEG. , Magnetoencephalography (MEG)Magnetoencephalography (MEG) Recording the tiny magnetic fields produced by cortical currents with superconducting sensors in a shielded room; less blurred by the skull than EEG, and far more expensive. , Volume conductionVolume conduction The spreading of currents from a source through the conductive tissues of the head, which is why one source shows up on many electrodes and why scalp EEG is blurry. , Microvolt (µV)Microvolt (µV) One millionth of a volt. Scalp EEG is tens of microvolts; a AA battery is 1.5 million of them. . Glossary entry study is within-subject for this reason. Between-subject designs (one group gets feedback, another does not) need several times the participants and are unavoidable only when the manipulation cannot be undone.
Explain what this page was about to your roommate in three sentences. No jargon they would not know.
We write one testable claim before we record, naming the site and the window such as 150 to 190 ms at the right back site with a size near 1 microvolt and a headcount near 16 people. We break predictability by drawing gaps from 1000 to 1500 ms and by shuffling trial order and response hands across people, with marker delay checked once by light sensor to catch a 30 ms offset. We file the dated plan before recording and run a pilot to freeze the steps while leaving its data out of the final test.
Yours does not have to match. Check that the three ideas are there.