Flaky Tests

A suite that fails randomly a fifth of the time is not a safety net. It is a lottery that people learn to re-roll.

Why they matter more than speed

Retrying becomes the reflex. Once a red build is assumed to be noise, nobody investigates, and genuine failures reach the main branch.

The retry costs the full duration again, so flakiness and slowness multiply.

They erode the argument for testing with everyone who was already sceptical.

They hide real bugs. A test that is flaky because of a race condition is telling you about a race condition in the code, not in the test.

Reliability is worth more than speed at the margin, and it is measured far less often.

Detecting them

You cannot fix what you cannot name.

Record every test result, not just the build outcome. Test name, result, duration, commit, run identifier.

A flake is a test that failed and then passed on the same commit with no code change. That definition is mechanical and can be computed automatically.

Report a flake rate per test over a rolling window.

Rank by frequency times suite position — a flaky test early in a long suite wastes more than one at the end.

Without this data, flakiness is anecdote, and the conversation is about whose tests are worst rather than about which tests to fix.

The usual causes

Timing. Fixed sleeps, race conditions, assumptions about ordering in asynchronous code. The largest category by far.

Shared state between tests, which makes results order-dependent.

Real time. Tests that behave differently at midnight, at month end, or across a daylight saving change.

External dependencies: a network call, a real service, a public API.

Resource contention under parallelism: ports, temporary files, database rows.

Randomness without a fixed seed.

Test pollution, where one test leaves state that breaks a later one — and the failing test is not the guilty one.

Quarantine, used properly

A flaky test that blocks merges gets deleted or ignored. Quarantine is the alternative that keeps the information.

Move it out of the blocking suite but keep running it.

Record why and when, with an owner and a date.

Report the quarantine list as a standing metric. It should be small and it should shrink.

Cap it. Quarantine with no limit becomes a graveyard, and a graveyard is a slow deletion with extra steps.

Set an expiry: a quarantined test that is not fixed within a defined period is deleted, deliberately, with the coverage gap recorded.

Fixing them

Reproduce first. Run in a loop, under load, in random order, with a different seed.

Never fix by adding a retry. That converts a known problem into a hidden one.

Never fix by adding a sleep. That converts an intermittent failure into a slow intermittent failure.

Poll on a condition with a timeout instead of sleeping.

Fix the shared state rather than ordering around it.

Control time and randomness by injecting them.

Check whether the test is right. A meaningful proportion of flaky tests are correctly detecting a real race condition, and the fix belongs in the code.

The organisational part

Someone must own the flake rate, or it is everyone's problem and nobody's work.

Budget time for it, explicitly. Flake fixing never wins against feature work in an unstructured priority contest.

Report the rate weekly alongside build duration. It is the number that determines whether any of the rest matters.

The quarantine ledger

Quarantine works only if it is bounded, and the bound needs a record.

One row per quarantined test: name, date, reason, owner, expiry.

A cap on the total, agreed in advance.

An expiry date, after which the test is fixed or deleted with the coverage gap recorded.

Reported weekly alongside the flake rate.

Reviewed monthly, which is when expired entries get acted on.

Without the cap and the expiry it becomes a graveyard, and a graveyard is deletion with extra bookkeeping.

The retry that hides everything

The configuration that makes a flaky suite look healthy and removes the data needed to fix it.

Automatic retry without recording turns an intermittent failure into a silent slowdown.

The build goes green, the flake rate reads zero, and the suite gets slower every month.

Record every retry and treat a run that passed on the third attempt as distinct from one that passed.

Report retries as a metric, which is the flake rate under another name.

Retry is acceptable; unrecorded retry is not.