Designing the Pipeline

Total pipeline duration is the number people quote. Time to a useful signal is the number that matters, and the two can differ by a factor of five.

Order by information per second

Run the cheapest, most discriminating checks first.

Lint and format: seconds, and they catch a real proportion of problems.

Compile: fast, and nothing downstream matters if it fails.

Unit tests: minutes.

Integration tests: longer.

End-to-end, performance, security scanning: longest.

A developer should learn about a syntax error in thirty seconds, not after a twelve-minute suite.

Fail fast, with a caveat

Stop the pipeline on the first failure to save compute and give the signal sooner.

The caveat: if lint fails, compile fails and three tests fail, a developer who sees only the lint failure fixes it, pushes, and waits again to find the next one.

Better: fail the stage but complete the parallel work within it. Report everything the pipeline learned before it stopped.

Cheap checks should not stop expensive ones from starting if they run in parallel anyway — the compute is already committed.

Parallel and sequential

Independent stages run in parallel. Lint does not need compile output.

The critical path is what matters, not the total work. Adding a parallel stage that takes two minutes to a pipeline whose longest path is ten changes nothing.

Compute the critical path and optimise that. Most pipelines have one stage that dominates and several that are irrelevant to duration.

Splitting blocking from non-blocking

The most useful structural decision available.

Blocking: what must pass before merge. Should be minutes.

Non-blocking: what runs after merge or on a schedule. Nightly end-to-end, performance, deep security scanning, full-suite runs.

Deciding which is which is a risk conversation, and it should be explicit rather than emerging from whatever was easy to configure.

A blocking pipeline that takes forty minutes will be worked around, and the workaround will be worse than the risk you were managing.

Queueing, separately

Queue time and execution time have different causes and different fixes.

Execution: the work itself. Fixed by caching, selection and parallelism.

Queue: insufficient capacity, or bursty demand. Fixed by capacity, scheduling or reducing runs.

Report them separately. A pipeline reported as fifteen minutes that is three minutes of work and twelve of waiting is a capacity problem, and every effort spent optimising the build is misdirected.

What to measure

Time to first useful signal, which is the headline.

Total blocking duration, median and ninetieth percentile.

Queue time, separately.

Critical path composition: which stages account for it.

Failure rate by stage, which shows which checks earn their place. A stage that has never failed is either redundant or not actually checking anything.

Retry rate, which is the flakiness signal in disguise.

Splitting blocking from the rest

The structural decision with the largest effect on developer experience, and it is a risk conversation rather than a technical one.

List every stage and what it protects against.

For each, ask what happens if it runs after merge instead. Frequently the honest answer is that the failure would be caught within the hour and reverted.

Move those. Nightly end-to-end suites, performance runs, deep security scans, full-suite verification.

Record the decision with the reasoning, because someone will ask why a check is not blocking.

Review it after any incident that a moved check would have caught.

Reporting everything learned

A design detail that decides whether a developer needs one cycle or three.

Fail the stage, but complete the parallel work within it.

A developer who sees only the lint failure fixes it, pushes, and waits again to discover the compile error.

Report every failure the pipeline found before it stopped.

Group them so the output is a list rather than a wall.

One cycle instead of three, which on a ten-minute pipeline is twenty minutes returned per occurrence.