Triaging a Red Pipeline

A developer facing a red build asks one question first: is this me? How long that takes to answer is largely determined by decisions made when the pipeline was built.

The categories

My change broke it. The intended case.

A flaky test. Passes on retry, no code change.

Infrastructure. Runner died, network timeout, registry unavailable.

The main branch was already broken when the change branched from it.

A dependency changed outside the repository.

A pre-existing failure now surfacing because selection or ordering changed.

Six categories, and the response differs completely for each.

Making the category obvious

Distinguish infrastructure failures from test failures in the reporting. A runner timeout should not look like an assertion failure.

Surface the actual failure, not a log to scroll. The first screen should contain the failing test name and message.

Flag known flaky tests in the output. If the test is on the flake list, say so at the point of failure rather than making the developer discover it.

Show the main branch status alongside the failure, so "was it already broken" is answered without investigation.

Link to the previous run of the same test, so a developer can see whether it has been failing intermittently for a week.

The log problem

Logs are the default interface and they are a bad one. Thousands of lines, the relevant twenty somewhere in the middle.

Extract the failure into a structured summary: which test, which assertion, expected and actual.

Keep the full log available and do not make it the first thing.

Preserve artefacts — screenshots, dumps, generated files — and link them from the summary rather than requiring a separate hunt.

Reproducing locally

The thing developers most need and most often cannot do.

Print the exact command the pipeline ran, copy-pasteable.

Pin the environment, so the local run matches. Most "cannot reproduce" is a version difference.

Support running a single test without the surrounding pipeline.

Where the pipeline environment genuinely cannot be reproduced locally, that is a finding, and it is worth fixing rather than accepting: it means every failure in that stage requires a full pipeline cycle to investigate.

The retry question

Retry is the reflex and it hides the signal.

Make retries visible. A run that passed on the third attempt is not the same as one that passed, and the report should say so.

Count retries as a metric, which is the flake rate under another name.

Do not retry automatically without recording it, which is the configuration that makes a flaky suite look healthy.

Ownership of the broken main branch

Someone must be responsible for the main branch being green, or a break persists while everyone assumes someone else is on it.

A rota, or the person whose change broke it, with a stated expectation.

Revert as the default, not fix-forward, unless the fix is obvious and immediate.

Measure time to green, which is the number that says whether this works.

The copy-pasteable command

The single most useful thing a pipeline can print, and it is frequently absent.

Print the exact command that ran, with the flags, ready to paste.

Print the versions of the toolchain it ran with.

Print how to run just the failing test, not the whole stage.

Where the environment cannot be reproduced locally, say so explicitly and print what would be needed.

This turns a twenty-minute pipeline cycle into a thirty-second local run, which is the difference between investigating a failure and retrying it.

Owning the red main branch

Without an owner, a break persists while everyone assumes someone else is on it.

A rota, or the author of the breaking change, stated explicitly.

Revert as the default, with fix-forward reserved for cases where the fix is obvious and immediate.

A stated expectation for time to green.

Measure it, which is the number that says whether the arrangement works.

Announce breaks somewhere people see, because a break nobody knows about is a break everyone is about to build on.