Seeing Inside the Pipeline

A pipeline that reports only a result and a duration cannot be improved except by guessing. The instrumentation is straightforward and rarely done.

What to collect

Per-run: trigger, commit, branch, result, queue time, execution time, runner type.

Per-stage: duration and result.

Per-step within a stage, where the system supports it.

Per-test: name, result, duration. This is the one most often missing and the one that makes flake detection possible.

Cache events: hit or miss, restore duration, size.

Retries, flagged as such rather than recorded as a fresh run.

Store it somewhere queryable. The CI system's own retention is usually short and its query capability limited.

The questions it answers

Where does the time go? Stage durations over a month, ranked.

What is the critical path? Which stages actually determine total duration.

Which tests are flaky? Failures that pass on retry at the same commit.

Which failures are real? Failure rate by stage, split by whether a retry succeeded.

Is it getting worse? Duration trend, which drifts upward without anyone noticing until it is unbearable.

Where is capacity short? Queue time by runner type and hour.

The trap of the average

Build durations are bimodal. Warm cache runs are fast, cold ones are slow, and the mean describes neither.

Report the median and the ninetieth percentile, always.

The gap between them is diagnostic: a wide gap means inconsistent caching, and the cause of the second mode is the finding.

People remember the tail, which is why the ninetieth percentile predicts complaints better than the median.

Attribution, carefully

Pipeline data contains who triggered each run, which is the point at which this becomes personal data.

Aggregate at ingestion. By repository, by branch, by stage — not by person.

There is no pipeline question that requires per-developer attribution. Whose build is slowest is not a useful question; which stage is slowest is.

State this before you start collecting, because a telemetry system that developers believe is aimed at them will be resisted, and local build instrumentation depends on that trust.

A weekly report

Time to first signal: median and ninetieth percentile.

Blocking pipeline duration, same two.

Queue time, separately.

Flake rate.

Cache hit rate.

Failure rate by stage.

One page, with the trend and a sentence on each. A dashboard with thirty tiles is read once.

Alerting, sparingly

On the main branch being red for longer than a threshold.

On duration crossing a level that changes behaviour — the point at which people start batching.

On flake rate rising.

Not on individual failures, which are normal and which generate the noise that makes people ignore the channel.

Storing it outside the CI system

CI platforms have short retention and limited query capability, which is enough for debugging one run and not for finding a trend.

Export runs, stages, tests and cache events to somewhere queryable.

Retain for at least a year, because the useful questions are about drift over quarters.

Aggregate at ingestion, dropping the identifiers that resolve to a person.

Keep the raw stage and test records, because the questions change and they cannot be reconstructed from summaries.

This is a modest amount of infrastructure and it is what makes every other note in this section possible.

Alerting sparingly

A channel that fires constantly is a channel nobody reads, which is the same failure as everywhere else.

Alert on the main branch red beyond a threshold.

Alert on duration crossing the level at which people start batching.

Alert on flake rate rising.

Do not alert on individual failures, which are normal.

Measure the alert volume and treat a high one as a defect in the rules rather than in the people ignoring them.