What to Measure

The measurement problem in this field is not a shortage of numbers. It is that the available numbers mostly describe activity rather than experience.

The measures worth keeping

Time from push to first useful signal. Not total pipeline duration — the point at which a developer learns something actionable.

Local incremental build duration, as a distribution rather than a mean.

Local test run duration for a focused subset.

Cache hit rate, local and remote.

Pipeline queue time, separated from execution time. These have completely different causes and are routinely reported together.

Flaky failure rate: proportion of failures that pass on retry with no code change.

Time to green after a failure, which measures how quickly a broken state gets fixed.

Onboarding time: clean machine to running application, measured by actually doing it.

Eight measures. Each points at a specific cause when it moves.

The measures that mislead

Total pipeline duration without the queue split, which conflates a capacity problem with a workload problem.

Average build time, which hides the tail. The ninetieth percentile is what people remember.

Lines of code, commits, pull requests per developer. These measure output volume, correlate poorly with value, and are the subject of their own warning note.

Test count. More tests is not better; faster and more reliable is better.

Coverage percentage as a target, which produces tests written to touch lines rather than to catch bugs.

Tool adoption, which measures rollout rather than effect.

Distributions, not averages

Worth stating separately because it is the most common analytical error here.

Build times are heavily skewed. Most runs hit a warm cache and are fast; some hit a cold one and are slow.

The mean sits between the two and describes neither.

Report the median and the ninetieth percentile, always. The gap between them is itself a finding: a wide gap usually means cache misses, and cache misses have identifiable causes.

Instrumenting it

Build tools can emit timings — most modern ones support this natively or through a plugin.

CI systems already have the data, usually under-used, and the queue-versus-execution split is often available and not reported.

Local telemetry needs deliberate collection, and it needs the boundary stated before it starts: aggregate, about builds not people, with the retention and purpose written down.

Onboarding time needs a human. Take a clean machine, follow the documentation, and time it. Nothing else measures this and every organisation's estimate is wrong.

Reporting

One page, weekly. Eight numbers with their trend and normal range.

Segment by repository or by team, because the aggregate hides the one place where it is bad.

Never segment by individual, which is the boundary this whole field has to hold.

Attach one sentence per number saying what moved and why. Numbers without interpretation stop being read within a month.

Reading a bimodal distribution

Build durations are almost never normally distributed, and treating them as though they were is the most common analytical error here.

Plot the histogram, not the average.

Two peaks is the usual shape: warm cache runs and cold ones.

The gap between them is the cost of a cache miss, which is the number that justifies caching work.

The size of the second peak is the miss rate.

A single mode that is slow means caching is not working at all, which is a different finding and a different fix.

Segmenting by repository

The aggregate almost always hides the one place where it is bad.

Report every measure per repository, not organisation-wide.

Rank by developers affected times duration, which identifies where attention is worth spending.

Look for the outlier, which usually has one specific cause rather than being generally slow.

Compare similar repositories to find practices worth copying — the useful output of a comparison, as opposed to a ranking.