Benchmarking Against Yourself

Published benchmarks are the most requested and least useful comparison in this field. Your own history is more informative and much harder to argue with.

Why external benchmarks mislead

Codebase size and language dominate. A build time that is excellent for a large compiled codebase is poor for a small interpreted one.

Deployment risk differs. A regulated environment cannot deploy at the frequency of a consumer product, and that is not a maturity gap.

Definitions differ. Two organisations reporting lead time are frequently measuring from different points.

Reporting is self-selected. Organisations with poor numbers do not publish them.

Which means a comparison tells you that you are different, not that you are worse, and the distinction matters when the number is being used to justify investment.

What a self-benchmark needs

A stable measure definition, written down and unchanged.

A representative workload, not a hand-picked fast case.

A long enough baseline to contain the normal variation — several weeks at minimum.

The distribution, not the mean.

The conditions recorded: codebase size, runner type, cache state.

Detecting regression

The main practical use, and the one most often missing.

Build performance decays continuously — a dependency added, a task made less incremental, a test that got slower. Each change is small; the sum is a build that doubled over a year and nobody noticed.

A scheduled benchmark run catches it. Same workload, same machine type, on a schedule, recorded.

Alert on a threshold crossing, not on every fluctuation.

Investigate promptly, because attributing a regression is easy within a week of the commit that caused it and very hard after a quarter.

Making the comparison fair

Same machine class. Comparing a run on a fast runner to one on a slow runner measures the runner.

Same cache state. Warm and cold runs are different populations and should be reported separately.

Same workload. A benchmark that drifts as the codebase grows is measuring growth, which is legitimate if you say so and misleading if you do not.

Record what changed, so a step in the series can be attributed rather than guessed at.

Where cross-team comparison is legitimate

Inside one organisation, carefully.

Comparing repositories to find practices worth copying is useful. "Why is that team's build a third of ours" is a good question.

Comparing to rank teams is the same error as ranking individuals, with the same consequences: defensive reporting, gamed definitions, and a loss of the cooperation the data depends on.

Lead with the mechanism, not the league table. "That repository has test selection configured and we do not" is the useful output.

Reporting it

Your own trend, over months.

The distribution, with median and ninetieth percentile.

Annotated with changes, so steps in the line have explanations.

No external comparison, unless someone insists — and then with the caveats attached in the report itself rather than in conversation, because the report is what circulates.

The scheduled benchmark

The mechanism that catches decay while it is still attributable.

A fixed workload, unchanged, on a fixed machine class.

On a schedule — nightly is usually right.

Warm and cold cache runs recorded separately.

Results stored with the commit, so a step in the series can be traced.

Alert on a threshold crossing, not on fluctuation.

Refresh the workload rarely and annotate when you do, because a changed benchmark and a regression look identical on the chart.

Annotating the series

A chart with unexplained steps invites the wrong conclusions.

Mark every deliberate change: a cache added, a runner class changed, a module split, a definition altered.

Mark the workload refreshes, which look identical to regressions.

Keep the annotations in the data, not in someone's memory.

A step with an annotation is a result; an unmarked one is an argument, and six months later nobody can settle it.