Stopping It Coming Back

Every improvement in this field is temporary unless something defends it. A build that was made twice as fast will be back where it started within a year.

How it decays

A dependency added, widening the graph.

A task made less incremental by someone declaring an input too broadly to fix a stale-output bug.

A test added that boots the application.

A pipeline stage added for a good reason, on the blocking path.

A cache key changed, silently dropping the hit rate.

Modules merged for convenience, collapsing the graph.

Each is small and defensible. The sum is the problem, and nobody sees the sum without a measurement.

Catching it

A scheduled benchmark. Same workload, same machine class, on a schedule, recorded.

Alert on threshold crossing, not on every fluctuation.

Attribute promptly. A regression identified within a week can be traced to a commit; one found after a quarter cannot.

Track the composition, not just the total. A total that holds while one stage doubles and another halves is hiding two findings.

Checks in review

Cheaper than the benchmark and catches things earlier.

Flag new dependencies between modules, particularly into shared ones.

Flag new blocking pipeline stages, so adding one is a decision rather than a commit.

Report test duration on the change, so a slow new test is visible at review rather than a month later.

Fail on a cycle in the dependency graph.

Warn on cache key changes, which are frequently accidental.

None of these blocks a change. They make the cost visible at the moment it is being incurred, which is when it is cheapest to reconsider.

Budgets

A stated ceiling: blocking pipeline under ten minutes, incremental build under thirty seconds, flake rate under a threshold.

Reported against, weekly.

Crossing it triggers work, not an exception.

Set the budget from what the team needs, not from where you currently are, or you have ratified the status quo.

Ownership

Someone must watch the trend, or the alert fires into an unwatched channel.

A named person, weekly, five minutes.

With authority to raise the regression as work, and a forum to raise it in.

Without that, the benchmark becomes a chart nobody reads, and the decay proceeds exactly as it would have without it.

Reviewing the review checks

Automated checks in review accumulate and stop being read, like every other alerting system.

Count how often each fires.

Anything firing on most changes is noise and will be clicked through.

Anything that has never fired is either misconfigured or protecting against something that does not happen.

Remove both categories, annually.

Keep the ones with a finding rate in between, which are the ones people still read.

The ratchet

The mechanism that distinguishes a lasting improvement from a good quarter.

When a measure improves, lower the budget to the new level.

Otherwise the gain is spent on the next twelve small regressions, each individually defensible, and the number returns to the old ceiling within a year.

It is unpopular, because it converts headroom into a constraint.

Do it anyway, and say why: without it the improvement was a loan rather than a gain.