Coverage, Used and Misused

Coverage measures which lines executed during a test run. That is all it measures, and almost every problem with it comes from expecting more.

What it tells you

Which code was not executed at all. This is genuinely useful: uncovered code is untested by definition.

Where to look. A module at very low coverage is worth a conversation.

Whether a new change is tested, which is the most useful application — coverage on the diff rather than on the codebase.

What it does not tell you

Whether the tests assert anything. A test that calls a function and asserts nothing produces full coverage of it.

Whether the behaviour is correct.

Whether the edge cases are covered. Line coverage says the line ran, not that it ran with the interesting input.

Whether the tests would catch a regression.

High coverage is consistent with a suite that catches nothing, and that is not a hypothetical failure mode — it is what a coverage target reliably produces.

What a target does

It becomes the goal. Once eighty percent is required to merge, the work becomes reaching eighty percent.

The cheapest route is tests without assertions, or tests of trivial code — getters, generated classes, configuration.

The expensive, valuable tests are unaffected, because they were already there.

So the number rises and the suite gets no better, which is the predictable outcome and the reason the practice persists: the metric moves.

It also punishes deletion. Removing dead code can lower coverage, which under a threshold makes cleanup a fight.

Using it well

Report it; do not gate on it. Visibility produces attention without producing gaming.

Measure it on the change, not the codebase. "Is this new code tested" is a good review question; "is the repository at eighty percent" is not.

Look at the uncovered list, which is where the information is. The percentage is a summary of it and less useful.

Segment by importance. Payment handling at sixty percent is a different situation from a formatting utility at sixty percent, and one number cannot say so.

Use branch coverage over line coverage where the tool supports it, since it at least notices untested conditions.

Better questions

What has never been executed by any test?

Which recent incidents had no test that would have caught them? This is the highest-value review available and it needs no coverage tooling.

Which tests have never failed? A test that has passed every run since it was written may be asserting nothing.

Would the suite catch a deliberately introduced bug? Mutation testing answers this properly, at a cost in run time, and is worth doing occasionally on the code that matters most.

The cost side

Coverage instrumentation slows tests, sometimes substantially.

Collect it on a schedule rather than on every run — nightly, or on the main branch — and keep the pull request loop uninstrumented.

Which is also an argument against gating on it: a gate requires collecting it every time, on the loop where speed matters most.

Coverage on the change

The one application of coverage that is useful rather than harmful.

Measure coverage of the lines the change touched, not of the codebase.

Report it in the review, as information rather than as a gate.

The question it answers is good: is this new code exercised by anything?

The question a repository-wide percentage answers is not: it summarises history and moves too slowly to inform any decision.

It also avoids the cost problem, since instrumenting one changed file is cheap where instrumenting every run is not.

Mutation testing, occasionally

The technique that answers the question coverage cannot.

It introduces deliberate small changes to the code and checks whether any test fails.

A surviving mutation means no test detected the change, which is the real question: would this suite catch a regression.

Expensive to run, which is why it belongs on a schedule and on the code that matters most rather than on everything.

Run it once on your most critical module. The result is usually sobering and it is a much better argument for test quality than any coverage percentage.