Queueing and Capacity

When a pipeline is reported as slow, the first question is how much of it was work. Frequently most of it was waiting for a machine.

Separating the two

Queue time: from a run being triggered to it starting.

Execution time: from starting to finishing.

Almost every CI system records both and reports the sum, which conflates a capacity problem with a workload problem and sends the improvement effort to the wrong place.

Split them first. Everything else in this note depends on it.

Why queues form

Insufficient capacity for the sustained load. The simple case.

Bursty demand, which is more common. Merges cluster around stand-ups, end of day and the hour before a release. Average utilisation looks comfortable and the peak is saturated.

Long-running jobs holding runners, so short jobs wait behind them.

Poor scheduling, where a large job waits for a big enough slot while small ones could have run.

Retries, which multiply demand exactly when things are already going badly.

Special runners — a particular operating system, architecture or licence — that are scarce and become a bottleneck invisible in the overall figures.

Reading the pattern

Plot queue time by hour of day and day of week. The shape says which problem you have.

Flat and high: capacity.

Spiky: burstiness, which is cheaper to fix by shifting demand than by adding machines.

High only for one runner type: a scarce resource, and the fix is specific.

Rising over weeks: demand growing past capacity, and the trend gives you time to plan.

Fixes, cheapest first

Reduce runs. Test selection, skipping unaffected pipelines, not running on documentation-only changes. Every run avoided is capacity returned.

Shorten jobs, which frees runners sooner. Caching helps queueing as well as execution.

Split long jobs, so they occupy a runner for less time.

Cancel superseded runs. If someone pushes three times, only the last needs to complete. This is a configuration setting and it is frequently off.

Schedule non-urgent work off-peak: nightly suites, scheduled scans.

Then add capacity, with autoscaling if the platform supports it, sized to the peak rather than the average.

The autoscaling caveat

Scaling up is not instant. A runner that takes three minutes to provision does not help a two-minute burst.

Warm pools cost money to keep idle and are the only thing that helps a spike.

Cold caches on new runners mean a freshly provisioned machine is slower than a warm one, so scaling out can raise execution time while lowering queue time.

Measure both after any scaling change, because improving one at the expense of the other is easy to do and easy to miss.

The compute bill

Queue time and cost pull in opposite directions. Idle capacity is expensive; insufficient capacity is expensive in a currency nobody invoices.

Which is why the reduction levers come first. Selection and caching lower both cost and queue time simultaneously, and capacity changes trade one for the other.

Report both numbers together, or the conversation becomes a choice between developer experience and the budget, which it need not be.

Cancelling superseded runs

A settings change that returns capacity immediately and is off by default in many systems.

If a developer pushes three times in ten minutes, only the last needs to complete.

Enable concurrency cancellation per branch or per pull request.

Exclude the main branch, where every commit should be verified.

Measure runs cancelled per week, which is capacity returned.

Expect a substantial proportion in an active repository. This is frequently the single cheapest queueing fix available.

The scaling trade-off

Adding capacity can improve one number and worsen another, which is easy to miss.

New runners start cold, with no caches.

Queue time falls; execution time rises.

Total time may not improve at all, and can get worse if the cache restore is slow.

Measure both before and after any scaling change, and report them separately.

Warm pools fix it and cost money to keep idle, which is a decision rather than an oversight.