Asking People, Properly

Instrumentation tells you a build took nine minutes. It cannot tell you that everyone has stopped running the tests locally because of it.

What only asking reveals

Workarounds. The steps people skip, the flags they pass, the parallel process they run because the sanctioned one is too slow.

Trust. Whether a red build is believed.

Friction that is not latency: a confusing error message, a tool that requires three commands, documentation that is wrong.

Where attention actually goes, which telemetry cannot see.

What people would fix first, which is frequently not what the dashboard suggests.

Writing questions that work

Specific and recent. "How long did you wait for CI feedback on your last change" beats "how satisfied are you with CI".

Behavioural, not attitudinal. "Did you run the full test suite locally before your last push" produces a fact; "do you value testing" produces a virtue.

About the system, not the person. "What slowed you down this week" invites a system answer; "how productive were you" invites defensiveness.

Short. Six questions, answered in three minutes. Longer surveys reduce both response rate and answer quality.

With one free text field, which carries most of the value.

The multidimensional point

The SPACE framework exists partly to argue that productivity cannot be reduced to one number, and that a balanced view needs several dimensions — satisfaction, performance, activity, communication, efficiency.

The practical version of that argument: any single measure will be optimised at the expense of the others. Deployment frequency alone produces trivial deployments; satisfaction alone produces comfortable stagnation.

Take two or three measures from different dimensions and read them together. That is what the framework is asking for, rather than a scorecard with five columns.

What breaks a survey

Attribution. The moment answers are identifiable, they become diplomatic. Aggregate, state the minimum group size, and hold to it.

No action afterwards. A survey that produces nothing visible reduces the next response rate permanently. This is the same failure as everywhere else in measurement, and it is the most common.

Too frequent. Quarterly is usually right. Monthly produces fatigue unless something visibly changes monthly.

Asking about things you cannot change, which teaches people the exercise is decorative.

Closing the loop

Report the results back, including the unflattering ones.

Name what you will act on, with an owner and a date.

Name what you will not act on and why, which is the part that buys credibility and the part organisations skip.

Report progress at the next survey, before asking the new questions.

Combining with telemetry

Use the survey to find where to look; use telemetry to size it.

Where they disagree, investigate. If people report builds as slow and the median is two minutes, look at the ninetieth percentile, at cold-cache runs, or at a specific repository. The perception is usually right about something the aggregate hides.

Perception lags reality. After a genuine improvement, reported satisfaction takes weeks to move. Do not conclude the change failed because the next survey looks the same.

Six questions

A survey that takes three minutes and produces answers you can act on.

How long did you wait for CI feedback on your last change?

Did you run the full test suite locally before your last push? If not, why not?

When a build failed last week, did you assume it was your change or the pipeline?

What did you have to look up that you should not have had to?

What one thing would you fix about the toolchain?

Anything else.

Five specific and behavioural, one open. The last one carries most of the value and the first five stop it being a complaints box.

When perception and telemetry disagree

A disagreement is a finding, not an error in one of them.

People report slow builds; the median is two minutes.

Check the ninetieth percentile, which is what they remember.

Check the cold-cache population, which may be most of their runs.

Check by repository, since one bad one colours the impression of all of them.

Check whether the complaint is latency at all — it is frequently a confusing error or a multi-step command, which no timing measure will show.

Perception is usually right about something the aggregate hides.