Collecting Telemetry Without Losing Trust

Local build instrumentation is the most valuable data in this field and the easiest to lose. It depends entirely on developers not believing it is aimed at them.

What the data actually contains

Worth being honest about, because "it is just build times" is not accurate.

When builds ran, which describes working hours.

How often, which describes activity.

What was built, which describes what someone was working on.

Which machine, which identifies a person.

Duration and failures, which is the part you actually want.

The first four are incidental and they are what make this sensitive.

What to collect

Task durations.

Cache hit and miss, with the reason.

Failure type, not failure content.

Toolchain and platform, which explains variance.

Repository and module.

That is enough to answer every question in these notes.

What to discard at source

User and machine identifiers. Aggregate before transmission, not after.

Timestamps beyond the granularity you need. Hour-of-day buckets answer capacity questions; second-precision timestamps reconstruct a working day.

File paths, which leak branch names, feature names and occasionally more.

Command lines, which can contain credentials.

Anything you would not want to explain, which is the practical test.

Aggregating at source

The design decision that makes the policy real rather than a promise.

Compute the metrics on the developer's machine and send those.

Send counts and durations, not events.

Bucket the time.

Include no identifier that resolves to a person.

Then test it: try to reconstruct one person's day from what was stored. If you can, the aggregation is nominal.

What to say before you start

Exactly what is collected, as a list.

Exactly what is not.

What it is used for, narrowly.

What it will never be used for — individual performance, in writing.

Who can access it.

How long it is kept.

Make it opt-out at minimum, and say what happens to the data if someone opts out.

Publish the collection code if you can. It is the only assurance that is verifiable rather than promised.

The first hard case

Someone will ask for per-developer numbers. Usually a manager, usually for a defensible reason.

Decide who decides before it happens, because deciding under pressure gets it wrong.

If the answer is yes, the data degrades within a quarter — people disable the agent, and the trust required for the next instrumentation is gone.

If the answer is no and the reason is stated publicly, the assurance becomes credible in a way no policy document achieves.

Answer the underlying question with system data, which usually can: "this team's builds are twice the median, and here is why".

The reciprocal

Give the data back. Developers should be able to see their own build performance and the team aggregate.

That converts telemetry from surveillance into a tool, which is the same principle as everywhere else in this collection.

It also improves the data, because people who can see the output care whether the input is right.

The reconstruction test

The check that tells you whether your aggregation is real or nominal.

Take the stored data.

Try to reconstruct one person's working day: when they started, what they worked on, how long they took.

If you can, the aggregation is decorative whatever the policy says.

Fix at the collection point, not by adding an access control on top.

Do it before the first collection, and again after any schema change, because a field added for a good reason frequently reintroduces the capability.

Giving the data back

The move that converts telemetry from surveillance into a tool.

Let developers see their own build performance and the team aggregate.

Show the comparison without a ranking: your median against the team median, not a position.

No commentary from the system, which is a warning with a friendly interface.

Provide a route to say the data is wrong, with a response.

People who can see the output care whether the input is right, which improves the quality of everything the programme depends on.