Adding a new Telemetry probe

In Firefox, the Telemetry system collects various measures of Firefox performance, hardware, usage and customizations and submit it to Mozilla. This article provides an overview of what is needed to add any new Telemetry data collection.

What is your goal?

We have various data collection tools available, each serving different needs. Before diving right into technical details, it is best to take a step back and consider what you need to achieve.

Your goal could be to answer product questions like “how many people use feature X?” or “what is the error rate of service Y?”.
You could also be focused more on answering engineering questions, say “which web features are most used?” or “how is the performance of engine Z?”.

From there, questions you should ask are:

• What is the minimum data that can answer your questions?
• How many people do you need this data from?
• Is data from the pre-release channels sufficient?

This also informs the data collection review, which requires a plan for how to use the data. Data collection review is required for all new data collection.

Data collection levels

Most of our data collection falls into one of two levels, opt-in and opt-out.

Opt-out data is recorded by default on all channels, users need to explicitly opt out to disable it. This has the strictest constraints for what data we can collect. "Most" users submit this data.

Opt-in data is not recorded by default on release, but it is enabled by default on our pre-release channels (Beta, Aurora and Nightly). This data is not useful on the release channel as few users have it enabled.

These levels cover what is described in the Firefox privacy notice. For other needs, there might be custom mechanisms that clearly require opt-in and show what data is collected.

Rich data & aggregate data

For the recording and transmission of data, we have various data types available. We can divide these data types into two large groups.

Aggregate data is aggregated on the client-side and cheap to send, process and analyze. This could e.g. be a simple count of tab opens or a histogram showing how long it takes to switch between tabs. This should be your default choice and is well supported in our analysis tools.

Rich data is used when questions can not be answered from aggregate data. When we send more detailed data we can e.g. see when a specific UI interaction happened and in which context.

As a general rule, you can inform the choice of data types from your goals like this:

Aggregate data

Most of our data collection happens through scalars and histograms:

• Scalars allow collection of simple values, like counts, booleans and strings.
• Histograms allow collection of multiple different values, but aggregate them into a number of buckets. Each bucket has a value range and a count of how many values we recorded.

Both scalars & histograms allow recording by keys. This allows for more flexible, two-level data collection.

Other collections can build on top of scalars & histograms. An example is use counters, which submit web feature usage through histograms.

We also collect environment data. This consists of mostly scalar values that capture the “working environment” a Firefox session lives in, and includes e.g. data on hardware, OS, add-ons and some settings. Any data that is part of the "working environment", or needs to split subsessions, should go into it.

Rich data

Aggregate data can tell you that something happened, but is usually lacking details about what exactly. When more details are needed, we can collect them using other tools that submit less efficient data. This usually means that we can't enable the data collection for all users, for cost and performance concerns.

There are multiple mechanisms to collect rich data:

Stack collection helps with e.g. diagnosing hangs. Stack data is recorded into chrome hangs and threadhang stats. To diagnose where rarely used code is called from, you can use stack capturing.

Event Telemetry provides a way to record both when and what happened. This enables e.g. funnel analysis for usage.

Custom pings are used when other existing data collection does not cover your need. Submitting a custom ping enables you to submit your own JSON package that will be delivered to the Telemetry servers. However, this loses you access to existing tooling and makes it harder to join your data with other sources.

Setup & building

Every build of Firefox has Telemetry enabled, but to actually submit data you need the following in your .mozconfig:

MOZ_TELEMETRY_REPORTING=1
MOZILLA_OFFICIAL=1

When adding any new scalar, histogram or event Firefox needs to be built. Artifact builds are currently not supported, even if code changes are limited to JavaScript.

Testing

Local confirmation

Your first step should always be to confirm your new data collection locally.

The about:telemetry page allows to view any data you submitted to Telemetry, whether it is in existing pings or in new custom pings.

If you need to confirm when - or if - pings are getting sent, you can run an instance of the gzipServer locally. It emulates roughly how the official Telemetry servers respond, and saves all received pings to disk for inspection.

Test coverage

Any data collection that you need to base decisions on needs to have test coverage. Using JS, you can access the recorded values for your data collection. For example:

• scalars or keyed scalars
• histograms or keyed histograms
• events

If you need to test that pings were correctly passed to Telemetry, you can use TelemetryArchiveTesting.

Validation

While it's important to confirm that the data collection works on your machine, the Firefox user population is very diverse. Before basing decisions on any new data, it should be validated. This could take various forms.

For new data collection using existing Telemetry data types, the transport mechanism is already tested. It is sufficient to validate the incoming values. This could happen through re:dash or custom analysis.

For new custom pings, you'll want to check schema validation results, as well as that the contents look valid.

Getting help

The Telemetry team is there to help with any problems. You can reach us via:

• IRC in #telemetry
• Slack in #fx-metrics
• the fx-data-dev mailing list
• flags for one of the peers on Bugzilla or direct e-mail