Cobalt: Telemetry with built-in privacy
Overview
Cobalt is a pipeline for collecting metrics data from user-owned devices in the field and producing aggregated reports.
Cobalt includes a suite of features for preserving user privacy and anonymity while giving product owners the data they need to improve their products.
- Local Differential Privacy. If you enable it, Cobalt can add random noise on the client during metrics collection.
- Shuffling. Our pipeline shuffles the collected records to break linkability. This also amplifies the level of differential privacy.
- No IDs. Cobalt does not use device IDs or user IDs; not even pseudonymous IDs.
- No exact timestamps. Cobalt uses day indices only.
- Only standard metric types. Not arbitrary protocol buffers.
- No ad-hoc, post-facto querying. Whereas on other telemetry systems standard practice is to “collect now; ask questions later”, with Cobalt the analyst must think in advance about what aggregations they wish to report and predeclare reports.
- Only aggregated data is released. It is not possible to query the raw collected data.
Cobalt Core
The code in the Git repo at https://fuchsia.googlesource.com/cobalt is known as Cobalt Core. It contains:
- Client-side libraries for:
- Logging metric events to Cobalt
- Local aggregation
- Privacy encoding
- Local data storage
- Upload orchestration
- Protocol buffers for the registry of Cobalt's projects, metrics and reports.
- Protocol buffers for data transport
- The registry parser
- A test application
In order to have a working Cobalt system you need to also have
- A server-side implementation of the Cobalt pipeline including a Shuffler and Analyzers (aka Report-Generators.)
- There is a server-side pipeline running at Google with code in the internal Google repository.
- As of this writing that is the only server-side implementation of Cobalt.
- An embedding of Cobalt Core into a client-side platform.
- This allows the code-to-be-measured to easily invoke Cobalt's Logger interface.
- There is an embedding into the Fuchsia operating system at //src/cobalt.
- If you have a Fuchsia checkout, you will find a copy of Cobalt Core has been pulled in at
//third_party/cobalt
. - As of this writing, Fuchsia is the only embedding of Cobalt Core.
This document describes how to develop, build and test Cobalt Core independently of Fuchsia.
Requirements
- We have only tested the stand-alone Cobalt Core build on Linux. You may or may not have success on other systems.
- Python version at least 2.7
- Git
Fetch the code
For example via
git clone https://fuchsia.googlesource.com/cobalt
cd cobalt
Run setup
./cobaltb.py setup
This will take a few minutes the first time. It does the following:
- Fetches third party code into the
third_party
dir via Git submodules. - Fetches the sysroot and puts it in the
sysroot
dir. This uses CIPD.
cobaltb.py
The Python script cobaltb.py
in the root directory is used to orchestrate building and testing Cobalt. It was already used above in ./cobaltb.py setup
.
./cobaltb.py -h
for general helpcobaltb.py <command> -h
for help on a command
Build
./cobaltb.py build
The Cobalt build uses GN and ninja.
Test
./cobaltb.py test
Formatting and Linting
./cobaltb.py fmt
./cobaltb.py lint
See: clang-tidy for an explanation of clang-tidy configuration options.
Contributing
Cobalt uses the Gerrit code review tool. Although Cobalt Core is independent of Fuchsia, we do share infrastructure with Fucshia and so much of the instructions from Fuchsia's Contributing Guide apply to Cobalt Core. The main difference is that Cobalt Core does not use Jiri--it uses Git submodules instead.
Generate a cookie
Follow the instructions here.
Create a change in Gerrit
To create a change in Gerrit do the following:
- Go to your Cobalt Core directory, for example:
cd ~/cobalt
- Create a new branch:
git checkout -b <branch_name>
- Create or edit files in the new branch.
- Add the updated files:
git add <files>
- Commit the updated files and write a change message.
git commit
- Upload the commit to Gerrit:
git push origin HEAD:refs/for/master
See the Gerrit documentation for more information.
Update a change in Gerrit
After creating a change, to upload a patch to your change, do the following:
- Create or edit files in the same branch.
- Add the updated files:
git add <files>
- Include the patch in the same commit using the
--amend
option: git commit --amend
- Upload the patch to Gerrit:
git push origin HEAD:refs/for/master