README.md - cobalt - Git at Google

 # Cobalt: Telemetry with built-in privacy

 [TOC]

 ## Overview

 Cobalt is a pipeline for collecting metrics data from user-owned devices in the
 field and producing aggregated reports.

 Cobalt includes a suite of features for preserving user privacy and anonymity
 while giving product owners the data they need to improve their products.

 - **Differential Privacy.** If you enable it, Cobalt can add random
 noise on the client during metrics collection.
 - **Shuffling.** Our pipeline shuffles the collected records to break
 linkability. This also amplifies the level of differential privacy.
 - **No IDs.** Cobalt does not use device IDs or user IDs; not even pseudonymous
 IDs.
 - **No exact timestamps.** Cobalt uses day indices only.
 - **Only standard metric types.** Not arbitrary protocol buffers.
 - **No ad-hoc, post-facto querying.** Whereas on other telemetry systems
 standard practice is to *"collect now; ask questions later"*, with Cobalt the
 analyst must think in advance about what aggregations they wish to report and
 predeclare reports.
 - **Only aggregated data is released.** It is not possible to query the raw
 collected data.

 ## Cobalt Core

 The code in the Git repo at
 [https://fuchsia.googlesource.com/cobalt](https://fuchsia.googlesource.com/cobalt)
 is known as **Cobalt Core**. It contains:

 - Client-side libraries for:
   - Logging metric events to Cobalt
   - Local aggregation
   - Privacy encoding
   - Local data storage
   - Upload orchestration
   - Protocol buffers for the registry of Cobalt's projects, metrics and reports.
   - Protocol buffers for data transport
   - The registry parser

 In order to have a working Cobalt system you need to also have
 - A server-side implementation of the Cobalt pipeline including a *Shuffler*
 and *Analyzers* (aka *Report-Generators*.)
   - There is a server-side pipeline running at Google with code in the internal Google
     repository.
   - As of this writing that is the only server-side implementation of Cobalt.
 - An *embedding* of Cobalt Core into a client-side platform.
   - This allows the code-to-be-measured to easily invoke Cobalt's *Logger* interface.
   - There is an embedding into the *Fuchsia* operating system at
     [//src/cobalt](https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/src/cobalt/).
   - If you have a Fuchsia checkout, you will find a copy of Cobalt Core has
     been pulled in at `//third_party/cobalt`.
   - As of this writing, Fuchsia is the only embedding of Cobalt Core.

 This document describes how to develop, build and test Cobalt Core independently
 of Fuchsia, as well as from in the Fuchsia tree.

 ## Requirements

   1. We have only tested the stand-alone Cobalt Core build on Linux. You may or
   may not have success on other systems.
   2. Python version at least 2.7
   3. Git

 ## Fetch the code

 To download just the Cobalt Core repo:

 - `git clone https://fuchsia.googlesource.com/cobalt`
 - `cd cobalt`

 To download the entire Fuchsia source code, including Cobalt Core, follow the
 [instructions](https://fuchsia.dev/fuchsia-src/get-started/get_fuchsia_source).
 Then change into the Cobalt Core directory:

 - `cd third_party/cobalt`

 ## Run setup

  ```./cobaltb.py setup```

 This command is required after a first checkout, before running any other
 `cobaltb.py` commands, and will take a few minutes the first time. It does
 the following:

   * Fetches third party code into Cobalt's `third_party` dir via Git submodules.
   * Fetches the sysroot and puts it in Cobalt's `sysroot` dir. This uses
   [CIPD](https://chromium.googlesource.com/chromium/src.git/+/main/docs/cipd.md).

 ### In the Fuchsia Tree

 If the `./cobaltb.py setup` command is run in the `third_party/cobalt` checkout
 of Cobalt Core in a Fuchsia tree, the Git submodules may interfere with the
 correct functioning of Jiri to update the repos in the Fuchsia tree. If
 problems occur, the submodules can be removed with: `./cobaltb.py deinit`.
 Before running any other `cobaltb.py` commands in this checkout,
 `./cobaltb.py setup` will need to be run again to refetch the submodules.

 ## cobaltb.py

 The Python script `cobaltb.py` in the root directory is used to orchestrate
 building and testing Cobalt. It was already used above in `./cobaltb.py setup`.

 * `./cobaltb.py -h` for general help
 * `cobaltb.py <command> -h` for help on a command

 ## Build

   ```./cobaltb.py build```

 The Cobalt build uses GN and ninja.

 ## Test

   ```./cobaltb.py test```

 This runs the tests from the currently built code (`build` must be run first).

 ## Formatting and Linting

   ```./cobaltb.py fmt```

   ```./cobaltb.py lint```

 See: [clang-tidy](docs/clang-tidy.md) for an explanation of clang-tidy configuration options.

 ## Rolling Submodules

   ```./cobaltb.py sync_with_fuchsia --make_commit```

 Cobalt Core's standalone build relies on git submodules to load several other
 git repositories, including the Cobalt Registry.  These submodules are not rolled
 automatically, so developers need to occasionally manually roll them to keep the
 libraries and the Cobalt Registry up to date.  The command above will update all
 Cobalt Core submodules to the latest versions used in Fuchsia's integration repo,
 syncs the prebuilts into `cobalt.ensure`, copies over some required files from the
 fuchsia repo, then creates a commit with a description of which submodules were
 updated.

 ## Building Cobalt in Fuchsia

 If the Cobalt Core checkout is part of a Fuchsia tree, it can also be built
 using Fuchsia's tools. This can be done to test the Cobalt code changes build
 and function correctly in Fuchsia. However, the build/test/fmt/lint commands
 above should always be used before contributing a change to Cobalt Core.

 - Build all of Fuchsia (assumes the Fuchsia build was already configured, see
 [instructions](https://fuchsia.dev/fuchsia-src/get-started/build_fuchsia)):
 ```fx build```

 In this way, the Cobalt Core and Fuchsia build systems can function correctly
 in parallel.

 ## Contributing

 Cobalt uses the [Gerrit](https://fuchsia-review.googlesource.com/)
 code review tool. Although Cobalt Core is independent of Fuchsia, we do
 share infrastructure with Fucshia and so much of the instructions from
 Fuchsia's
 [Contributing Guide](https://fuchsia.dev/fuchsia-src/development/source_code/contribute_changes)
 apply to Cobalt Core. The main difference is that Cobalt
 Core does not use Jiri--it uses Git submodules instead.

 ### Generate a cookie

 Follow the [instructions here.](https://fuchsia.dev/fuchsia-src/development/source_code/contribute_changes#generate-a-cookie)

 ### Create a change in Gerrit

 To create a [change](https://gerrit-review.googlesource.com/Documentation/concept-changes.html)
 in Gerrit do the following:

 - Go to your Cobalt Core directory, for example:
 ```cd ~/cobalt```
 - Create a new branch:
 ```
 git checkout -b <branch_name>
 ```
 - Create or edit files in the new branch.
 - Add the updated files:
 ```git add <files>```
 - Commit the updated files and write a change message.
 ``` git commit```
 - Upload the commit to Gerrit:
 ```git push origin HEAD:refs/for/main```

 See the [Gerrit documentation](https://gerrit-documentation.storage.googleapis.com/Documentation/2.12.3/intro-user.html#upload-change) for more information.

 ### Update a change in Gerrit

 After creating a change, to upload a patch to your change, do the following:

 - Create or edit files in the same branch.
 - Add the updated files:
 ```git add <files>```
 - Include the patch in the same commit using the `--amend` option:
 ```git commit --amend```
 - Upload the patch to Gerrit:
 ```git push origin HEAD:refs/for/main```
	# Cobalt: Telemetry with built-in privacy

	[TOC]

	## Overview

	Cobalt is a pipeline for collecting metrics data from user-owned devices in the
	field and producing aggregated reports.

	Cobalt includes a suite of features for preserving user privacy and anonymity
	while giving product owners the data they need to improve their products.

	- Differential Privacy. If you enable it, Cobalt can add random
	noise on the client during metrics collection.
	- Shuffling. Our pipeline shuffles the collected records to break
	linkability. This also amplifies the level of differential privacy.
	- No IDs. Cobalt does not use device IDs or user IDs; not even pseudonymous
	IDs.
	- No exact timestamps. Cobalt uses day indices only.
	- Only standard metric types. Not arbitrary protocol buffers.
	- No ad-hoc, post-facto querying. Whereas on other telemetry systems
	standard practice is to "collect now; ask questions later", with Cobalt the
	analyst must think in advance about what aggregations they wish to report and
	predeclare reports.
	- Only aggregated data is released. It is not possible to query the raw
	collected data.

	## Cobalt Core

	The code in the Git repo at
	[https://fuchsia.googlesource.com/cobalt](https://fuchsia.googlesource.com/cobalt)
	is known as Cobalt Core. It contains:

	- Client-side libraries for:
	- Logging metric events to Cobalt
	- Local aggregation
	- Privacy encoding
	- Local data storage
	- Upload orchestration
	- Protocol buffers for the registry of Cobalt's projects, metrics and reports.
	- Protocol buffers for data transport
	- The registry parser

	In order to have a working Cobalt system you need to also have
	- A server-side implementation of the Cobalt pipeline including a Shuffler
	and Analyzers (aka Report-Generators.)
	- There is a server-side pipeline running at Google with code in the internal Google
	repository.
	- As of this writing that is the only server-side implementation of Cobalt.
	- An embedding of Cobalt Core into a client-side platform.
	- This allows the code-to-be-measured to easily invoke Cobalt's Logger interface.
	- There is an embedding into the Fuchsia operating system at
	[//src/cobalt](https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/src/cobalt/).
	- If you have a Fuchsia checkout, you will find a copy of Cobalt Core has
	been pulled in at `//third_party/cobalt`.
	- As of this writing, Fuchsia is the only embedding of Cobalt Core.

	This document describes how to develop, build and test Cobalt Core independently
	of Fuchsia, as well as from in the Fuchsia tree.

	## Requirements

	1. We have only tested the stand-alone Cobalt Core build on Linux. You may or
	may not have success on other systems.
	2. Python version at least 2.7
	3. Git

	## Fetch the code

	To download just the Cobalt Core repo:

	- `git clone https://fuchsia.googlesource.com/cobalt`
	- `cd cobalt`

	To download the entire Fuchsia source code, including Cobalt Core, follow the
	[instructions](https://fuchsia.dev/fuchsia-src/get-started/get_fuchsia_source).
	Then change into the Cobalt Core directory:

	- `cd third_party/cobalt`

	## Run setup

	```./cobaltb.py setup```

	This command is required after a first checkout, before running any other
	`cobaltb.py` commands, and will take a few minutes the first time. It does
	the following:

	* Fetches third party code into Cobalt's `third_party` dir via Git submodules.
	* Fetches the sysroot and puts it in Cobalt's `sysroot` dir. This uses
	[CIPD](https://chromium.googlesource.com/chromium/src.git/+/main/docs/cipd.md).

	### In the Fuchsia Tree

	If the `./cobaltb.py setup` command is run in the `third_party/cobalt` checkout
	of Cobalt Core in a Fuchsia tree, the Git submodules may interfere with the
	correct functioning of Jiri to update the repos in the Fuchsia tree. If
	problems occur, the submodules can be removed with: `./cobaltb.py deinit`.
	Before running any other `cobaltb.py` commands in this checkout,
	`./cobaltb.py setup` will need to be run again to refetch the submodules.

	## cobaltb.py

	The Python script `cobaltb.py` in the root directory is used to orchestrate
	building and testing Cobalt. It was already used above in `./cobaltb.py setup`.

	* `./cobaltb.py -h` for general help
	* `cobaltb.py <command> -h` for help on a command

	## Build

	```./cobaltb.py build```

	The Cobalt build uses GN and ninja.

	## Test

	```./cobaltb.py test```

	This runs the tests from the currently built code (`build` must be run first).

	## Formatting and Linting

	```./cobaltb.py fmt```

	```./cobaltb.py lint```

	See: [clang-tidy](docs/clang-tidy.md) for an explanation of clang-tidy configuration options.

	## Rolling Submodules

	```./cobaltb.py sync_with_fuchsia --make_commit```

	Cobalt Core's standalone build relies on git submodules to load several other
	git repositories, including the Cobalt Registry. These submodules are not rolled
	automatically, so developers need to occasionally manually roll them to keep the
	libraries and the Cobalt Registry up to date. The command above will update all
	Cobalt Core submodules to the latest versions used in Fuchsia's integration repo,
	syncs the prebuilts into `cobalt.ensure`, copies over some required files from the
	fuchsia repo, then creates a commit with a description of which submodules were
	updated.

	## Building Cobalt in Fuchsia

	If the Cobalt Core checkout is part of a Fuchsia tree, it can also be built
	using Fuchsia's tools. This can be done to test the Cobalt code changes build
	and function correctly in Fuchsia. However, the build/test/fmt/lint commands
	above should always be used before contributing a change to Cobalt Core.

	- Build all of Fuchsia (assumes the Fuchsia build was already configured, see
	[instructions](https://fuchsia.dev/fuchsia-src/get-started/build_fuchsia)):
	```fx build```

	In this way, the Cobalt Core and Fuchsia build systems can function correctly
	in parallel.

	## Contributing

	Cobalt uses the [Gerrit](https://fuchsia-review.googlesource.com/)
	code review tool. Although Cobalt Core is independent of Fuchsia, we do
	share infrastructure with Fucshia and so much of the instructions from
	Fuchsia's
	[Contributing Guide](https://fuchsia.dev/fuchsia-src/development/source_code/contribute_changes)
	apply to Cobalt Core. The main difference is that Cobalt
	Core does not use Jiri--it uses Git submodules instead.

	### Generate a cookie

	Follow the [instructions here.](https://fuchsia.dev/fuchsia-src/development/source_code/contribute_changes#generate-a-cookie)

	### Create a change in Gerrit

	To create a [change](https://gerrit-review.googlesource.com/Documentation/concept-changes.html)
	in Gerrit do the following:

	- Go to your Cobalt Core directory, for example:
	```cd ~/cobalt```
	- Create a new branch:
	```
	git checkout -b <branch_name>
	```
	- Create or edit files in the new branch.
	- Add the updated files:
	```git add <files>```
	- Commit the updated files and write a change message.
	``` git commit```
	- Upload the commit to Gerrit:
	```git push origin HEAD:refs/for/main```

	See the [Gerrit documentation](https://gerrit-documentation.storage.googleapis.com/Documentation/2.12.3/intro-user.html#upload-change) for more information.

	### Update a change in Gerrit

	After creating a change, to upload a patch to your change, do the following:

	- Create or edit files in the same branch.
	- Add the updated files:
	```git add <files>```
	- Include the patch in the same commit using the `--amend` option:
	```git commit --amend```
	- Upload the patch to Gerrit:
	```git push origin HEAD:refs/for/main```