Migrate to google.golang.org/protobuf

Generated protos still contain references to github.com/golang/protobuf
until protoc is updated past 3.14.0.

Bug: 70570
Change-Id: Ia3bffa1485d34dd3df0b9d818fa9bd7f4461895d
Reviewed-on: https://fuchsia-review.googlesource.com/c/cobalt/+/504899
Commit-Queue: Tamir Duberstein <tamird@google.com>
Reviewed-by: Jared Weinstein <jaredweinstein@google.com>
19 files changed
tree: 7263d03b6ba44f06134762a80ba17f18a002dc78
  1. build/
  2. docs/
  3. keys/
  4. manifest/
  5. meta/
  6. src/
  7. third_party/
  8. tools/
  9. .cipd_version
  10. .clang-format
  11. .clang-tidy
  12. .gitignore
  13. .gitmodules
  14. .gn
  15. AUTHORS
  16. BUILD.gn
  17. cipd
  18. cobalt.cmake
  19. cobalt.ensure
  20. cobaltb.py
  21. LICENSE
  22. metrics_registry.gni
  23. OWNERS
  24. PATENTS
  25. README.md
  26. setup.sh
  27. SYSROOT.md
README.md

Cobalt: Telemetry with built-in privacy

Overview

Cobalt is a pipeline for collecting metrics data from user-owned devices in the field and producing aggregated reports.

Cobalt includes a suite of features for preserving user privacy and anonymity while giving product owners the data they need to improve their products.

  • Local Differential Privacy. If you enable it, Cobalt can add random noise on the client during metrics collection.
  • Shuffling. Our pipeline shuffles the collected records to break linkability. This also amplifies the level of differential privacy.
  • No IDs. Cobalt does not use device IDs or user IDs; not even pseudonymous IDs.
  • No exact timestamps. Cobalt uses day indices only.
  • Only standard metric types. Not arbitrary protocol buffers.
  • No ad-hoc, post-facto querying. Whereas on other telemetry systems standard practice is to “collect now; ask questions later”, with Cobalt the analyst must think in advance about what aggregations they wish to report and predeclare reports.
  • Only aggregated data is released. It is not possible to query the raw collected data.

Cobalt Core

The code in the Git repo at https://fuchsia.googlesource.com/cobalt is known as Cobalt Core. It contains:

  • Client-side libraries for:
    • Logging metric events to Cobalt
    • Local aggregation
    • Privacy encoding
    • Local data storage
    • Upload orchestration
    • Protocol buffers for the registry of Cobalt's projects, metrics and reports.
    • Protocol buffers for data transport
    • The registry parser
    • A test application

In order to have a working Cobalt system you need to also have

  • A server-side implementation of the Cobalt pipeline including a Shuffler and Analyzers (aka Report-Generators.)
    • There is a server-side pipeline running at Google with code in the internal Google repository.
    • As of this writing that is the only server-side implementation of Cobalt.
  • An embedding of Cobalt Core into a client-side platform.
    • This allows the code-to-be-measured to easily invoke Cobalt's Logger interface.
    • There is an embedding into the Fuchsia operating system at //src/cobalt.
    • If you have a Fuchsia checkout, you will find a copy of Cobalt Core has been pulled in at //third_party/cobalt.
    • As of this writing, Fuchsia is the only embedding of Cobalt Core.

This document describes how to develop, build and test Cobalt Core independently of Fuchsia.

Requirements

  1. We have only tested the stand-alone Cobalt Core build on Linux. You may or may not have success on other systems.
  2. Python version at least 2.7
  3. Git

Fetch the code

For example via

  • git clone https://fuchsia.googlesource.com/cobalt
  • cd cobalt

Run setup

./cobaltb.py setup

This will take a few minutes the first time. It does the following:

  • Fetches third party code into the third_party dir via Git submodules.
  • Fetches the sysroot and puts it in the sysroot dir. This uses CIPD.

cobaltb.py

The Python script cobaltb.py in the root directory is used to orchestrate building and testing Cobalt. It was already used above in ./cobaltb.py setup.

  • ./cobaltb.py -h for general help
  • cobaltb.py <command> -h for help on a command

Build

./cobaltb.py build

The Cobalt build uses GN and ninja.

Test

./cobaltb.py test

Formatting and Linting

./cobaltb.py fmt

./cobaltb.py lint

See: clang-tidy for an explanation of clang-tidy configuration options.

Contributing

Cobalt uses the Gerrit code review tool. Although Cobalt Core is independent of Fuchsia, we do share infrastructure with Fucshia and so much of the instructions from Fuchsia's Contributing Guide apply to Cobalt Core. The main difference is that Cobalt Core does not use Jiri--it uses Git submodules instead.

Generate a cookie

Follow the instructions here.

Create a change in Gerrit

To create a change in Gerrit do the following:

  • Go to your Cobalt Core directory, for example: cd ~/cobalt
  • Create a new branch:
git checkout -b <branch_name>
  • Create or edit files in the new branch.
  • Add the updated files: git add <files>
  • Commit the updated files and write a change message. git commit
  • Upload the commit to Gerrit: git push origin HEAD:refs/for/master

See the Gerrit documentation for more information.

Update a change in Gerrit

After creating a change, to upload a patch to your change, do the following:

  • Create or edit files in the same branch.
  • Add the updated files: git add <files>
  • Include the patch in the same commit using the --amend option: git commit --amend
  • Upload the patch to Gerrit: git push origin HEAD:refs/for/master