An analytics pipeline with built-in user-privacy.
You may encounter this document in one of two scenarios:
https://fuchsia.googlesource.com/cobalt
, or//third_party/cobalt.
This document should be used only in the first context. It describes how to build and test Cobalt independently of Fuchsia. Stand-alone Cobalt includes server-side components that run in Linux on Google Kubernetes Engine and a generic client library that is compiled for Linux using the build described in this document..
When imported into //third_party/cobalt
within a Fuchsia checkout, the Cobalt client library is compiled for Fuchsia and accessed via a FIDL wrapper. If you are trying to use Cobalt from an application running on Fuchsia, stop reading this document and instead read Cobalt's README.md in the Fuchsia repo.
sudo apt-get install libstdc++-8-dev
For example via
git clone https://fuchsia.googlesource.com/cobalt
cd cobalt
./cobaltb.py setup
. This will take a few minutes the first time. It does the following:
third_party
dir via git submodules.sysroot
dir. This uses a tool called cipd
or Chrome Infrastructure Package Deployer.The Python script cobaltb.py in the root directory is used to orchestrate building, testing and deploying to Google Kubernetes Engine. It was already used above in ./cobaltb.py setup
.
./cobaltb.py -h
for general helpcobaltb.py <command> -h
for help on a commandcobaltb.py <command> <subcommand> -h
for help on a sub-commandIf you pass the flag --verbose
to various cobaltb.py
commands you will see more verbose output. Pass it multiple times to increase the verbosity further.
./cobaltb.py clean
./cobaltb.py build
The Cobalt build uses GN and ninja.
./cobaltb.py lint
See: clang-tidy for an explanation of clang-tidy configuration options.
./cobaltb.py test
This runs the whole suite of tests finally running the the end-to-end test. The tests should all pass.--tests=
argument../cobaltb.py test -h
for documentation about the --tests=
argument.Cobalt uses the Gerrit code review tool. See the submitting-changes section of the Fuchsia contributing doc for more info about using Gerrit. But note that the stand-alone Cobalt build currently does not use Jiri. Use the command git push origin HEAD:refs/for/master
. Also the other sections of that document are not relevant to stand-alone Cobalt.
The source layout is related to Cobalt‘s process architecture. Here we describe the source layout and process architecture together. Most of Cobalt’s code is C++. The Shuffler is written in Go.
The most interesting contents of the root directory are the .proto files observation.proto
, which contains the definitions of Observation and Envelope, and encrypted_message.proto
, which contains the definition of EncryptedMessage. Observations are the basic units of data captured by a Cobalt client application. Each Observation is encrypted and the bytes are stored in an EncryptedMessage. Multiple EncryptedMessages are stored in an ObservationBatch. Multiple ObservationBatches are stored in an Envelope. Envelopes are sent via gRPC from the Encoder to the Shuffler.
This directory contains GN files needed for building Cobalt. Most of these files should be similar if not identical to the associated file in the Fuchsia repository here.
This directory contains development keys for encrypting observations/envelopes to the backend.
This directory contains a Jiri manifest. It is used to integrate Cobalt into the rest of the Fuchsia build when Cobalt is imported into third_party/cobalt. This is not used at all in Cobalt's stand-alone build.
This directory contains cmx files for running Cobalt's tests in fuchsia.
This directory is the root directory of the Cobalt source code. All new code should be put in a subdirectory of src/
This directory contains the implementations of Cobalt's privacy- preserving algorithms. This code is linked into both the Encoder, which uses it to encode Observations, and the ReportMaster, which uses it to decode Observations.
This directory contains the code for Cobalt's Encoder, which is a client library whose job is to encode Observations using one of several privacy-preserving encodings, and send Envelopes to the Shuffler using gRPC.
This directory contains the code for Cobalt 1.0's Logger, which is a client library whose job is to encode Observation2s using one of several privacy-preserving encodings, and send Envelopes to the shuffler.
This directory contains the code for a linux test client for cobalt.
This directory contains the implementation of Cobalt's config registration system. A client that wants to use Cobalt starts by registering configurations of their Metrics, Encodings and Reports.
This directory contains build, test and deployment tooling.
This directory contains utility libraries used by the Encoder and Analyzer.