Continuous integration guide
This is the documentation I wish I had when I was setting up CI
for System UI.
Change-Id: I739d9a1c4e0e350f5e7ba7f8aa625b984e4b6346
diff --git a/docs/guide/README.md b/docs/guide/README.md
new file mode 100644
index 0000000..525db93
--- /dev/null
+++ b/docs/guide/README.md
@@ -0,0 +1,213 @@
+# Continuous integration guide for Fuchsia
+
+This is a guide to using LUCI, Chromium's build system, to continuously build
+and test Fuchsia projects.
+
+We're going to do this in two steps:
+
+1. Configure **scheduled builds**. Tell LUCI how to build your project and set
+up a schedule for continuous integration.
+2. After this is done, you can set up a **commit queue** to block commits in
+Gerrit that would break the build.
+
+If you've already done part of this work, or if it's set up already and you
+just need to change the configuration, this guide should still help you
+understand the LUCI concepts and how Fuchsia uses them.
+
+This repo also contains example configuration files that you can copy into your
+project and alter as needed.
+
+## Step one: Set up scheduled builds
+
+A branch named `infra/config` is the first place LUCI will look for
+configuration. Copy the example files from this directory to get started. The
+following sections will go through the individual files.
+
+### `project.cfg` on the `infra/config` branch
+
+This is the top-level configuration file for a project. In LUCI, **project** is
+a high-level organizing concept. In Fuchsia, one project usually corresponds to
+one repo.
+
+### `cr-buildbucket.cfg` on the `infra/config` branch
+
+`cr-buildbucket.cfg` tells LUCI how to build the project using Buildbucket.
+
+**Buildbucket** is LUCI's build queue. It defines operations for:
+
+- Scheduling a build job.
+- Leasing a build in order to execute it.
+- Marking a build as successful or failed.
+
+Buildbucket is a generic build queue, which means it's just a set of API
+definitions, and we need a service that actually implements it. We use
+**Swarming** as our implementation. Swarming is a big Chromium project that
+serves many purposes, but you can think of it as the thing that actually runs
+the Fuchsia build jobs.
+
+You probably want to keep the ACL section as-is. In the example file, the
+`project-fuchsia-tryjob-access` group can schedule builds, and anyone can see
+them.
+
+In the Swarming secion of the `cr-buildbucket.cfg`, we specify the hostname of
+our Swarming instance (always the main Chromium instance at
+`chromium-swarm.appspot.com`) and a set of **builders.**
+
+In Buildbucket terminology, a **builder** is a specification for a machine that
+can carry out a specific build task. You'll want a builder for each target
+platform, and possibly separate debug and release builders. The example file
+defines two builders: x86 and ARM.
+
+Each builder specifcation includes a `category` field, to group related builds.
+
+The actual task that the builder carries out is called a **recipe**. A recipe is
+basically just a sequence of commands to run on the builder machine, specified
+in Python. The recipe framework has a
+[user guide](https://chromium.googlesource.com/external/github.com/luci/recipes-py/+/master/doc/user_guide.md)
+if you want to learn more about it, but most Fuchsia projects share the same
+build recipe,
+[which is defined in the "infra/recipes" repo](https://fuchsia.googlesource.com/infra/recipes/+/master/recipes/fuchsia.py).
+
+Recipes are configured in two parts of `cr-buildbucket.cfg`. First, in the
+`swarming` field, there is a `builder_defaults` field which contains recipe
+settings common to all builders. In the example file, this is where we say to
+use the standard Fuchsia recipe, and we give it a `modules` property to pass in,
+along with some other standard properties like the URL of the Fuchsia manifest
+repo. `module` should refer to the name of a [config file in the packages
+repo](https://fuchsia.googlesource.com/packages/+/master/gn/), which will build
+a particular Fuchsia package.
+
+### `luci-logdog.cfg` on the `infra/config` branch
+
+LogDog is the part of LUCI that collects logs from the recipes when they run. A
+`luci-logdog.cfg` file is required alongside `cr-buildbucket.cfg` to tell LUCI
+where to put the logs and what permissions to use. We generally use the same
+LogDog settings across Fuchsia projects, so you don't need to change anything.
+
+### `luci-scheduler.cfg` on the `infra/config` branch
+
+This file tells LUCI which builds to run continuously. It consists of multiple
+`job` declarations, each one of which refers to a builder using the names
+defined in `cr-buildbucket.cfg`. This is basically cron for LUCI, and you can
+use cron expressions in the `schedule` parameter of each job. By convention in
+Fuchsia, we use "with 10m interval" as the schedule for our continuous jobs.
+
+### Global LUCI configuration
+
+The last thing you need to do before you test your configuration is make sure
+that LUCI knows about your repo in its global configuration. Currently we share
+a LUCI instance with the Chrome team, and this configuration is in their
+Google-internal infrastructure repo. Ask someone from the Fuchsia infrastructure
+team to get your project added.
+
+### Testing your configuration
+
+Right now the only real way to test your LUCI configuration is to commit a
+change and see if it works, then commit a fix if it doesn't. Once your
+configuration gets committed, your scheduled builds should start showing up on
+the [LUCI Scheduler](https://luci-scheduler.appspot.com/) web interface.
+
+However, you can test your recipe locally, by looking at the recipe properties
+in `cr-buildbucket.cfg` and passing them as JSON to the `--properties`
+argument of the referenced `recipes.py` script. In the example
+`cr-buildbucket.cfg`, the recipe is configured like this:
+
+```
+recipe {
+ name: "fuchsia"
+ repository: "https://fuchsia.googlesource.com/infra/recipes"
+ properties: "remote:https://fuchsia.googlesource.com/manifest"
+ properties: "manifest:userspace"
+ properties_j: "modules:[\"example\"]"
+}
+```
+
+This says that the recipe can be found in the "infra/recipes" repo, with the
+name "fuchsia", and that it should be always be run with a certain set of
+properties. (This is a little confusing because one of those properties is the
+URL of a different repo, the Fuchsia manifest. Note that these URLs are used by
+different systems at different times: `repository` is used by Buildbucket to
+find the recipe. `remote` is used by the Fuchsia recipe to find the Fuchsia
+manifest.)
+
+There are also additional properties for each specific builder. For example, the
+x86 builder includes this:
+
+```
+recipe {
+ properties: "build_type:debug"
+ properties: "target:x86-64"
+}
+```
+
+So, to test the x86 build, you can put all of these properties together in a
+temporary file called `example.json`, like this:
+
+```
+{
+ "remote": "https://fuchsia.googlesource.com/manifest",
+ "manifest": "userspace",
+ "modules": ["example"],
+ "build_type": "debug",
+ "target": "x86-64"
+}
+```
+
+And then run the `recipes.py` script from the root directory of
+[infra/recipes](https://fuchsia.googlesource.com/infra/recipes), which is the
+place we told Buildbucket to find our recipe:
+
+```
+python recipes.py run --properties-file example.json fuchsia
+```
+
+If that correctly builds the part of Fuchsia you want to build, then your
+recipe is probably OK. However, you'll still need to actually commit to make
+sure your configuration is valid.
+
+## Step two: Set up the commit queue
+
+### `refs.cfg` on the `infra/config` branch
+
+This file tells LUCI where in the repo it should look for additional
+configuration files. In the case of a Fuchsia repo that uses the standard
+Fuchsia build recipe, the additional config file we care about is `cq.cfg`,
+described below.
+
+By convention in Fuchsia we always use the path `infra/config` on the `master`
+branch. Note that this is different from the `infra/config` branch, which can be
+a bit confusing.
+
+### `infra/config/cq.cfg` on the `master` branch
+
+This the configuration file for the Chromium Commit Queue, which is a thing that
+watches Gerrit for CLs it can build, test, and commit.
+
+We always use the same `cq_status_url` for Fuchsia, and the `git_repo_url` is
+specific to the repo. `cq_name` is a unique identifier for the project's CQ,
+and it can usually just be the same as the all-lowercase name of the repo.
+
+A LUCI commit queue configuration consists of one or more **verifiers**, which
+are steps that must pass before committing is allowed. The basic configuration
+for a Fuchsia CQ is a single **try-job verifier**, which means that every commit
+must pass a check that involves running a set of Builbucket jobs.
+
+These jobs are specified in `cr-buildbucket.cfg` on the `infra/config` branch
+(see above) and referred to by their `name` field. You probably want to run all
+of your builders on the CQ. That means listing each one individually here. The
+example file includes the standard ARM/x86 builders.
+
+### Testing the commit queue
+
+To test the commit queue, create a new commit for your project in Gerrit. The
+contents of the commit don't matter, since you will only be doing a dry run.
+
+If you have the right permissions in Gerrit, you should see an option labeled
+"Commit-Queue" when you click on the "Reply" button, like so:
+
+
+
+Select the "+1" option as shown above, which signals a dry run, and click
+"Send". Very soon after that you should see a comment from the CQ bot, with a
+link to its progress. Check to see if it completes all the builds you configured
+in the above steps.
diff --git a/docs/guide/cq.cfg b/docs/guide/cq.cfg
new file mode 100644
index 0000000..e65140d
--- /dev/null
+++ b/docs/guide/cq.cfg
@@ -0,0 +1,28 @@
+# Copyright 2017 The Fuchsia Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# See http://luci-config.appspot.com/schemas/projects/refs:cq.cfg for the
+# documentation of this file format.
+
+version: 1
+cq_name: "example"
+cq_status_url: "https://fuchsia-cq-status.appspot.com"
+git_repo_url: "https://fuchsia.googlesource.com/example"
+
+gerrit {}
+
+verifiers {
+ try_job {
+ buckets {
+ name: "luci.example.ci",
+ builders {
+ name: "Linux x86-64"
+ }
+ builders {
+ name: "Linux arm64"
+ }
+ }
+ }
+ sign_cla {}
+}
diff --git a/docs/guide/cr-buildbucket.cfg b/docs/guide/cr-buildbucket.cfg
new file mode 100644
index 0000000..4711a7b
--- /dev/null
+++ b/docs/guide/cr-buildbucket.cfg
@@ -0,0 +1,77 @@
+# Copyright 2017 The Fuchsia Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# Defines buckets on cr-buildbucket.appspot.com, used by to schedule builds
+# on buildbot.
+#
+# See http://luci-config.appspot.com/schemas/projects:buildbucket.cfg for
+# schema of this file and documentation.
+#
+# Please keep this list sorted by bucket name.
+
+buckets {
+ name: "luci.example.ci"
+ acls {
+ role: READER
+ group: "all"
+ }
+ acls {
+ role: SCHEDULER
+ group: "project-fuchsia-tryjob-access"
+ }
+
+ swarming {
+ hostname: "chromium-swarm.appspot.com"
+ url_format: "https://luci-milo.appspot.com/swarming/task/{task_id}"
+
+ builder_defaults {
+ swarming_tags: "allow_milo:1"
+ dimensions: "pool:Fuchsia"
+
+ # We use the fuchsia.py recipe from the manifest repo for the builds.
+ recipe {
+ name: "fuchsia"
+ repository: "https://fuchsia.googlesource.com/infra/recipes"
+ properties: "remote:https://fuchsia.googlesource.com/manifest"
+ properties: "manifest:userspace"
+ properties_j: "modules:[\"example\"]"
+ }
+ cipd_packages {
+ package_name: "infra/tools/cipd/${platform}"
+ version: "git_revision:26deefff659f87c8589a43046982f7a4900e83cc"
+ path: "cipd"
+ }
+ cipd_packages {
+ package_name: "fuchsia/tools/jiri/${platform}"
+ version: "latest"
+ path: "packages/jiri"
+ }
+ execution_timeout_secs: 3600 # 60 min
+ }
+
+ # Keep builders sorted by category, then name.
+
+ builders {
+ category: "Example"
+ name: "Linux x86-64"
+ dimensions: "os:Ubuntu-14.04"
+ dimensions: "cpu:x86-64"
+ recipe {
+ properties: "build_type:debug"
+ properties: "target:x86-64"
+ }
+ }
+
+ builders {
+ category: "Example"
+ name: "Linux arm64"
+ dimensions: "os:Ubuntu-14.04"
+ dimensions: "cpu:x86-64"
+ recipe {
+ properties: "build_type:debug"
+ properties: "target:arm64"
+ }
+ }
+ }
+}
diff --git a/docs/guide/images/gerrit-cq.png b/docs/guide/images/gerrit-cq.png
new file mode 100644
index 0000000..8826b0c
--- /dev/null
+++ b/docs/guide/images/gerrit-cq.png
Binary files differ
diff --git a/docs/guide/luci-logdog.cfg b/docs/guide/luci-logdog.cfg
new file mode 100644
index 0000000..12ceb79
--- /dev/null
+++ b/docs/guide/luci-logdog.cfg
@@ -0,0 +1,16 @@
+# Copyright 2017 The Fuchsia Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# For the schema of this file and documentation, see ProjectConfig message in
+# https://luci-config.appspot.com/schemas/services/luci-logdog:logdog.cfg
+
+# Auth groups who can read log streams.
+reader_auth_groups: "all"
+# Auth groups who can register and emit new log streams.
+writer_auth_groups: "luci-logdog-fuchsia-writers"
+
+# The base Google Storage archival path for this project.
+#
+# Archived LogDog logs will be written to this bucket/path.
+archive_gs_bucket: "fuchsia-logdog"
diff --git a/docs/guide/luci-scheduler.cfg b/docs/guide/luci-scheduler.cfg
new file mode 100644
index 0000000..0f8c5b7
--- /dev/null
+++ b/docs/guide/luci-scheduler.cfg
@@ -0,0 +1,30 @@
+# Copyright 2017 The Fuchsia Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# Defines cron jobs on luci-scheduler.appspot.com.
+
+# Launch Buildbucket tasks continuously.
+job {
+ id: "linux-x86-64"
+ schedule: "with 10m interval"
+ task: {
+ buildbucket_task: {
+ server: "https://cr-buildbucket.appspot.com"
+ bucket: "luci.example.ci"
+ builder: "Linux x86-64"
+ }
+ }
+}
+
+job {
+ id: "linux-arm64"
+ schedule: "continuously"
+ task: {
+ buildbucket_task: {
+ server: "https://cr-buildbucket.appspot.com"
+ bucket: "luci.example.ci"
+ builder: "Linux arm64"
+ }
+ }
+}
diff --git a/docs/guide/project.cfg b/docs/guide/project.cfg
new file mode 100644
index 0000000..33fe50f
--- /dev/null
+++ b/docs/guide/project.cfg
@@ -0,0 +1,9 @@
+# Copyright 2017 The Fuchsia Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# For the schema of this file and documentation, see ProjectCfg message in
+# https://luci-config.appspot.com/schemas/projects:project.cfg
+
+name: "Example"
+access: "group:all" # public
diff --git a/docs/guide/refs.cfg b/docs/guide/refs.cfg
new file mode 100644
index 0000000..be11710
--- /dev/null
+++ b/docs/guide/refs.cfg
@@ -0,0 +1,11 @@
+# Copyright 2017 The Fuchsia Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# For the schema of this file and documentation, see RefsCfg message in
+# https://luci-config.appspot.com/schemas/projects:refs.cfg
+
+refs {
+ name: "refs/heads/master"
+ config_path: "infra/config"
+}
