This is a guide to using LUCI, Chromium's build system, to continuously build and test Fuchsia projects.
We're going to do this in two steps:
If you‘ve already done part of this work, or if it’s set up already and you just need to change the configuration, this guide should still help you understand the LUCI concepts and how Fuchsia uses them.
This repo also contains example configuration files that you can copy into your project and alter as needed.
A branch named infra/config
is the first place LUCI will look for configuration. Copy the example files from this directory to get started. The following sections will go through the individual files.
project.cfg
on the infra/config
branchThis is the top-level configuration file for a project. In LUCI, project is a high-level organizing concept. In Fuchsia, one project usually corresponds to one repo.
cr-buildbucket.cfg
on the infra/config
branchcr-buildbucket.cfg
tells LUCI how to build the project using Buildbucket.
Buildbucket is LUCI's build queue. It defines operations for:
Buildbucket is a generic build queue, which means it's just a set of API definitions, and we need a service that actually implements it. We use Swarming as our implementation. Swarming is a big Chromium project that serves many purposes, but you can think of it as the thing that actually runs the Fuchsia build jobs.
You probably want to keep the ACL section as-is. In the example file, the project-fuchsia-tryjob-access
group can schedule builds, and anyone can see them.
In the Swarming secion of the cr-buildbucket.cfg
, we specify the hostname of our Swarming instance (always the main Chromium instance at chromium-swarm.appspot.com
) and a set of builders.
In Buildbucket terminology, a builder is a specification for a machine that can carry out a specific build task. You'll want a builder for each target platform, and possibly separate debug and release builders. The example file defines two builders: x86 and ARM.
Each builder specifcation includes a category
field, to group related builds.
The actual task that the builder carries out is called a recipe. A recipe is basically just a sequence of commands to run on the builder machine, specified in Python. The recipe framework has a user guide if you want to learn more about it, but most Fuchsia projects share the same build recipe, which is defined in the “infra/recipes” repo.
Recipes are configured in two parts of cr-buildbucket.cfg
. First, in the swarming
field, there is a builder_defaults
field which contains recipe settings common to all builders. In the example file, this is where we say to use the standard Fuchsia recipe, and we give it a modules
property to pass in, along with some other standard properties like the URL of the Fuchsia manifest repo. module
should refer to the name of a config file in the packages repo, which will build a particular Fuchsia package.
luci-logdog.cfg
on the infra/config
branchLogDog is the part of LUCI that collects logs from the recipes when they run. A luci-logdog.cfg
file is required alongside cr-buildbucket.cfg
to tell LUCI where to put the logs and what permissions to use. We generally use the same LogDog settings across Fuchsia projects, so you don't need to change anything.
luci-scheduler.cfg
on the infra/config
branchThis file tells LUCI which builds to run continuously. It consists of multiple job
declarations, each one of which refers to a builder using the names defined in cr-buildbucket.cfg
. This is basically cron for LUCI, and you can use cron expressions in the schedule
parameter of each job. By convention in Fuchsia, we use “with 10m interval” as the schedule for our continuous jobs.
The last thing you need to do before you test your configuration is make sure that LUCI knows about your repo in its global configuration. Currently we share a LUCI instance with the Chrome team, and this configuration is in their Google-internal infrastructure repo. Ask someone from the Fuchsia infrastructure team to get your project added.
Right now the only real way to test your LUCI configuration is to commit a change and see if it works, then commit a fix if it doesn't. Once your configuration gets committed, your scheduled builds should start showing up on the LUCI Scheduler web interface.
However, you can test your recipe locally, by looking at the recipe properties in cr-buildbucket.cfg
and passing them as JSON to the --properties
argument of the referenced recipes.py
script. In the example cr-buildbucket.cfg
, the recipe is configured like this:
recipe { name: "fuchsia" repository: "https://fuchsia.googlesource.com/infra/recipes" properties: "remote:https://fuchsia.googlesource.com/manifest" properties: "manifest:userspace" properties_j: "modules:[\"example\"]" }
This says that the recipe can be found in the “infra/recipes” repo, with the name “fuchsia”, and that it should be always be run with a certain set of properties. (This is a little confusing because one of those properties is the URL of a different repo, the Fuchsia manifest. Note that these URLs are used by different systems at different times: repository
is used by Buildbucket to find the recipe. remote
is used by the Fuchsia recipe to find the Fuchsia manifest.)
There are also additional properties for each specific builder. For example, the x86 builder includes this:
recipe { properties: "build_type:debug" properties: "target:x86-64" }
So, to test the x86 build, you can put all of these properties together in a temporary file called example.json
, like this:
{ "remote": "https://fuchsia.googlesource.com/manifest", "manifest": "userspace", "modules": ["example"], "build_type": "debug", "target": "x86-64" }
And then run the recipes.py
script from the root directory of infra/recipes, which is the place we told Buildbucket to find our recipe:
python recipes.py run --properties-file example.json fuchsia
If that correctly builds the part of Fuchsia you want to build, then your recipe is probably OK. However, you'll still need to actually commit to make sure your configuration is valid.
refs.cfg
on the infra/config
branchThis file tells LUCI where in the repo it should look for additional configuration files. In the case of a Fuchsia repo that uses the standard Fuchsia build recipe, the additional config file we care about is cq.cfg
, described below.
By convention in Fuchsia we always use the path infra/config
on the master
branch. Note that this is different from the infra/config
branch, which can be a bit confusing.
infra/config/cq.cfg
on the master
branchThis the configuration file for the Chromium Commit Queue, which is a thing that watches Gerrit for CLs it can build, test, and commit.
We always use the same cq_status_url
for Fuchsia, and the git_repo_url
is specific to the repo. cq_name
is a unique identifier for the project's CQ, and it can usually just be the same as the all-lowercase name of the repo.
A LUCI commit queue configuration consists of one or more verifiers, which are steps that must pass before committing is allowed. The basic configuration for a Fuchsia CQ is a single try-job verifier, which means that every commit must pass a check that involves running a set of Builbucket jobs.
These jobs are specified in cr-buildbucket.cfg
on the infra/config
branch (see above) and referred to by their name
field. You probably want to run all of your builders on the CQ. That means listing each one individually here. The example file includes the standard ARM/x86 builders.
To test the commit queue, create a new commit for your project in Gerrit. The contents of the commit don't matter, since you will only be doing a dry run.
If you have the right permissions in Gerrit, you should see an option labeled “Commit-Queue” when you click on the “Reply” button, like so:
Select the “+1” option as shown above, which signals a dry run, and click “Send”. Very soon after that you should see a comment from the CQ bot, with a link to its progress. Check to see if it completes all the builds you configured in the above steps.