Recipe Bootstrapping

The recipe bootstrapping tool reads a build.proto from stdin, resolves the appropriate recipe version to use, and executes the build.

The tool implements the luciexe protocol. Fuchsia Buildbucket builders which require recipe versioning should configure the tool as their executable. See the lucicfg documentation for more information.

Recipe version resolution

The “recipe version” is a recipes.git revision, which is resolved through a decision tree on build.proto. See main file.

Recipe checkout & execution

Once the recipe version is resolved, a checkout of recipes.git is performed. Within the checkout, recipes.py luciexe is exec'd to advance into build execution.

Properties

recipe_bootstrap reads several optional builder properties to determine its behavior:

recipe_integration_remote

Example: “https://fuchsia-internal.googlesource.com/integration”

If a build is triggered by a cron schedule rather than by a commit (via LUCI Scheduler) or CL (via Commit Queue), then it won't have a gitiles_commit or gerrit_change in its input.

For public builders it's safe to fall back to using the public integration repository. However, many internal builders assume access to the internal integration repository, so they need to fall back to using internal integration instead.

recipe_integration_remote lets internal builders that may be triggered on a cron job fall back to checking out the specified integration repository (generally the internal integration repository) rather than the default.

recipes_host_override

Example: “fuchsia.googlesource.com”

Some builders are triggered by commits in Git-on-Borg repositories that are not on one of the Fuchsia-owned Git-on-Borg hosts. Only Fuchsia-owned hosts have integration repositories, so for these builds it‘s not valid to try to checkout the integration repository that’s on the same host and ref as the triggering commit.

To avoid that, a builder that‘s triggered by a non-Fuchsia-owned repo can set recipes_host_override to hostname of the Fuchsia-owned Git-on-Borg instance that holds the desired integration repository. If recipes_host_override is set, we use the refs/heads/master ref for checkout because there’s no guarantee for non-Fuchsia-owned repositories that the triggering ref will correspond to a ref of the integration repository.

recipes_integration_ref_override

Example: “refs/heads/main”

This property overrides the integration.git ref that recipe_bootstrap resolves in the case where a resolved integration base revision isn't already specified in the build input.

Similar to recipes_host_override, this is useful for builders that are triggered by non-Fuchsia-owned repositories, for which we can't assume that the triggering ref will correspond to a valid integration.git ref.

Testing with led

recipe_bootstrap has unit tests, and in general as much logic as possible should be covered by unit tests. However, because it has a large interface with Buildbucket and depends on services like Git-on-Borg, unit tests aren't sufficient to guarantee correctness of recipe_bootstrap changes.

The easiest way to integration-test recipe_bootstrap changes is with led. Testing recipe_bootstrap changes with led is slightly more involved than testing recipe changes, because by default led jobs run recipes.py directly instead of using recipe_bootstrap.

To circumvent this behavior, you need to build recipe_bootstrap locally and upload the resulting executable to a file with the special name luciexe in the led job's isolated input.

  1. Choose the ID of a Buildbucket build that you want to re-run with the new version of recipe_bootstrap.
    • Make sure not to choose a builder that has side effects like uploading official images.
    • Also note that if the led job launches any subbuilds, those subbuilds will not use the custom version of recipe_bootstrap.
    • For these two reasons, build-only CQ builders are ideal for testing recipe_bootstrap changes.
  2. led get-build <build-id> > led.json to download the build's specification locally.
  3. Open led.json and delete the cipd_package and cipd_version fields under exe. Those fields pin the build to the currently live version of recipe_bootstrap, which the led job will use unless you clear those fields. field.
  4. Run cat led.json | led edit-isolated | led launch. That will create and print the path to a temporary local directory that you can add files to for them to be included in the led job‘s isolated input, and block until you’re done assembling the inputs.
  5. In another terminal, run go build -o $ISOLATED_INPUT_DIR/kitchen-checkout/luciexe ./cmd/recipe_bootstrap where $ISOLATED_INPUT_DIR is the temporary directory created by edit-isolated. If the OS and/or CPU of the machine that will run the led job differ from your local machine, make sure to set the GOOS and GOARCH environment variables appropriately for the go build step. Otherwise the resulting executable may not be able to run in the led job. (Most builds run on Linux machines, so if you‘re building recipe_bootstrap on a Linux machine you probably don’t need to worry about this.)
  6. Hit Enter in the terminal running the pipeline of led commands. led will upload the recipe_bootstrap executable to Isolated, and then start a task that uses that recipe_bootstrap executable.