cmd/recipe_bootstrap/README.md

# Recipe Bootstrapping

The recipe bootstrapping tool reads a build.proto from stdin, resolves the
appropriate recipe version to use, and executes the build.

The tool implements the
[luciexe protocol](https://pkg.go.dev/go.chromium.org/luci/luciexe). Fuchsia
Buildbucket builders which require recipe versioning should configure the tool
as their executable. See the
[lucicfg documentation](https://chromium.googlesource.com/infra/luci/luci-go/+/main/lucicfg/doc/#luci.executable)
for more information.

## Behavior

### Builder properties resolution

recipe_bootstrap will resolve the build's input properties by reading properties
from a file in integration.git, at a revision determined by the input
build.proto.

The path to the properties file is calculated based on the builder information
in the input build.proto, specifically
`infra/config/generated/<project>/properties/<bucket>/<builder>.json`,
where `<project>`, `<bucket>`, and `<builder>` each come from the corresponding
field of the input build.proto under the top-level `builder` field.

This makes it possible to "version" builder properties – i.e. property changes
can be tested in CQ, and properties don't need to be kept backwards compatible
with old versions of code on release branches.

However, some properties cannot be versioned because they're needed by
recipe_bootstrap to resolve the integration repository and revision to checkout.
See the "Properties" section for a full listing.

### Recipe version resolution

The "recipe version" is a recipes.git revision, which is resolved through a
decision tree on build.proto. See [main file](main.go).

The recipe version is normally read from a file in the same integration.git
checkout that the properties file comes from. However, if the build input
contains a change to the recipes repository, recipe_bootstrap will check out the
recipes repository at the input change, rather than at the version pinned in
integration. That allows us to test recipe changes in presubmit using regular
tryjobs.

Pinning a recipe version in integration.git makes it possible to "version"
recipes by using correspondingly old versions of recipes on old release
branches. That way, HEAD of recipes does not need to maintain backwards
compatibility with every integration.git release branch we wish to build and
test, making recipe development much easier.

### Recipe checkout & execution

After recipe_bootstrap has resolved a recipe version, is checks out recipes.git
at the resolved revision and execs `recipes.py luciexe` within the recipes
checkout to take over the remainder of the build execution.

## Properties

recipe_bootstrap reads several optional builder properties to determine its
behavior:

### `recipe_integration_remote`

Example: "https://fuchsia-internal.googlesource.com/integration"

If a build is triggered by a cron schedule rather than by a commit (via LUCI
Scheduler) or CL (via Commit Queue), then it won't have a `gitiles_commit` or
`gerrit_change` in its input.

For public builders it's safe to fall back to using the public integration
repository. However, many internal builders assume access to the internal
integration repository, so they need to fall back to using internal
integration instead.

`recipe_integration_remote` lets internal builders that may be triggered on a
cron job fall back to checking out the specified integration repository
(generally the internal integration repository) rather than the default.

### `recipes_host_override`

Example: "fuchsia.googlesource.com"

Some builders are triggered by commits in Git-on-Borg repositories that are
not on one of the Fuchsia-owned Git-on-Borg hosts. Only Fuchsia-owned hosts
have integration repositories, so for these builds it's not valid to try to
checkout the integration repository that's on the same host and ref as the
triggering commit.

To avoid that, a builder that's triggered by a non-Fuchsia-owned repo can
set `recipes_host_override` to hostname of the Fuchsia-owned Git-on-Borg
instance that holds the desired integration repository. If `recipes_host_override`
is set, we use the `refs/heads/main` ref for checkout because there's no
guarantee for non-Fuchsia-owned repositories that the triggering ref will
correspond to a ref of the integration repository.

### `recipes_integration_ref_override`

Example: "refs/heads/main"

This property overrides the integration.git ref that recipe_bootstrap resolves
in the case where a resolved integration base revision isn't already specified
in the build input.

Similar to `recipes_host_override`, this is useful for builders that are
triggered by non-Fuchsia-owned repositories, for which we can't assume that the
triggering ref will correspond to a valid integration.git ref.

## Testing with led

recipe_bootstrap has unit tests, and in general as much logic as possible should
be covered by unit tests. However, because it has a large interface with
Buildbucket and depends on services like Git-on-Borg, unit tests aren't
sufficient to guarantee correctness of recipe_bootstrap changes.

The easiest way to integration-test recipe_bootstrap changes is with
[`led`](http://go/luci-how-to-led). Testing recipe_bootstrap changes with `led`
is slightly more involved than testing recipe changes, because by default `led`
jobs run `recipes.py` directly instead of using `recipe_bootstrap`.

To circumvent this behavior, you need to build recipe_bootstrap locally and
upload the resulting executable to a file with the special name `luciexe` in the
led job's isolated input. This work is handled by
`scripts/led-edit-recipe-bootstrap.py`, which can be pipelined with other led
commands. For example:

```sh
led get-build 8841535313446777120 | ./scripts/led-edit-recipe-bootstrap.py | led launch
```