Recipes for building and testing

Clone this repo:
  1. a135437 Tolerate float test_timeout_secs by Nodir Turakulov · 2 days ago master
  2. 8d89295 [recipes][sdk] Remove garnet based SDK and "topaz" buckets. by Alain Vongsouvanh · 2 days ago
  3. de82c36 [autoroll] Commit untracked files before diff step by Nathan Mulcahey · 4 days ago
  4. b28f4dd [cherrypick] Handle case where repo has "/" in the name by Nathan Mulcahey · 4 days ago
  5. f0e5a35 [fuchsia_connectivity] Add failure step. by Kevin Cho · 4 days ago

Fuchsia Recipes

This repository contains recipes for Fuchsia.

A recipe is a Python script that runs a series of commands, using the recipe engine framework from the LUCI project. We use recipes to automatically check out, build, and test Fuchsia in continuous integration jobs. The commands the recipes use are very similar to the ones you would use as a developer to check out, build, and test Fuchsia in your local environment.

Setting up your environment

A recipe will not run without vpython and cipd.

Option 1: use Fuchsia binaries

Prebuilt versions of vpython and cipd are downloaded into the //buildtools subtree of a Fuchsia checkout, though they live in different directories. Where $FUCHSIA is the root of a Fuchsia checkout and $OSTYPE is linux or mac, add to your PATH:

  • $FUCHSIA/buildtools (for cipd)
  • $FUCHSIA/buildtools/$OSTYPE-x64 (for vpython)

Option 2: use Chrome's depot_tools

Chrome's depot_tools repository provides these binaries, along with other tools necessary for interacting with the Chrome source and infrastructure.

See the depot_tools Tutorial for installation instructions.

Recipe concepts

Properties

Recipes are parameterized using properties. The values for these properties can be set in the Buildbucket configuration. In the recipe code itself, they are specified in a global dictionary named PROPERTIES and passed as arguments to a function named RunSteps. The recipes engine automatically looks for these two objects at the top level of the Python file containing the recipe.

When writing a recipe, you can make your properties whatever you want, but if you plan to run your recipe on the Gerrit commit queue, there will be some standard ones starting with patch_, which give information about the commit being tested, and which you can see in the existing recipe code.

Steps

When a recipe executes, it interacts with the underlying machine by running steps.

A step is basically just a command, represented as a Python list of the arguments. You give the step a name, specify the arguments, and the recipe engine will run it in a subprocess, capture its output, and mark the job as as failed if the command fails.

Here's an example:

api.step('list temporary files', ['ls', '/tmp'])

This will execute the command ls /tmp on the machine where the recipe is running, and it will cause a failure if, for example, there is no /tmp directory. When the recipe gets run on Swarming (which is the scheduling system we use to run Fuchsia continuous integration jobs) this step will show up with the label “list temporary files” in a list of all the steps that ran.

Modules

Code is reused across recipes in the form of modules, which live either in the recipe_modules directory of this repo, or in the same directory of the recipe engine repo. The recipe engine's modules provide general functionality, and we have some modules specific to Fuchsia in this repo, such as wrappers for QEMU and Jiri.

The recipe engine looks for a list named DEPS at the top level of the Python file containing the recipe, where you can specify the modules you want to use. Each item in DEPS is a string in the form “repo_name/module_name”, where the repo name is “recipe_engine” to get the dependency from the recipe engine repo, or “infra” to get it from this repo.

Unit tests

The reason it's important to only interact with the underlying machine via steps is for testing. The recipes framework provides a way to fake the results of the steps when testing the recipe, instead of actually running the commands. It produces an “expected” JSON file, which shows exactly what commands would have run, along with context such as working directory and environment variables.

You write tests using the GenTests function. Inside GenTests, you can use the yield statement to declare individual test cases. GenTests takes an API object, which has functions on it allowing you to specify the properties to pass to the recipe, as well as mock results for individual steps.

Here's an example test case for a recipe that accepts input properties “manifest”, “remote”, “target”, and “tests”:

yield api.test('failed_tests') + api.properties(
    manifest='fuchsia',
    remote='https://fuchsia.googlesource.com/manifest',
    target='x64',
    tests='tests.json',
) + api.step_data('run tests', retcode=1)

In this example:

  • api.test simply gives the test case a name, which will be used to name the generated JSON “expected” file.
  • api.properties specifies the properties that will be passed to RunSteps.
  • api.step_data takes the name of one of the steps in the recipe, in this case “run tests”, and specifies how it should behave. This is where you can make the fake commands produce your choice of fake output. Or, as in this example, you can specify a return code, in order to cover error-handling code branches in the recipe.

To run the unit tests and generate the “expected” data, run the following command from the root of this repo:

python recipes.py test train

# Optionally specify a configuration file with --package
# (default is infra/config/recipes.cfg)
python recipes.py --package infra/config/recipes.cfg test train

The name of the recipe is simply the name of the recipe's Python file minus the .py extension. So, for example, the recipe in recipes/fuchsia.py is called “fuchsia”.

After you run the test train command, the JSON files with expectations will be either generated or updated. Look at diff in Git, and make sure you didn't make any breaking changes.

To just run the tests without updating the expectation files:

python recipes.py test run --filter [recipe_name]

To debug a single test, you can do this, which limits the test run to a single test and runs it in pdb:

python recipes.py test debug --filter [recipe_name].[test_name]

Choosing unit test cases

When you write new recipes or change existing recipes, your basic goal with unit testing should be to cover all of your code and to check the expected output to see if it makes sense. So if you create a new conditional, you should add a new test case.

For example, let‘s say you’re adding a feature to a simple recipe:

PROPERTIES = {
  'word': Property(kind=str, default=None),
}

def RunSteps(api, word):
  api.step('say the word', ['echo', word])

def GenTests(api):
  yield api.test('hello', word='hello')

And let's say you want to add a feature where it refuses to say “goodbye”. So you change it to look like this:

def RunSteps(api, word):
  if word == 'goodbye':
    word = 'farewell'
  api.step('say the word', ['echo', word])

To make sure everything works as expected, you should add a new test case for your new conditional:

def GenTests(api):
  yield api.test('hello', word='hello')
  yield api.test('no_goodbye', word='goodbye')

There will now be two generated files when you run test train: one called hello.json and one called no_goodbye.json, each showing what commands the recipe would have run depending on how the word property is set.

End-to-end testing

Unit tests should be the first thing you try to verify that your code runs. But when writing a new recipe or making major changes, you'll also want to make sure the recipe works when you actually run it.

To run the recipe locally, you need to be authorized to access LUCI services. If you've never logged in to any LUCI service (cipd, isolated, etc), login with:

cipd auth-login

Now you can run the recipe with:

python recipes.py run --properties-file test.json [recipe_name]

For this command to work, you need to create a temporary file called test.json specifying what properties you want to run the recipe with. Here's an example of what that file might look like, for the fuchsia.py recipe:

{
  "project": "garnet",
  "manifest": "manifest/garnet",
  "remote": "https://fuchsia.googlesource.com/garnet",
  "packages": ["garnet/packages/default"],
  "target": "x64",
  "build_type": "debug",
  "run_tests": true,
  "runtests_args": "/system/test",
  "snapshot_gcs_bucket": "",
  "tryjob": true,
  "$recipe_engine/source_manifest": {"debug_dir": null}
}

The last line helps prevent an error during local execution. It explicitly nullifies a debug directory set by the environment on actual bots.

If something strange is happening between re-runs, consider deleting the local work directory as is done on bots (rm -rf .recipe_deps).

Debugging

To run a test under PDB (the Python DeBugger), run: sh python recipes.py test debug --filter [recipe_name]

Developer workflow

Formatting

We format python code according to the Chrome team's style, using yapf. After committing your changes you can format the files in your commit by running this in your recipes project root: (Make sure yapf is in your PATH)

git diff --name-only HEAD^ | grep -E '.py$' | xargs yapf -i
  • --name-only tells git to list file paths instead of contents.
  • HEAD^ specifies only files that have changed in the latest commit.
  • -E enables regular expressions for grep.
  • -i instructs yapf to format files in-place instead of writing to stdout.

Naming steps

Occasionally you‘ll be confronted with the task of naming a step. It’s important that this name is informative as it will appear within the UI. Other than that, there are only two rules:

  1. Do not use the “.” character in step names. Currently that is used for indicating step nesting in the UI, but should hopefully change in the future.
  2. Step names must be unique within a single execution of a recipe. The reason for this is because recipe engine relies on unique step names for mocking out step data when testing. For this reason, step names will then be extended with a number such as “(3)” which generally isn't very useful to a reader. This is also subject to change in the future.

Existing Fuchsia recipes

See the generated documentation for our existing recipes and modules for more information on what they do and how to use them.