The Fuchsia project uses Chrome Infrastructure Package Deployment (CIPD{: .external}) to store and distribute prebuilt files.
A CIPD package is an arbitrary collection of files, stored on a remote content-addressed store{: .external}, and distributed to a Fuchsia checkout through the jiri{: .external} tool. It is also posible to download them directly using the cipd command-line tool (e.g. to verify their content).
Once you set up continuous integration (CI) with Fuchsia, whenever you publish new versions of the prebuilt packages to CIPD, Fuchsia’s CI system fetches those new packages and rolls them into Fuchsia through the global integration process.
CIPD packages are typically used to distribute:
This page explains how to integrate your software into the Fuchsia project as a prebuilt package, i.e. how to publish it to CIPD, and how to update the Fuchsia Jiri manifests to ensure it is available to Fuchsia platform developers and Fuchsia continuous integration (CI).
Note: CIPD is not a package repository for Fuchsia devices. A running Fuchsia device doesn't install prebuilt packages from CIPD.
If you plan on publishing a prebuilt FAR, you need to know how to build a prebuilt package.
If your prebuilt package contains ELF binaries (whether inside a FAR or not), they need to be stripped. For Fuchsia ELF binaries, a companion symbols package also needs to be generated and uploaded, to support symbolization and debugging.
Each CIPD package has a path on the CIPD store. Path naming convention depends on whether the prebuilts are generated and/or maintained by the Fuchsia team itself.
For a CIPD package contributed by an external team or project:
Use the following CIPD path naming convention:
<PROJECT>/fuchsia/<PACKAGE>-<CIPD_TARGET_ARCHITECTURE>
Where <CIPD_TARGET_ARCHITECTURE> is a CIPD-compatible CPU architecture name (e.g. amd64 for 64-bit Intel binaries, not x64 or x86_64). For example, chromium/fuchsia/webrunner-arm64{: .external} and chromium/fuchsia/castrunner-amd64{: .external}.
Include a LICENSE file that contains the legal notices of the software.
Provide a Fuchsia archive (FAR) file per prebuilt package. For example, chromium.far, webrunner.far.
For prebuilts that are generated or maintained by the Fuchsia project however, the following path conventions apply:
Use fuchsia/ as the root project directory for all prebuilt packages.
Prebuilts generated directly from third-party open-source projects need to use a path under fuchsia/third_party/.
E.g. fuchsia/third_party/ninja/ contains several packages for prebuilt binaries of the Ninja build tool, while fuchsia/sdk/core contains several packages corresponding to Core Fuchsia SDK archives generated by the Fuchsia team.
Host prebuilts should use <CIPD_OS>-<CIPD_HOST_ARCHITECTURE> as their last directory entry, where <CIPD_OS> is a CIPD-compatible OS name, and <CIPD_HOST_ARCHITECTURE> a CIPD-compatible CPU architecture name
This means linux or mac for <CIPD_OS> (but not Linux, osx, darwin or mach), and also i386, amd64 or arm64 for <CIPD_HOST_ARCHITECTURE> (but not x86, x64 or aarch64).
E.g. fuchsia/third_party/clang/mac-amd64 is the path used for the CIPD package that contains the 64-bit Intel OS X Clang toolchain binaries.
Fuchsia ELF prebuilts should use a path that ends with arch/<ARCH>, where <ARCH> is a Fuchsia-compatible CPU architecture name (e.g. x64 or arm64, not amd64, x86_64 or aarch64).
E.g. fuchsia/third_party/swiftshader/arch/x64 is the path used for the SwitfShader-based Vulkan ICD/driver, built for Fuchsia.
Note that in the case of 64-bit Intel binaries, amd64 is only used for host prebuilts, while x64 is only used for Fuchsia ones.
Fuchsia developers need to be able to identify which source code is used to generate a CIPD package instance. Fuchsia recommends that your project‘s documentation provides instructions on how to obtain the sources that match a given package instance’s version identifier.
To identify the source of prebuilt packages, tag{: .external} each revision with a version identifier in the form of:
version:<VERSION_ID_OF_INSTANCE>
For example, version:77.0.3835.0 and version:176326.
Additionally, when available (e.g. building third-party open-source projects), use:
git_revision:<GIT_COMMIT_HASH>
This git_revision tag identifies the exact Git commit of the source tree used to build the binaries.
If several revisions are different but built from the same Git hash (e.g. because the first one was mistakenly built in debug, instead of release mode), it is usual to use a suffix separated with a dash, as in git_revision:<commit>-2, for the new revision's tag.
When you publish a new revision of your CIPD package, the latest ref{: .external} automatically point to the new revision. Fuchsia’s CI system monitors your package’s latest ref; when it detects that it was updated, the system fetches the new package and rolls it into Fuchsia.
Note: Whenever possible, CIPD packages should be built and uploaded using LUCI Recipes{: .external}, which guarantee hermetic and reproducible builds, running on securely segregated bots. Follow the content of this section only when this is not possible.
Uploading a new CIPD package revision manually requires to be authentified with cipd auth-login, then to run the cipd create command.
Example instructions are available from this CIPD page{: .external}. Using a cipd.yaml file is strongly recommended. A good practice is to auto-generate it from your rebuild script, and include uploading instructions in a comment, as in this pseudo-example:
# This file was auto-generated by build-project-fuchsia-package.sh # For more information, see http://myproject.url/docs/fuchsia/ # # To upload the files in this CIPD package do: # # TAG=git_revision:cd13435a975057b0ad0b72636f6c8323113c9c8b # VERSION=version:22.0.773.345 # cipd create -pkg-file cipd.yaml -tag $TAG -tag $VERSION # package: myproject/fuchsia/mypackage-arm64 description: mypackage's Fuchsia binaries from myproject. data: - file: LICENSE - file: libpackage.so
Don't forget to publish a companion symbols package if your package contains Fuchsia ELF binaries.
Once a prebuilt package has been uploaded to CIPD, it can be made available to Fuchsia team developers by updating a Jiri manifest, which is listed under ${FUCHSIA_DIR}/integration/. To do so:
Locate the appropriate manifest file to modify. For instance prebuilts are listed in integration/fuchsia/prebuilt if you use the Google-internal repository checkout. Location might be different for an open-source checkout.
Determine the Jiri checkout path. This should be under ${FUCHSIA_DIR}/prebuilt/ with a path reflecting the CIPD one. For example ${FUCHSIA_DIR}/prebuilt/third_party/ninja/linux-x64/ is used for the package at CIPD path fuchsia/third_party/ninja/linux-amd64. Note that they use different conventions for the CPU architecture.
Add or update the Jiri manifest entry appropriately, looking at other existing manifests for inspiration. Here's the one for the QEMU emulator binary:
Note that in the example above:
The name attribute points to the CIPD package path. And ${platform} will expand to a string matching the CIPD-compatible host platform name (e.g. linux-amd64).
The version attribute points to a CIPD tag for the package's revision to download.
The path attribute points to the checkout path under ${FUCHSIA_DIR}, and uses a special pattern that will expand to a Fuchsia-compatible system and CPU architecture names for the developer's or infra bot machine (e.g. linux-x64).
The platforms attribute is optional and restricts the list of supported build platforms (Jiri will not download this specific package on unlisted systems).
Update the Jiri lock files if necessary. This applies to the Google-internal checkout. Invoke the integration/update-lockfiles.sh. This script will update a couple of necessary jiri.lock files.
Create a new branch under integration (or whatever directory contains your Jiri manifest for prebuilts), that includes your manifest change and jiri.lock changes, if any, and upload it to Gerrit as usual for review.
A CIPD prebuilt package that contains Fuchsia ELF binaries should also have a companion symbols package that contains the debug information for the binaries. These are using for symbolization in logs, crashes and for debugging.
Each ELF binary in the original CIPD package must include an ELF NT_GNU_BUILD_ID note section, which includes a unique build-id hash value that uniquely identifies its non-debug-related content.
This note is created at link time when producing the ELF binary. Recent versions of GCC, or the prebuilt Fuchsia Clang toolchain, produce it automatically (however, regular Clang requires passing a special linker flag (i.e. -Wl,--build-id).
Note: To print the build-id of an existing library, use either one of file <LIBRARY> or readelf -n <LIBRARY> | grep "Build ID"
The symbols package must use the directory layout typical of .build-id directories used to store debug information in separate files{: .external}. This means that each file must be stored as <xx>/<xxxxxxxxxx>.debug, where <xx> and <xxxxxxxxx> are hex strings derived from the build-id hash value, of the unstripped binary. Each such file should match one stripped ELF binary with the same build-id from the original package.
Note: The symbols package should not include a top-level .build-id directory.
The symbols package must use the same version identifiers (tags) as the original CIPD package they refer to. This allows them to be rolled together.
If several CIPD packages that contain stripped ELF binaries are rolled together (using the same version identifiers), then grouping the debug symbols for all of them in a single CIPD symbols package is acceptable, but not required.
The CIPD path for the symbols package should include the -debug-symbols-<ARCH> suffix, E.g. myproject/fuchsia/mypackage-debug-symbols-amd64 contains the symbols for the myproject/fuchsia/mypackage-amd64 prebuilt package.
The Jiri checkout path for all symbols package must be ${FUCHSIA_DIR}/prebuilt/.build-id.
This example shows the directory structure of a CIPD symbols package with unstripped ELF binaries:
1d/ bca0bd1be33e19.debug 1f/ 512abdcbe453ee.debug 90dd45623deab1.debug 2b/ 0e519bcf3942dd.debug 3d/ aca0b11beff127.debug 5b/ 66bc85af2da641697328996cbc04d62b84fc58.debug
The usual way to generate this package is to:
Compile all your ELF binaries with DWARF debug information (e.g. passing the -g flag to the compiler, even in release mode).
Ensure the ELF binaries include an NT_GNU_BUILD_ID note. This is the default on recent GCC versions and the Fuchsia prebuilt Clang toolchain, but regular Clang requires passing a special flag to the linker (i.e. -Wl,--build-id).
It is possible to generate the stripped binary and the corresponding build id directories with a single call to llvm-objcopy as in:
# Copy out/libfoo.so to symbols/<xx>/<xxxxxxx>.debug according to its # `build-id` value (requires the library to be linked with -Wl,--build-id when # using Clang), and also copy the stripped version of the library to # stripped/libfoo.so # # NOTE: To strip executables, instead of libraries, replace --strip-all below # with --strip-sections # UNSTRIPPED_LIB=out/libfoo.so STRIPPED_LIB=stripped/libfoo.so SYMBOLS_DIR=./symbols llvm-objcopy --strip-all \ --build-id-link-dir="${SYMBOLS_DIR}" \ --build-id-link-input=.debug \ "${UNSTRIPPED_LIB}" "${STRIPPED_LIB}"
Repeat as many times as necessary to populate the symbols/ directory, then upload its content (i.e. files under symbols/) as your symbols package.
Do not forget to copy the content of the stripped/ directory to your prebuilt CIPD package as well.
