{% set rfcid = “RFC-0100” %} {% include “docs/contribute/governance/rfcs/_common/_rfc_header.md” %}
Fuchsia Software Development Kits (SDKs) currently ship so called device profile metadata (or, in this document, simply the metadata), whose schema is defined in //build/sdk/meta/device_profile.json
. The metadata is intended to describe system images and packages that can be installed on a device. We have found that the schema is incomplete and needs modernization. In addition, tools outside the SDK now need the metadata, increasing its importance.
This RFC introduces a new schema for this information as a contract between the platform and tools along with a process to update that schema in the future.
Currently, the metadata is shipped with SDKs. It currently contains the name of a device, a text description of that device, and URIs from which a user can download image and package information. We have found that this metadata is both difficult to keep up-to-date and incomplete, making it unusable.
Tools end up relying on unofficial conventions about images and packages, or hard-coding assumptions that may change (such as the structure of URIs for downloads). This makes the tools susceptible to breakages when these conventions change.
We wish to resolve these problems. We will specify a formal schema for the metadata, so that tools can consume it.
When we distribute metadata that is both complete and easy to use, SDK users will be able to choose which image they want to flash to their device or boot with an emulator without significant help from Fuchsia's infrastructure or SDK teams. SDK developers will have a single source of truth for how they surface information about products to users. Tool developers will have a well-defined contract for how they consume this information.
The metadata is currently hard coded as part of SDK generation that needs to be updated by hand whenever we publish a new image. This makes adding, renaming or removing metadata laborious and error prone. The information regularly falls out of date - or worse, is never added at all.
In practice, because SDK consumers cannot rely on the accuracy of metadata shipped with the SDK, they hard code the location of packages and images, as well as other brittle implementation-dependent information, into their tools. For example, in addition to the hard coding of URLs, the out-of-tree emulator boot relies on an undocumented images.json
file's presence in the system image.
This proposal provides a new schema for our metadata that we expect external users to rely upon. This will provide motivation for us to keep it up to date.
The existing device metadata is missing crucial detail. For example, tools for flashing a device require lists of partition names, as well as file names to be written into those partitions. This is not currently formalized, so we have no contract between what we distribute and tools that perform flashing. This makes it challenging to provide stability / compatibility guarantees.
When you combine the lack of completeness of the schema with the difficulty of updating the metadata, the problem is compounded. For example, consider the hypothetical example of an emulator tool that wants to make sure that it starts an emulator with the same architecture as the image the user wants to run. Without attempting to boot the image, how is the tool supposed to know whether any image in a particular location is compatible with a particular architecture? With an updated schema, we can have a single set of metadata that describes both the architecture of an image and its location. The tool can then simply choose the correct architecture for a given image.
This proposal will outline an initial schema for our metadata, and provide a path forward for updates to ensure that they can be kept relevant.
In order to provide useful, up-to-date glossary.product metadata with our SDKs, we will:
Note that, while we define an initial schema here, we will allow evolution to take place through API-Council review, without additional RFCs.
For the purposes of this document, we use the following definitions:
We will use JSON to represent our metadata. Schemata are expressed in the Draft 7 format defined at json-schema.org.
This section details the initial requirements and initial changes we will make to the schemata.
gs
, http(s)
and file
URI schemes. The file
URIs are required for local access.There are a number of possible future directions for evolution of the schema. For example:
To account for this, the schema is versioned. The version is a string that is updated whenever an element of the schema is added or removed. Tools may target multiple different versions of the schema, or users may need to resort to older versions of tools to use older schemata.
To ease the transition between versions we propose a flexible parsing and versioning scheme. We introduce a versioned envelope that simplifies the parsing of JSON. A parsing tool can read the version field while ignoring the rest of the JSON document. This allows the tool to choose a correct parser for each supported version of the schema.
We propose to version each schema file independently using a randomly generated 8 digit hexadecimal number. The number must be unique across all existing schema files. To guard against collisions, it is advisable to implement a build-time collision detection to prevent accidental duplicate submissions.
Since a schema may include other schemas via the $ref
facility, we propose to append the version of the schema to the schema file name. This will not only disambiguate between multiple versions of common files such as //build/sdk/meta/common.json
, it will also trigger the version recomputation of the referrer schema.
Each incompatible schema change such as adding or removing a required field will trigger a version recomputation.
We allow schemata to be changed via the API review process. Changes to the schema fall under the Developer functional area.
The implementation of the schema itself is largely a matter of adding the schema to fuchsia.git and ensuring that it gets API reviewed. There are two other questions: where is the metadata generated? and where is the metadata consumed?
Device metadata will initially be generated:
fuchsia.git
;It may also be generated in other places, especially for images and packages published outside of an SDK release.
Device metadata will largely be consumed by the ffx
tool, which will use them to flash devices and launch emulators. Because we will publish this schema with SDKs, tools outside of the Fuchsia tree may also consume metadata that adhere to it.
With this approach, all flashing and emulation workflows, whether for in-tree or out-of-tree use, will use device metadata for specification.
Both of these efforts require design beyond the scope of this RFC.
We will extend the common.json schema definitions as follows. There is no need to account for backwards incompatibility, because this is not an incompatible change. The existing definitions in common.json are omitted below for brevity.
For brevity, we omit the original contents of the file, using ...
to indicate an omission.
/** build/sdk/meta/common.json */ { "$schema": "http://json-schema.org/draft-07/schema#", "id": "http://fuchsia.com/schemas/sdk/common.json", "definitions": { ..., "envelope": { "description": "A versioned envelope.", "type": "object", "properties": { "version": {"$ref": "#/definitions/version"}, "data": { "description": "The payload." } }, "required": [ "version", "data" ], "additionalProperties": false }, "versioned_sdk_element": { "type": "object", "allOf": [ {"$ref": "#/definitions/envelope"}, { "type": "object", "properties": { "data": { "oneOf": [ {"$ref": "#/definitions/sdk_element"} ] } } } ] }, "version": { "description": "An opaque version string. The string may include any characters. Tools must not attempt to draw any conclusions about inter version compatibility other than the version 'X' manifest complies with the version 'X' of the schema and is therefore parsable by the version 'X' parser. There are no guarantees, for example, a parser for version 'B' may be able to parse a JSON document versioned 'A'.", "type": "string", "minLength": 1 }, } }
Note that, because this schema defines what version of the schema we are using, it cannot be easily changed. Tools will have to read files conforming to this schema to understand the enveloped JSON.
Since multiple images may be installed on a device, we factor out the device hardware specification into its own definition. The device hardware specification serves two purposes:
We introduce two new versioned SDK elements for physical and virtual devices.
At the moment, we only care about the CPU architecture of a physical device. Additional hardware attributes will be added in the future versions of the schema. For the purposes of providing a complete example, the schema definition below combines the envelope schema with the sdk_element
schema[^1].
[^1]: All metadata is distributed as SDK elements.
/** build/sdk/meta/physical_device-c906d79c.json */ { "$schema": "http://json-schema.org/draft-07/schema#", "id": "http://fuchsia.com/schemas/sdk/physical_device-c906d79c.json", "description": "A physical device specification.", "type": "object", "allOf": [ {"$ref": "common.json#/definitions/envelope"}, { "type": "object", "properties": { "data": { "allOf": [ {"$ref": "common.json#/definitions/sdk_element"}, { "properties": { "type": { "allOf": [ {"$ref": "common.json#/definitions/type"}, {"enum": ["physical_device"]} ] } } }, {"$ref": "hardware-f9928aa4391e2ae3644ce712074a1ef7.json#/definitions/requirements"} ] } } } ] }
This schema allows us to represent device hardware requirements. This allows tools to ensure that the device hardware matches the artifacts we will use for provisioning. Currently, it only describes the CPU architecture.
/** build/sdk/meta/hardware-c0a116ca.json */ { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "requirements": { "properties": { "hardware": { "additionalProperties": false, "properties": { "cpu": { "additionalProperties": false, "properties": { "arch": { "oneOf": [ {"$ref": "common.json#/definitions/target_arch"} ] } }, "required": ["arch"], "type": "object" } }, "required": ["cpu"], "type": "object" } }, "required": ["hardware"], "type": "object" }, }, "description": "Hardware requirements for running a product image.", "id": "http://fuchsia.com/schemas/sdk/hardware-c0a116ca.json" }
The virtual device specification allows us to select the appropriate product bundle for booting the emulator. In the future revisions, the schema will be developed further to configure the emulator. For the sake of an example, the schema definition below uses a newly defined versioned_sdk_element
schema.
/** build/sdk/meta/virtual_device-8a8e2ba9.json */ { "$schema": "http://json-schema.org/draft-07/schema#", "id": "http://fuchsia.com/schemas/sdk/virtual_device-8a8e2ba9.json", "description": "A virtual device specification for launching emulators.", "type": "object", "allOf": [ {"$ref": "common.json#/definitions/versioned_sdk_element"}, { "type": "object", "properties": { "data": { "type": "object", "properties": { "type": { "allOf": [ {"$ref": "common.json#/definitions/type"}, {"enum": ["virtual_device"]} ] }, "description": {"type": "string"}, "virtual": {"$ref": "virtual_hardware-4c5d1a5d.json#/definitions/spec"} }, "required": ["virtual"] } } } ] }
The virtual hardware specification is used for selecting the appropriate product bundle when launching the emulator.
/** build/sdk/meta/virtual_hardware-4c5d1a5d.json */ { "$schema": "http://json-schema.org/draft-07/schema#", "id": "http://fuchsia.com/schemas/sdk/virtual_hardware-4c5d1a5d.json", "description": "A virtual device specification for launching emulators.", "definitions": { "spec": { "emu": { "type": "object", "properties": { "cpu": { "type": "object", "properties": { "arch": { "oneOf": [ {"$ref": "common.json#/definitions/target_arch"} ] } }, "required": ["arch"], "additionalProperties": false } }, "required": ["cpu"], "additionalProperties": false } }, "required": ["emu"] } } }
The images and packages bundle provides access to the software artifacts installed on a device.
/** build/sdk/meta/product_bundle-514c2856.json */ { "$schema": "http://json-schema.org/draft-07/schema#", "id": "http://fuchsia.com/schemas/sdk/product_bundle-514c2856.json", "description": "Artifacts required to boot and run software on a device.", "type": "object", "allOf": [ {"$ref": "common.json#/definitions/versioned_sdk_element"}, { "type": "object", "properties": { "data": { "type": "object", "properties": { "type": { "allOf": [ {"$ref": "common.json#/definitions/type"}, {"enum": ["product_bundle"]} ] }, "description": { "description": "A human readable description of the product bundle.", "type": "string" }, "device_refs": { "description": "A list of physical or virtual device names this product can run on.", "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }, "metadata": { "description": "A list of key-value pairs describing product dimensions. Tools must not rely on the presence or absence of certain keys. Tools may display them to the human user in order to assist them in selecting a desired image or log them for the sake of analytics. Typical metadata keys are: build_info_board, build_info_product, is_debug.", "$ref": "common.json#/definitions/key_value_list" }, "manifests": { "description": "Manifests describing how to boot the product on a device.", "flash": {"$ref": "flash_manifest-c85dbd8e.json#/definitions/manifest"}, "emu": {"$ref": "emu_manifest-b0708439.json#/definitions/manifest"} }, "images": { "description": "A list of system image bundles. Each image bundle must be equivalent to all others. I.e., all image bundle URIs are effectively mirrors of each other. Their formats may vary. Pick an entry that best suits your needs.", "type": "array", "minItems": 1, "items": {"$ref": "#/definitions/image_bundle"} }, "packages": { "description": "A list of package bundles. Each package bundle must be equivalent to all otherwise. I.e., all package bundle URIs are effectively mirrors of each other. Their formats may vary. Pick an entry that best suits your needs.", "type": "array", "minItems": 1, "items": {"$ref": "#/definitions/package_bundle"} } }, "required": ["device_refs", "manifests", "images", "packages"] } } } ], "definitions": { "image_bundle": { "description": "A set of artifacts necessary to provision a physical or virtual device", "type": "object", "properties": { "base_uri": { "description": "A base URI for accessing artifacts in the bundle.", "$ref": "#/definitions/bundle_uri" }, "format": { "description": "Bundle format: files - a directory layout; tgz - a gzipped tarball. In case of the 'files' format, the base base path points to a directory. The manifest paths is relative to the directory. To get a full path append a path inside the manifest to base_uri. In case of the 'tgz' format, the base path points to the archive. The manifest path is relative within the archive." "enum": [ "files", "tgz" ] } }, "required": [ "base_uri", "format" ], "additionalProperties": false }, "package_bundle": { "description": "A set of artifacts necessary to run a physical or virtual device", "type": "object", "properties": { "repo_uri": { "description": "A package repository URI. This may be an archive or a directory.", "$ref": "#/definitions/bundle_uri" }, "format": { "description": "Repository format: files - a directory layout; tgz - a gzipped tarball.", "enum" : [ "files", "tgz" ] }, "blob_uri": { "description": "An optional blob repository URI. If omitted, it is assumed to be <repo_uri>/blobs. If repo_uri refers to a gzipped tarball, ./blobs directory is expected to be found inside the tarball.", "$ref": "#/definitions/bundle_uri" } }, "required": [ "repo_uri", "format" ], "additionalProperties": false }, "bundle_uri": { "description": "Allowed system image and package bundle URIs.", "type": "string", "format": "uri", "pattern": "^(?:http|https|gs|file):\/\/" } } }
Manifests based on the flash schema describe image artifacts necessary to flash the device. The manifest matches the existing flash.json produced by the build. The flash metadata will be embedded in the product bundle SDK element.
/** build/sdk/meta/flash_manifest-c85dbd8e.json */ { "$schema": "http://json-schema.org/draft-07/schema#", "id": "http://fuchsia.com/schemas/sdk/flash_manifest-c85dbd8e.json", "description": "A manifest that describes how to flash a device.", "type": "object", "properties": { "manifest": {"$ref": "#/definitions/manifest"} }, "required": ["manifest"], "additionalProperties": false, "definitions": { "manifest": { "description": "A named list of partitions and OEM files necessary to flash a device.", "type": "object", "properties": { "hw_revision" : { "description": "A board name used to verify whether the device can be flashed using this manifest.", "type": "string" }, "products": { "description": "A list of product specifications that can be flashed onto the device.", "type": "array", "items": {"$ref": "#/definitions/product"}, "minItems": 1 } }, "required": ["hw_revision", "products"], "additionalProperties": false }, "product": { "description": "A named product specification.", "type": "object", "properties": { "name": { "description": "A unique name of this manifest.", "type": "string" }, "partitions": { "description": "A list of partition names and file names corresponding to the partitions.", "$ref": "common.json#/definitions/key_value_list" }, "bootloader_partitions": { "description": "A list of partition names and file names corresponding to the partitions.", "$ref": "common.json#/definitions/key_value_list" }, "oem_files": { "description": "A list of OEM command and file names corresponding to the command.", "$ref": "common.json#/definitions/key_value_list" } }, "required": ["name", "partitions"], "additionalProperties": false } } }
The emulator schema specifies image artifacts necessary to boot the product on the emulator.
/** build/sdk/meta/emu_manifest-b0708439.json */ { "$schema": "http://json-schema.org/draft-07/schema#", "id": "http://fuchsia.com/schemas/sdk/emu_manifest-b0708439.json", "definitions": { "manifest": { "description": "A manifest that describes how to boot an emulator.", "type": "object", "properties": { "kernel": { "description": "A path to the kernel image file. The path is relative to the image bundle base.", "type": "string", "minLength": 1 }, "initial_ramdisk": { "description": "A path to the initial ramdisk, the kernel ZBI. The path is relative to the image bundle base.", "type": "string", "minLength": 1 }, "disk_images": { "description": "A list of one or more disk image paths to FVM images. Each path is relative to the image bundle base.", "type": "array", "items": { "type": "string", "minLength": 1, }, "minItems": 1 } }, "required": ["kernel", "initial_ramdisk", "disk_images"], "additionalProperties": false } } }
Below are extra additions to the common definitions. See the section on common.json for version related additions. The existing definitions in common.json are omitted below for brevity.
/** build/sdk/meta/common.json */ { "$schema": "http://json-schema.org/draft-07/schema#", "id": "http://fuchsia.com/schemas/sdk/common.json", "definitions": { ..., "key_value": { "description": "A key-value pair.", "type": "array", "items": [ {"type": "string"}, {"type": ["number", "string"]} ], "minItems": 2, "additionalItems": false }, "key_value_list": { "description": "A list of key-value pairs.", "type": "array", "items": {"$ref": "#/definitions/key_value"} }
Below is an example of device hardware specifications.
{ "version": "c906d79c", "data": { "type": "physical_device", "name": "generic-x64", "description": "A generic x64 device.", "hardware": { "cpu": { "arch": "x64" } } } }
This is an example of a generic terminal product bootable on a generic x64 device containing two archived mirrors for both images and packages accessible via gs
and https
based URIs respectively.
{ "version": "514c2856", "data": { "type": "product_bundle", "name": "generic-x64", "description": "A terminal x64 product.", "device_refs": ["generic-x64"], "metadata": [ ["build-type", "release"], ["product", "terminal"] ], "manifests": { "flash": { "hw_revision": "x64", "products" : [ { "bootloader_partitions": [], "name": "fuchsia", "oem_files": [], "partitions": [ ["", "fuchsia.zbi"], ["", "zedboot.zbi"], ["", "fuchsia.vbmeta"], ["", "zedboot.vbmeta"] ] } ] } }, "images": [ { "base_uri": "gs://fuchsia/development/0.20201216.2.1/images/generic-x64.tgz", "format": "tgz" }, { "base_uri": "https://storage.googleapis.com/fuchsia/development/0.20201216.2.1/images/generic-x64.tgz", "format": "tgz" } ], "packages": [ { "repo_uri": "gs://fuchsia/development/0.20201216.2.1/packages/generic-x64.tar.gz", "format": "tgz" }, { "repo_uri": "https://storage.googleapis.com/fuchsia/development/0.20201216.2.1/packages/generic-x64.tar.gz", "format": "tgz" } ] } }
This is an example of a generic terminal product bootable on an x64 emulator. Both images and packages are accessible directly from GCS.
{ "version": "514c2856", "data": { "type": "product_bundle", "name": "generic-x64", "description": "A terminal x64 product.", "device_refs": ["qemu-x64"], "metadata": [ ["build-type" , "debug"], ["product", "terminal"] ], "manifests": { "emu": { "kernel": "qemu-kernel.kernel", "initial_ramdisk": "zircon-a.zbi", "disk_images": ["storage-sparse.blk"] } }, "images": [ { "base_uri": "https://storage.googleapis.com/fuchsia-artifacts/builds/8852232026486839104/images/", "format": "files" } ], "packages": [ { "repo_uri": "https://storage.googleapis.com/fuchsia-artifacts/builds/8852232026486839104/packages/", "blob_repo_uri": "https://storage.googleapis.com/fuchsia-artifacts/blobs/", "format": "files" } ] } }
This is unlikely to have an effect on performance because metadata generation and parsing is not resource intensive.
Publishing standardized metadata should make integrations simpler by removing the need for brittle hard-coded configuration.
We choose JSON because it is a well established format, and is the de facto standard for Fuchsia configuration files. The JSON schema format is formalized and peer reviewed.
Introduction of metadata does not affect backwards compatibility, as there are no current users of the existing metadata.
Future revisions of these schemata will be API reviewed and change managed using a standard LSC process involving testing and direct interaction with high priority customers. This process will be aided by versioning the schema definitions.
Metadata based on these schemata are no less secure than the existing solution, which is an ad hoc hard-coding of similar information.
We will increase the security of artifact delivery early in the life cycle of these schemas. Depending on in-progress feedback from the security team, this may be prior to deployment. Critically, tools must verify the integrity of the contents of the downloaded artifacts: the artifacts must be what was intended when the metadata was published. This can be accomplished either by adding the output of a hash function to the schema, or by using a protocol that supports integrity checks, such as TUF.
Tools vended as part of a Fuchsia SDK that consume this schema and fetch artifacts must implement additional security measures to ensure transport security, to prevent attacks such as person-in-the-middle. They may do this by refusing to fetch or use artifacts from insecure / untrusted locations.
This proposal has no known privacy considerations. There may be privacy considerations for people or teams that publish metadata; we expect them to conduct their own privacy review. For example, teams that publish metadata may need to consider carefully how they track downloads.
Although the existence of a schema does not need testing, we will ensure that metadata produced by our build meets the schema. We will also have compatibility testing to ensure that tools continue to consume older schema as necessary.
No documentation is currently planned outside of the schema itself, although that will likely change with demand. A developer creating or updating a schema will likely need to provide a glossary of terms, descriptions of how each schema is used, and how they interact with each other.
We are committing to a particular API for specifying device images. This has the advantage of increasing compatibility across versions, leveraging existing change management processes, and providing us the ability to distribute tools that work consistently, but the disadvantage of decreased flexibility (it takes longer to make a change to the schema because of bureaucratic hurdles).
This document collects and systematizes common configurations throughout the tree, which are mentioned above. Much of the existing metadata can be found in //build/images/BUILD.gn
; for example, the current version of the flashing schema can be found there.