{% set rfcid = “RFC-0212” %} {% include “docs/contribute/governance/rfcs/_common/_rfc_header.md” %}
{# Fuchsia RFCs use templates to display various fields from _rfcs.yaml. View the #} {# fully rendered RFCs at https://fuchsia.dev/fuchsia-src/contribute/governance/rfcs #}
This RFC provides an official definition for the term “Package Set” and proposes an updated framework for package sets. This framework expresses packages sets in terms of the availability and version control they provide and accomodates:
Packages are the unit of software distribution on Fuchsia. Packages are used to deliver both platform software and product software and are also used in a range of different testing and development workflows.
Each Fuchsia product uses some mixture of product-specific packages and platform-provided packages; the content of this mixture differs from product to product. Not all the packages that a product uses need to be delivered at the same time or updated using the same strategy. For example, some packages may be downloaded as a part of an over the air update while other packages may not be downloaded until the user takes some action.
A “package set” is a collection of packages used by a product where all packages in the set are delivered and updated using the same strategy. Three different package sets are currently in use on Fuchsia: base, cache, and universe.
The current state of package sets on Fuchsia is insufficient for several reasons:
This RFC intends to achieve the following:
Facilitator:
Reviewers:
Consulted:
Socialization:
The work that led to this RFC was discussed in a series of design discussions internal to the software delivery team. Early versions of the text in this RFC were reviewed with stakeholders in the Software Assembly, Server Infrastructure, First Party Product, and PM teams.
de07b9f06ae01181b0536b61c7f643496025598c3a2908dca5c72fce191be5d1
). For this RFC we use only the former meaning.de07b9f06ae01181b0536b61c7f643496025598c3a2908dca5c72fce191be5d1
). Fuchsia does not currently require numbering (or even ordering) of package versions. The CUP metadata for some eager update packages may contain an ordered version number used in fallback checks, as defined in RFC-0145.A package set is a collection of packages where all the packages in the set are delivered and updated using the same strategy. For example, in the base package set all packages are fetched during a system update at a version that is determined by the product release. Packages in this base package set are guaranteed to be present on a device whenever the device is running.
The contents of a package set are defined by a product. A particular package may be present in one package set in product A but a different package set (or omitted entirely) in product B. The package set that a package is placed in reflects the relationship between a package and the product using it; packages that are critical to the operation of a product should be placed in a package set that guarantees these packages will always be present on the device. Typically a package will only exist in one package set for a given product, but there are cases where a package can be present in multiple package sets.
Several related but subtly different definitions of “package set” have evolved based on the context in which the term is used. We introduce some of these below, along with a name that may be used to unambiguously refer to the definition.
The remainder of this document focuses on the design of the software delivery system and therefore the “eligible package set” definition.
The software delivery system answers two key questions:
This RFC proposes a two dimensional design space for categorizing package sets based on these two questions as illustrated below:
Note that in both cases there is no one “best” answer; each axis represents a trade between two desirable properties. For some use cases one end of the axis may be the best fit while for other use cases the opposite end of the spectrum is a better fit.
Determining the current correct version for a package is critical to the security and the updatability of the system. Changing the version of a package is necessary to release new functionality and to fix bugs, yet if an entity with the power to choose the package version is compromised it may be used to deploy inappropriate or malicious versions of the software. Changing the version of a package very frequently often involves running software that is less mature or combinations of software that have not been tested together.
The version control axis is a trade between high agility and high determinism. For example:
Packages exist to enable some useful functionality on a Fuchsia device, and they can only do this when they are present (and known to be correct) on that device.
The availability axis is a trade between high availability and low resource utilization. For example:
The existing package sets and package-related RFCs fall into four different version control categories:
The existing package sets and package-related RFCs fall into four different availability categories:
Subpackage references include the content hash of the target therefore subpackages follow the version control category of their parent package. Subpackages are resolved atomically with their parent (and cannot be resolved without a context from resolving the parent) and therefore subpackages follow the availability of their parent package.
In summary, the existing package concepts fit into the proposed design space as illustrated below:
When mapping the existing package concepts into this design space as shown above, several problems are immediately visible:
This RFC proposes a series of changes that together address these problems and lead to a more comprehensive and consistent coverage of the design space as illustrated below:
Each of the changes is discussed in more detail in the following sections. Note that we introduce these in the order we intend to start them, but many changes can be conducted in parallel. Where changes depend on each other the relationship is discussed in the text.
Before this RFC, package sets were named with a single word. This was viable since there were only three sets and they lay along a single axis, but even then the terse names have caused some confusion.
This RFC proposes referring to package sets with a short phrase describing the properties they provide. Each phrase contains one term describing the availability of packages in the set and a second term describing the versioning of packages in the set, as defined in the table below:
For example, an eagerly updated package without a fallback would be referred to in full as a member of the “automatic availability, upgradable versioning” package set. In situations where context and meaning are clear this could be abbreviated to “automatic & upgradable”.
In the bottom left segment we have two distinct package sets with the same versioning and availability behavior - base and bootfs. We will retain these names to refer to those package sets while using “permanent availability, anchored versioning” to refer to the more general behavior they share.
We will retain the existing “cache” name to refer to the cache package set until this set is replaced (see change 6).
The word “universe” has conflicting definitions on Fuchsia. The original definition was the set of all built packages which was consistent with the set theory meaning of the term. Today more commonly “universe” refers to only those packages that are built but not included in another package sets, i.e. base packages are not a part of universe. This definition is inconsistent with the set theory meaning of the term. To avoid this confusion we use the name “discoverable” rather than “universe” when referring to the top right segment of the design space.
The cache package set and eagerly updated packages with fallback are two different solutions that address the same need: let a version control server provide newer versions of a package while ensuring that some version of the package remains available, even when the device is offline.
Eagerly updated packages as defined in RFC-0145 provide a more robust solution to this problem:
Cache packages are only used in developer flows at the moment but (due to the problems above and others) they are not a very good fit for these use cases. Given the advantages of eagerly updated packages it will never be desirable to use cache packages in production flows.
We will deprecate the cache package concept, meaning SWD will not invest in further improvements to the flow. We will neither remove the flow nor the developer documentation until a suitable replacement has been deployed and adopted (see change 6).
Note that deprecating cache packages does not mean deprecating the similarly named “package cache” component.
RFC-0154 required that subpackages be placed “in the same package set as” their parent. This requirement would become difficult to meet as we introduce new package categories because it would require that the product maintainer update the package set definition every time the set of transitive subpackages changed.
Implementation work since RFC-0154 has already loosened the same package set constraint but this RFC officially removes it. As discussed above, subpackages still follow the availability and version control of their parent (and therefore still effectively behave as members of their parent's package set) but product maintainers no longer need to independently include subpackages as top-level members of the package sets that reference them.
Security audit tools are used to verify that each release of a product contains the expected set of packages. The golden files used for this verification will include all the subpackages included in anchored version package sets, transitively. This ensures that any addition of a subpackage or nested subpackage to an anchored version package set will receive a human review.
Fuchsia's current package garbage collection (GC) algorithm was designed very early in the program and has not scaled well as software delivery evolved. Even today we experience failure modes where necessary packages are deleted and where unnecessary packages are retained. The current GC algorithm was not equipped to handle the introduction of new package sets.
We will publish the requirements and algorithm for a replacement GC implementation that can accommodate more package sets on a follow-on RFC in the near future.
During each release of a product the product maintainers must test fixed versions of many different platform and product packages together to ensure they integrate correctly. Today, the only way for a product to include a fixed version of a package is to include it in the base package set. This means that two copies of all these packages may need to be stored on disk concurrently during a system update, making it challenging to run Fuchsia on devices with limited storage space.
This problem can be solved by implementing an automatic availability, anchored version package set, i.e. supporting packages whose version is fixed by the product but that are not downloaded until after a system update has been successfully committed. We will publish a design for this package set as a follow-on RFC in the near future.
As discussed in change 2 above, the cache package flow is not a good fit for the developer need to easily, quickly, and reliably update packages during development. We will research a better solution by focussing on the developer needs, then publish a design as a follow-on RFC.
We expect that this new developer-focused flow discussed will act as an overlay on the design space in engineering builds, but this will be explored further in the follow-on RFC.
These package sets would let product maintainers define packages with either a fixed version or a constrained version that are not fetched until they are needed.
These package sets are potentially useful for future products but, unlike the automatic & anchored package set in change 5, we do not yet have a launch customer.
We will design and implement one or both of these package sets when a specific product or developer need arises.
Once these changes are complete, including the proposed follow-on RFCs, the software delivery stack will offer significantly more flexibility to product developers. For example:
The changes described in this RFC also have significant implications for assumptions inside the SWD stack. For example:
The initial implementation of this RFC will be mainly documentation changes covered in more detail below. The names of the most heavily used package sets (i.e. base, bootfs, and cache) are not changed by this RFC. We will update references to “universe” with “discoverable” in source code and tooling as opportunities arise. Where deprecated source code and tools use the term “universe” we will not generally not update these references.
Proposed changes 4, 5, and 6 require follow-on RFCs and these RFCs will document the implementation of the changes they introduce.
In the long term this RFC will reduce the size of the base package set, with some packages currently in base moving to the newer package sets with lower resource requirements. Over the long term this will improve several performance metrics, particularly disk and network utilization.
The latency to resolve a package may increase if a product chooses to move the package from a package set with permanent availability to one with automatic or on demand availability.
The net security impact of this RFC is believed to be positive:
The net privacy impact of this RFC is believed to be neutral:
Proposed changes 4, 5, and 6 require follow-on RFCs and these RFCs will document the testing of the changes they introduce.
Upon acceptance of this RFC we will:
The RFC introduces a change in nomenclature for universe packages and eagerly updated packages. Any naming change incurs some cost in developer confusion, but these terms were not used by many developers and were not used consistently previously. We believe the confusion of a name change will be outweighed by the long term usability of names that are logical and self consistent.
The proposed changes section above marks the lower right section of the design space “not applicable”. The reason we do not support this combination is we argue it is illogical: in order for a package to be guaranteed or loaded automatically it must be explicitly specified by the product release at which point it meets the criteria for inclusion in an upgradeable package set.