docs/contribute/roadmap/2021/product_assembly.md

# Scaling Product Assembly and Configuration

Project lead: aaronwood@google.com
Area(s): Build, Developer

## Problem statement

[Product][glossary.product] assembly, the process of creating an "image" out of
the built software and configuration data in the Fuchsia platform (fuchsia.git)
and the product's own repo(s), is currently restricted to a compilation-time
operation that's part of fuchsia.git.  It can only be done "in-tree" after the
compilation steps for all software are completed.

There has been a continual explosion of product configurations, across multiple
dimensions (`_eng`, `_eng_arrested`, `_user`, `_userdebug`, LSDi, etc), to
allow for the ability to name each of the configurations that developers,
testers, and customers need.

Product Owners and Developers still cannot easily express the combination that
they wish to build/run at any given time.

The majority of these definitions are due to the combinatorial expansion of the
following dimensions:

 * Base Product that will ship
 * Components Version: stable, latest, developer
 * Product Type: `eng`, `eng_arrested`, `userdebug`, `user`

Through the use of explicit package URLs to components in other packages, the
component topology itself is crystalized within the `fuchsia-pkg://` namespace
on the device, as each package's full URL is listed directly in its parent's
cml:

 * root.cml
 * core.cml

To change which package's component is being used for the implementation of
some protocol, some other package's contents have to change to reference that
different package url.

Fuchsia developers and release teams must manage all of our partners' products
in tree, and it's impossible for out of tree product owners to release or
update products on their own schedules.  This creates significant load on the
Fuchsia organization and adds friction with the product owners.

## Solution statement

To address this, we propose to create a set of tools that can run out-of-tree,
which combine the notion of assembly and configuration into parts of the same
process:

 * assembly is the process of specifying:
   * which software pieces are to be used
   * why they relate to each other the way that they do (topology of a single
     component)
   * how they are to be configured
   * when they can be reconfigured (and by who)

 * configuration is the providing of:
   * the data that each component needs to properly operate
   * the CFv1 sysmgr configuration of the product
   * the overall CFv2 topology of the product
   * kernel boot arguments

These are two halves of the same process: how to assemble and configure Fuchsia
for a given product.

To allow for more controlled use and configuration of Fuchsia, we propose to
introduce the concept of "sub-assemblies".  Each sub-assembly defines a set of
software and files, in a fragment of topology, with configuration points and
values, with explicit documentation for what configuration can be modified
during the assembly process.

When this is complete:

 * The Fuchsia Platform defines sub-assemblies that products can use to choose
   what platform behaviors they want to assemble when doing this out-of-tree
assembly
   * One example is the software delivery stack sub-assembly that contains the
     `omaha-client` (userdebug/user builds) vs. the one that uses
     `system-update-checker` (eng builds).
 * Customer products can be assembled entirely out-of-tree, without using
   `fuchsia.git` directly
 * Multiple products variants can be assembled out of the same set of compiled
   software components.

## Dependencies

Migration of customer products to use sub-assemblies.  This will require
consulting with yaar@ and other stakeholders to ensure that we've appropriately
captured the right aspects of them.

Migration of the Fuchsia platform to use sub-assemblies.  This will require
consulting with the various subsystem teams to ensure that we've appropriately
captured the right variations.

To create out-of-tree images, certain tools (fvm, blobfs, minfs, zbi, avbtool)
will need to be ported to a format that can be used out-of-tree (static library
for linking with Rust, or entirely ported to Rust).

## Risks and mitigations

The main risks are:

 * New tools don't produce the same output as the existing ones
   * Mitigation: Run them both in parallel in CI/CQ, validating the new tools
     produce the same output, and then switch to using the new tools.

 * In-Tree and Out-of-Tree tools could diverge in capabilities
   * Mitigation: Use the Out-of-Tree tools as the in-tree tools, so we don't
     have two sets of tools to maintain.

 * Sub-Assembly Schema design could become a place of seeking perfection.
   * Mitigation: All schemas will be versioned with the express intent of
     revising them during the process, focusing on pragmatic solutions over
     perfectly modeling the system.

 * Large migration effort (products and platform)
   * Mitigation: This can be done in phases, starting with sets of
     components/labels and working up to fully-defined sub-assemblies.
   * Mitigation: Implementation plan is centered around a measured approach
     without boiling the ocean.

[glossary.product]: /docs/glossary/README.md#product