{% set rfcid = “RFC-0202” %} {% include “docs/contribute/governance/rfcs/_common/_rfc_header.md” %}
{# Fuchsia RFCs use templates to display various fields from _rfcs.yaml. View the #} {# fully rendered RFCs at https://fuchsia.dev/fuchsia-src/contribute/governance/rfcs #}
Currently a test on Fuchsia is executed as a dynamic child of Test Manager. This design enables Test Manager to launch tests in any realm (that product owner's have permission to create) thereby allowing test authors to route required runners and capabilities without making changes to Test Manager.
Test Manager launches a test as a dynamic child providing it with various hermetic/non-hermetic capabilities and in-tree runners. The current design of Test Manager is to run with a static set of built-in test runners and test realms. It does not allow out-of-tree (OOT) and in-tree customers to include their own test runners and required capabilities, but as we gain more customers we will need to support their use-cases. This document proposes “Test Manager as a service” which will allow our customers to run tests in a test realm of their choosing, potentially outside of the Test Manager's realm. This reduces responsibility of the Test Manager in routing required system capabilities to the tests.
Facilitator: davemoore@google.com
Reviewers: geb@google.com, shayba@google.com, richkadel@google.com, kjharland@google.com, crjohns@google.com, cgonyeo@google.com, aaronwood@google.com, satsukiu@google.com, xbhatnag@google.com, yaneury@google.com, hjfreyer@google.com, akbiggs@google.com
Consulted:
Component Framework team was consulted relating to questions about capability routing, framework and testing APIs used in this design.
Socialization:
This RFC went through a design review with the Testing Architecture and Component Framework team.
In this design product owners will decide where the test should run and the test executors (ffx test/run-test-suite) will pass in required information to the Test Manager. We will add a new method to the RunBuilder protocol called AddSuiteInRealm and pass in required realm information.
AddSuiteInRealm(resource struct { // The realm which contains the collection to launch the test in realm client_end:fuchsia.component.Realm; // All offers from the realm to the test collection offers: Vec<Capabilities> // the test collection to launch the test in. test_collection: string // ... existing fields from AddSuite });
Test Manager will use above information to launch the test in the specified collection using Realm Builder while providing isolated logger, coverage collection, in-tree runners, etc to support test execution. Component manager will route LifecycleController and RealmQuery protocol (scoped to “/”) to the test executor.
Test realms can either be authored by the platform itself, or by product owners by using some existing/new mechanism. So far only the platform could define test realms, and that this work seeks to democratize test realm definitions and management.
This design assumes two types of users.
Test Realm author will create a test realm, set it up and integrate with build tools to allow Test author to run the test. They need to install Realm Builder shard in their test realm for this to work.
Sample test realm:
{ include: [ "sys/component/realm_builder.shard.cml", ], collections: [ // The collection to launch test in { name: "tests", environment: "#test_env", durability: "transient", }, ], offer: [ { protocol: [ // Some system or mocked protocol "fuchsia.foo.bar", ... ], from: "parent", to: [ "#tests", ], }, ... ], environments: [ { name: "test_env", extends: "realm", runners: [ // Offer some OOT runner to the test { runner: "fuchsia_oot_runner", from: "parent", }, // TODO(https://fxbug.dev/42063673): Abstract out into a shard. // This is important so that Realm Builder can work. { runner: "realm_builder", from: "#realm_builder_server", }, ], resolvers: [ // This is important so that Realm Builder can work. { resolver: "realm_builder_resolver", from: "#realm_builder_server", scheme: "realm-builder", }, ], }, ] }
The realm author will provide integration with the build tools so that test executors can read the moniker and the test collection during execution.
Ergonomics section briefly touches on some solutions to pass in moniker and test collection name to the test executor and detailed discussion is out of scope for this document.
Test author will use one of the ergonomic solutions (TBD) to run their test, which will pass in the information to test executor to run the test.
The test executor will use RealmQuery API to query the realm object for the realm and all the offers to the test collection and pass them to Test Manager using the proposed AddSuiteInRealm API.
It will also use LifecycleController API to resolve the moniker if required.
Following changes are required for this design:
fuchsia.component/Realm
proxy.offers
and route them using Realm Builder.Test Manager can connect to all capabilities from the test realm because now it has handles to the component instance.
Test Manager will collect all the artifacts and upload test results after the test ends. Tests will have access to any capability that its parent realm provides and all the hermetic capabilities provided by Test Architecture.
Advantages of this approach
Additional benefits
The drawbacks of this solution are:
ffx test
directly will need to provide their moniker and collection name to the tool.The diagram of the test topology once Test Manager can be used as a service.
Future work:
This change will have no or minimal performance impact as this will not affect how the test is launched, it only affects where the test is launched in the topology.
This sections describes some solutions to pass in moniker and test collection to the test executor. Detailed solutions are out of scope for this document.
Once a custom realm is defined, product owners can define test categories in their build system. Test author will use that category and the build system will produce corresponding moniker and test collection as inputs to the test executor.
We can embed moniker and test collection information inside test facet. Test executor will need to resolve and read the component manifest file to obtain the information.
For better ergonomics, Test Realm author will provide manifest shards which the Test author can include in their test manifest.
We will keep supporting current tests and will always support hermetic test realm which does not need custom runners.
We do not collect any personal data, so this design will not have any privacy effects.
Current tests along with new integration and unit tests to launch tests in custom realm should fully test this feature.
Document this feature explaining the use-case and implementation guide for developers to create their own test realms.
Also document the way for test developers to use custom realms.
Test developers will launch their own Test Manager in their topology. ffx test will use RCS to connect to fuchsia.test.manager.RunBuilder protocol to execute and collect results from the test.
Benefits of this solution:
The problems with this solution are:
Sub-assemblies could solve our problem of bring-your-own-test-runner but it is not as flexible as proposed solution.
The proposed solution allows us the flexibility to in future remove all hard coded test realms from test manager and provide full ownership to the product owners.
Tests can package their own runner and run their test components using them
Benefits of this solution:
The problems with this solution are:
Parameterize a test with capabilities and the source of those capabilities and then pipe those to the test realm itself.
"fuchsia.test.additional_capabilities": { "runner": "dart_runner", "source": "/core/dart/runner" }
For routable capabiltiies, Test Manager can use RealmQuery API to proxy the request using Realm Builder. For runners, Test Manager can use hub to proxy runner protocol.
Benefits of this solution:
The problems with this solution are:
In this design the Test Manager will have access to RealmQuery API and use that to query the required information from the test realm to launch the test.
The test author will use shard to include realm and test collection information in their test manifest.
Shard:
{ facets: { "fuchsia.test": { launch: { realm: "/core/foo/bar/test_realm", collection: "tests" // default is "tests", can be omitted. } }, }, }
test.cml:
{ include: [ "syslog/client.shard.cml", "//some/path/oot_runner/default.shard.cml", "//some/path/test_realm/default.shard.cml", ], program: { binary: "bin/sample_test", }, use: [ { protocol: [ "fuchsia.foo.bar", ... ], }, ... ], }
Test manager will read the facets and use provided capability to query and launch the test in specified test realm.
This design proposes that there would be a common known place in topology (eg /core/tests), where the Test Manager can execute all the system tests. The platform developers will create and maintain this known realm and the Test Manager will only provide the mechanism to execute and collect artifacts from the tests.
This is a concept unique to Fuchsia, so no prior art for the same.