Consider streamlining the building of your RealmBuilder test realm using tools in //src/ui/testing/ such as the following:
We have Fuchsia-based products built on the Fuchsia platform. As Fuchsia platform developers, we want to ship a solid platform, and validate that the the platform works correctly for all our supported products. Integration tests ensure we uphold correctness and stability of platform functionality that spans two or more components, via our prebuilt binaries (such as Scenic) and API contracts (over FIDL). This is especially valuable in validating our ongoing platform migrations. One example is the set of touch dispatch paths, such as from Scene Manager to Scenic to a UI client.
Integration tests model a specific product scenario by running multiple Fuchsia components together. For example, to ensure that the “touch dispatch path from device to client” continues to work as intended, we have a “touch-input-test” that exercises the actual components involved in touch dispatch, over the actual FIDLs used in production.
Because integration tests are a model, there can (and should) be some simplification from actual production. Obviously, these tests won‘t run the actual product binaries; instead, a reasonable stand-in is crafted for a test. The idea is that it’s the simplest stand-in that can be used in a test, which still can catch serious problems and regressions.
Sometimes, it‘s not straightforward for the test to use an actual platform path used in production; we use a reasonable stand-in for these cases too. For example, we can’t actually inject into /dev/class/input-report, so we have a dedicated API surface on Input Pipeline to accept injections in a test scenario.
The important thing is that the test gives us confidence that evolution of platform code and platform protocols will not break existing product scenarios.
When the scenario involves graphics, it's very easy to accidentally introduce flakiness into the test, robbing us of confidence in our changes. Graphics APIs operate across several dimensions of lifecycle, topology, and synchronization/signaling schemes, in the domains of components, graphical memory management, view system, and visual asset placement. Furthermore, these APIs provide the basis for, and closely interact with, the Input APIs and the Accessibility APIs.
The principal challenge is to write tests that set up a real graphics stack, in a way that is robust against elasticity in execution time. We talk about those challenges in “Synchronization challenges”, below. There are other reasons for tests going wrong, and most of them can be dealt with by enforcing hermeticity at various levels. We talk about these first. A final challenge is to author tests that model enough of the interesting complexity on just the platform side, so that we know complex product scenarios don't break with platform evolution.
In general this translates to no sleeps or waits. If a test requires sleeps or waits, this is a bug and should be treated appropriately. Every action by the test should be gated on a logical condition that the test can observe. E.g., inject touch events only when the test observes the child view is actually connected to the view tree and vending content to hit.
At the bottom we have graphics tests. Input tests build on top of graphics tests. And accessibility tests build on top of input tests. Hence they have all the same problems, just with more components. It is thus critical that a basic graphics test is reasonable to write and understand, because they form the basis for “higher level” tests that inherently have more complexity.
Product owners must write e2e tests to ensure their product is safe from platform changes. E2e tests are big, heavy, and expensive to run; often, they are flaky as well. They are authored in a different repository, and run in their own test automation regime (“CQ”). And they care about the subset of OS functionality that their product relies on.
Given these realities, platform developers cannot rely on these e2e tests to catch problems in platform APIs and platform code.
By authoring platform-side integration tests, platform developers can get breakage signals much faster with less code in the tests, and systematically exercise all the functionality used across the full range of supported products. Product owners benefit by increased confidence in the platform's reliability.
Deterministic, flake-free tests increase the signal-to-noise ratio from test runs. They make life better.
When our tests rain down flakes every day, we ignore these tests, and they become noise. But when we try to fix the source of flakes, it often reveals a defect in our practices, or APIs, or documentation, which we can fix (think “impact”). Each of these hermeticity goals address a real problem that someone in Fuchsia encountered. When we have hermeticity, everyone benefits, and Fuchsia becomes better.
Fuchsia's platform teams often have important migrations in progress that affect products. Integration tests are a critical method of guaranteeing that our platform changes are safe and stable with respect to our product partners.
The Fuchsia Compatibility Test Suite ensures that the implementations offered by the Fuchsia platform conform to the specifications of the Fuchsia platform. An effective CTS will have UI integration tests, and so this guidance doc applies to those UI integration tests.
Various types of hermeticity make our tests more reliable.
All components used in the test should come from the same test package. This can be verified by examining the fuchsia-pkg URLs launched in the test; they should reference the test package. Don‘t invoke components from another test’s package!
If we don‘t have package hermeticity, and a component C is defined in the universe U, then the C launched will come from U, instead of your locally modified copy of C. This issue isn’t so much a problem in CQ, because it rebuilds everything from scratch. However, it is definitely an issue for local development, where it causes surprises - another sharp corner to trap the unwary. That is, a fix to C won't necessarily run in your test, and hampers developer productivity.
All components in the test should be brought up and torn down in a custom Fuchsia environment. In component framework v2, the RealmBuilder is responsible for rebuilding the component topology for each test run and shutting down the components in an ordered manner.
This practice forces component state to be re-initialized on each run of the test, thereby preventing inter-test state pollution.
The advantages of doing so are:
injected-servicesIn component framework v2, it‘s possible to declare injected-services in a test’s CML manifest. Declaring injected-services is somewhat of an anti-pattern. It, too, also constructs a test environment, but all the test executions run in the same environment. If a service component had dirtied state, a subsequent TEST_F execution will inadvertently run against that dirtied state.
All components in the test should not be exposed to the actual root environment. For FIDL protocols, this is not so much an issue. However, there are other types of capabilities that have leaks. A good example is access to device capabilities, such as /dev/class/input-report and /dev/class/display-coordinator. Components that declare access to device capabilities will actually access these capabilities, on the real device, in a test environment.
We can gain capability hermeticity by relying on a reasonable fake. Two examples.
Correct, flake-free inter-component graphics synchronization depends intimately on the specific graphics API being used. As a rule of thumb, if you imagine the timeline of actions for every component stretching and shrinking by arbitrary amounts, a robust test will complete for all CPU-schedulable timelines. The challenge is to construct action gates where the test will hold steady until a desired outcome happens. Sleeps and timeouts are notoriously problematic for this reason. Repeated queries of global state (such as a pixel color test) are another mechanism by which we could construct higher-level gates, but incur a heavy performance penalty and adds complexity to debugging.
Another dimension of complexity is that much of client code does not interact directly with Fuchsia graphics APIs; instead they run in an abstracted runner environment. Web is a good example where the client code cannot directly use Scenic APIs. Some facilities can be piped through the runner, but tests generally cannot rely on full API access. Some runners even coarsen the timestamps, which also complicates testing a bit.
One more subtlety. We're interested in the “state of the scene graph”, which is not precisely the same thing as “state of the rendering buffer”. For most purposes, they are loosely equivalent, because the entity taking a visual screenshot is the same entity that holds the scene graph - Scenic. However, specific actions, like accessibility color adjustments, will not be accurately portrayed in a visual screenshot, because the color adjustment takes place in hardware, below Scenic.
Use fuchsia.ui.observation.geometry.Provider to correctly know when a view is present in the scene graph. Refer here for more details on how to use the API
The Touch Input Test is constructed using the Realm Builder library. This library is used to construct the test Realm in which the components under test operate. The test suite, hereafter test driver component, is a component.
Realm Builder (and CFv2 in general) allows us to be explicit about what capabilities are routed to and from components. This is crucial for testing because it allows test authors to have fine-grained control over the test environment of their components. Take for example scenic. In touch-input-test, a handle to fuchsia.hardware.display.Provider from fake-display-coordinator-connector#meta/display-coordinator-connector.cm is routed. By providing a fake hardware display provider, we can write integration tests without having to use the real display controller. This mapping of source and target is explicitly written in the test file during Realm construction.
The graphics API allows each product to generate an arbitrarily complex scene graph. However, the products we have today typically rely on a few “core topologies” that are stable and suitable for the product to build on.
It's a valuable exercise to capture each of these core topologies in our platform integration tests. Some examples:
Developing new models are also how we test new topologies and interaction patterns to make sure the APIs are sensible and usable, and serve as as a foundation for converting an entire product.