{% set rfcid = “RFC-0221” %} {% include “docs/contribute/governance/rfcs/_common/_rfc_header.md” %}
{# Fuchsia RFCs use templates to display various fields from _rfcs.yaml. View the #} {# fully rendered RFCs at https://fuchsia.dev/fuchsia-src/contribute/governance/rfcs #}
Approve Python for out of tree (OOT) system testing.
There is a known need for improving Fuchsia's end-developer facing system testing support. With driver development moving out-of-tree (OOT), this presents an immediate demand for shipping a standardized system testing solution via the SDK to support hardware qualification.
As a prerequisite to starting this work, the programming language to build the solution with needs to be selected. This document considers various programming languages for OOT system testing by proposing an evaluation criterion and providing data points for each language. The findings from the evaluation exercise will be used to make an informed recommendation on the programming language Fuchsia should use for meeting today's OOT system testing needs.
Facilitator:
TBD
Reviewers:
Consulted:
Socialization:
The OOT system testing evaluation rubric and the evaluation of each programming language in consideration were discussed in Google Docs with members from the FEC, Testing Architecture team, Testing Infrastructure team, and Tools Team.
Summary of the product motivation and requirements to support next generation OOT system testing for Fuchsia.
Primary user(s): Developers and system integrators testing the Fuchsia platform and products.
Environments: In-tree, and all environments and petals that the SDK ships to.
Test framework adoption depends heavily on language choice. An easy to use language that is already widely adopted in this system testing use case is highly beneficial to expand beyond core SWEs and include Test Engineers and System Integrators as well.
The general pros and cons of each language (with the exception of TypeScript and JavaScript) are already well documented in the Fuchsia Programming Language Policy (FPLP). The metrics used in this document cover language aspects that are specifically relevant to OOT system testing.
Evaluation rubric (in no particular order):
Ecosystem adoption: Language popularity in the OOT system testing space
Ergonomics: How well a language supports the common modes of system testing usages
Fuchsia portability: Ease of running on both host and Fuchsia target
Investment required: Work required to productionize an OOT system testing solution
Maintainability: Work required to sustain an OOT system testing solution
This section goes over several languages that are being considered for OOT system testing and provides evidence for/against each evaluation metric. To make the comparisons between the languages simpler, a score of either Infeasible, Bad, Neutral, or Good is assigned for every category in the evaluation rubric for each language.
Note: This coarse-grained bucketing obfuscates many nuances within or around the same “score”, but the aim of this approach is to surface the significant differences between the languages.
Ecosystem adoption: Neutral
Ergonomics: Good
Fuchsia portability: Neutral
Investment required: Neutral
gofmt
and govet
are already enabled in Fuchsia CI.Maintainability: Good
Ecosystem adoption: Good
Ergonomics: Good
Fuchsia portability: Neutral
Investment required: Good
Pro: Python Mobly-based system testing solution already exists in-tree.
Pro: Pigweed in Fuchsia is heavily invested in Python.
Neutral: Requires investment in OOT similar to other languages in consideration.
Maintainability: Bad (Neutral with investment)
Ecosystem adoption: Neutral
Ergonomics: Good
Fuchsia portability: Good
Investment required: Bad
Maintainability: Good
The programming languages are removed from in-depth analysis due to infeasibilities in one or more evaluation metrics.
Ergonomics: Infeasible
Ecosystem adoption: Infeasible
Ergonomics: Infeasible
Investment required: Infeasible
Maintainability: Infeasible
Here is a visual summary of programming language evaluation for OOT system testing.
Based on the evaluation above, it's seen that Go, Python, and TypeScript are all languages that can feasibly fit the task of OOT system testing. However, each language has its own tradeoffs that make selecting a clear consensus “winner” difficult. Ultimately, customer demand (“Ecosystem Adoption” dimension) is indexed heavier to tie-break in order to align on a language to unblock our OOT customers today.
Python is recommended for OOT system testing approval at the current time primarily due to its alignment with Product requirements; especially in its ease of integration with Google‘s existing test frameworks. Python may not be absolutely preferred for system testing due to technical reasons, but Fuchsia’s OOT customers require it.
In summary, Python's advantages:
While additional investment is required both in-tree and OOT to reach maintainability parity with other statically typed languages, there is line-of-sight to achieve parity through type hints and coverage enforcement via improving Fuchsia's Python static analysis tooling.
With that said, the use of Python for OOT system testing will be revisited if there is a significant amount of bugs and concerns related to Python maintainability 6 months after type hints and coverage have been enforced in-tree, and type hints have been enforced in one OOT repo.
Note that the approval of Python for OOT system testing in this RFC does not foreclose other languages from being used in this space. In fact, the key OOT system testing component that Fuchsia will be exporting in the SDK will be the Fuchsia Controller which is designed specifically to be extensible to any host-side language to support host-to-Fuchsia-device communication.
The Fuchsia Programming Language Policy will need to be updated to reflect this RFC's proposal of approving Python for OOT system testing.
Discussed above for each language as part of the evaluation metric - ergonomics.
Discussed above for each language as part of the evaluation rubric - maintainability.
Discussed above in Languages in consideration.
N/A
N/A
N/A
N/A