{% set rfcid = “RFC-0120” %} {% include “docs/contribute/governance/rfcs/_common/_rfc_header.md” %}
This RFC formalizes the requirements to use (i.e. encode and decode) the FIDL wire format absent of a transport. It also specifies a rubric on how bindings should expose this functionality. We introduce the concept of wire format metadata describing the revision and the features of the wire format, and require its usage in encoding and decoding APIs, such that:
A core principle of Fuchsia is to be updatable. We have invested heavily in ABI compatibility when FIDL is used in an IPC context, such as two peers speaking a FIDL protocol over a Zircon channel. The standalone use cases of the FIDL wire format on the other hand, by virtue of being relatively rare, have seen less attention to compatibility. For example, it was sometimes incorrectly assumed that passing the encoded bytes of a FIDL message alone would result in an evolvable ABI.
Both the Driver metadata RFC and RFC-0109: Fast datagram sockets call for sending FIDL over byte-oriented interfaces. Now is a good time to formalize the standalone uses of the FIDL wire format to provide them with evolution and interoperability guarantees.
Bindings MUST support encoding and decoding the FIDL wire format without a transport, an API whose requirements are detailed below. Note that many bindings already have some form of public encoding/decoding API (e.g. fidl::Encode
in the high-level C++ bindings). They should be adjusted in accordance with this RFC. This part of the RFC can thus be seen as a formalization of a core functionality, clarifying the layering of FIDL.
The focus of the FIDL wire format is binary compatibility: a set of guarantees around schema evolution to support reading data written using a different version of that schema. For example, a type with layout struct{uint8;uint8;}
may evolve into layout struct{uint16;}
. While FIDL provides extensible data structures, those do not support the evolution of the wire format itself, such as switching FIDL tables to a more efficient representation. Two pieces of information in the transactional header aid the binary compatibility of the FIDL wire format when used over a protocol and a transport:
When the FIDL wire format is used standalone, this information is missing from the encoded results. We propose to incorporate a subset of that information into encoding and decoding. Specifically:
In pseudocode, they would have the following function signatures:
function Encode<T>(object: T) -> (EncodedMessage, WireFormatMetadata); function Decode<T>(message: EncodedMessage, metadata: WireFormatMetadata) -> T;
EncodedMessage
contains the encoded bytes as well as any handles in the message. Most bindings do define a type with similar or equivalent purpose.
The wire format metadata itself will have an ABI compatible with a 64-bit integer. Its layout is as follows, in pseudo-C-notation:
struct fidl_wire_format_metadata_t { uint8_t disambiguator; uint8_t magic_number; uint8_t at_rest_flags[2]; uint8_t reserved[4]; };
RFC-0138: Handling unknown interactions proposes subdividing the flags in the transactional header into dynamic_flags
- ones that concern the request/response interaction model of a protocol, and at_rest_flags
- ones that concern the wire format. This RFC assumes that design, but could be easily adapted accordingly without losing its key properties.
The wire format metadata should have an alignment of 8 bytes, to facilitate in-place decoding of messages. Bindings MUST represent the metadata externally as an opaque structure that is 8 bytes long and has an alignment of 8 bytes (e.g. a struct with a single uint64
field). This allows the metadata itself to evolve by preventing users from depending on specific fields inside the metadata.
Bindings MUST check that the reserved
bytes are zero. Bindings MUST NOT depend on the at_rest_flags
bytes to have any particular value. Bindings MUST validate that the magic_number
represents a wire format revision that is supported.
Bindings MUST check that the disambiguator
byte is zero. Having a zero byte in the front of the metadata prevents programs from mistaking a FIDL message as text when the message is persisted as a file (see Convention for data persistence).
Note that the information contained in a FIDL transactional header is a superset of that in the wire format metadata. The semantics of the at_rest_flags
field and magic_number
field is identical between the transactional header and the wire format metadata.
Each message MUST only be used with its corresponding piece of metadata. In other words, it is not allowed to share metadata (e.g. use metadata A
to decode both message A
and message B
) or swap metadata (e.g. use metadata A
to decode message B
and use metadata B
to decode message A
). This allows the message wire format revision to be altered at run-time, such as during wire format soft migrations.
Bindings MUST support standalone usage with the following top-level types:
Encoding and decoding functions MUST fail given any other data type. The failure SHOULD occur at compile-time where possible.
The FIDL language does not mandate how the wire format metadata is transmitted or associated with the encoded message. For example, the metadata could be derived from the transactional message header when FIDL is used in a production IPC context.
To better support the motivating use cases, we would like to specify a convention for attaching the metadata to the encoded message, where the byte content of the message is prefixed by the metadata. Bindings SHOULD support this prefixed flavor of the standalone wire format usage, referred to as persistence.
The following persistence use cases are in scope:
The following use cases are out of scope:
Using this prefixed API flavor improves ergonomics and safety in a number of ways:
The persistence API MUST support the following top-level types:
Persistence MUST fail given any other data type. The failure SHOULD occur at compile-time where possible.
In pseudocode, the persistence API would have the following function signatures:
function Persist<T>(object: T) -> vector<uint8>; function Unpersist<T>(bytes: vector<uint8>) -> T;
Bindings MAY use alternate naming/method signatures that are the most appropriate in the target language, as long as they follow the shape of the API from a data flow perspective.
Bindings MAY support a vectorized Persist
variant that supports vectorized output, such as producing a zx_channel_iovec_t
or zx_iovec_t
that links to a number of buffers, or integrating with the idiomatic writer interfaces of target languages. Bindings SHOULD provide the vectorizing variant if they already use that in IPC code paths.
Bindings MAY support a vectorized Unpersist
variant the takes vectorized input, such as consuming zx_iovec_t
that links to a number of buffers, or integrating with the idiomatic reader interfaces of target languages.
Note that persistence results in bytes, as opposed to standalone encoding/decoding which may result in handles.
Bindings MUST support persisting large values that cause the encoded message size to exceed 64 KiB.
The FIDL style guide and API rubric should be updated to include persistence considerations:
This RFC does not change the FIDL source language.
Bindings should adjust their standalone encode/decode API to align with the proposed design involving metadata. They do not have to exactly follow the function signatures, as long as the functions are consistent with the proposal from a data dependency perspective. For example, the behavior of the decoder must be configurable via some means by the metadata.
The same standalone encode/decode API should be used to implement messaging, such as transactional message dispatching over Zircon channels.
Binding support for the persistence API flavors can be added independently.
There is already an implementation of persistence APIs in the Rust bindings, but the data format and API do not match the design in this RFC. The Rust implementation will be adjusted to align with the accepted design.
Currently, the Rust bindings provide the following functions:
fn create_persistent_header() -> PersistentHeader; fn encode_persistent_header(header: &mut PersistentHeader) -> Result<Vec<u8>>; fn encode_persistent<T: Persistable>(body: &mut T) -> Result<Vec<u8>>; fn encode_persistent_body<T: Persistable>(body: &mut T, header: &PersistentHeader) -> Result<Vec<u8>>; fn decode_persistent<T: Persistable>(bytes: &[u8]) -> Result<T>; fn decode_persistent_header(bytes: &[u8]) -> Result<PersistentHeader>; fn decode_persistent_body<T: Persistable>(bytes: &[u8], header: &PersistentHeader) -> Result<T>;
These should be replaced with the following (exact signature might differ due to borrowing and lifetime subtleties):
fn persist<T: Persistable, W: std::io::Write>(body: &mut T, writer: W) -> Result<()>; fn unpersist<T: Persistable, R: std::io::Read>(reader: R) -> Result<T>; fn standalone_encode<T: TopLevel, W: std::io::Write, H: core::iter::Extend<HandleDisposition>>(body: &mut T, writer: W, out_handles: &mut H) -> Result<WireMetadata>; fn standalone_decode<T: TopLevel, R: std::io::Read>(reader: R, handles: &mut [HandleInfo], metadata: &WireMetadata) -> Result<T>; struct WireMetadata { /* private fields */ }
The TopLevel
trait is implemented for structs, unions, and tables.
In particular, the user can no longer create a persistent header out of nothing and reuse the same header for encoding multiple messages.
Additionally, the bindings SHOULD provide a way to serialize/deserialize WireMetadata
to/from bytes, to support passing the metadata out-of-band.
Standalone encoding and decoding is part of the transactional usages of FIDL, while persistence APIs should share a majority of the code paths. Therefore, we can reuse the same standards and performance benchmarks.
Binding ergonomics should be designed to encourage the persistence convention. For example, a binding could use a shorter and more idiomatic function name to represent the persistent flavor (e.g. fidl::persist
), and use a longer and more explicit function name to represent the public standalone encoding/decoding API (e.g. fidl::standalone::encode
).
This change itself is backwards compatible since it is purely additive, with the exception of the Rust FIDL persistence implementation. To our knowledge, all current readers, writers, and stored data of Rust FIDL persistence always evolve in lockstep.
Adding wire format metadata improves future backwards compatibility, in anticipation of upcoming FIDL wire format migrations.
The wire format metadata contains 5 reserved bytes. Those bytes may be repurposed to take on additional meaning in the future. For example, we might use one byte to describe persistence-specific concerns.
The validation requirements of the FIDL wire format apply here, and hold the same security properties.
Of note, FIDL is not a self-describing format. Successfully deserializing a persisted message using one message type does not guarantee the data was originally serialized using that same message type:
A program may be confused over whether a FIDL message contains a prefixed metadata header, or if the metadata is passed out-of-band, leading to incorrect input parsing. We believe such errors tend to be caught early at the testing stage. Coupled with clear documentation, the security risk of this confusion should be small.
A malicious actor may trick a program into overwriting a persisted FIDL message with type Foo
with another message of a different type Bar
, which the attacker controls, by exploiting vulnerabilities in path processing. This allows the malicious actor to indirectly influence the contents of the Foo
message.
The alternatives section presents a more involved format that mitigates this risk, by extending the metadata header with information about the message type.
Padding bytes in the FIDL wire format are required to be zero, which helps avoid leaking sensitive information.
Persistent data tends to carry larger privacy concerns compared to ephemeral data in IPC that is swiftly consumed, but we also have IPC data being sent to a component that will persist it or send it over a network. As a result, the privacy concerns are similar between IPC and persistent APIs.
It is worth noting that developers can always manually persist FIDL data via other means, such as JSON or XML, even if we do not provide an API. The usual privacy reviews should apply when a future design mentions that it involves persisting user or other sensitive data, regardless of if the methodology is via FIDL persistence.
Privacy annotations on FIDL API elements will simplify privacy reviews and enable better downstream tooling (e.g. automated redaction); they are out of scope of this RFC which focuses on a particular method of transmitting FIDL messages.
We will extend GIDL, the FIDL conformance test suite, to test encoding and decoding of the persistent format.
Augment the bindings spec to include the added requirements from this RFC (e.g. LLCPP).
Create a reference page about FIDL for standalone encoding/decoding and persistence, and the relationship between the two APIs.
Add to //examples/fidl/
in all languages demonstrating standalone encoding decoding and persistence, and add corresponding tutorials.
We could go one step further on the convention and prescribe it as the standard: all metadata must precede the message payload. While sufficient for the use cases we have observed today, this direction runs the risk of becoming too rigid in the future. By providing both an un-opinionated standalone encoding/decoding API and an opinionated persistence API on top, users will be able to pick one most fitting to their design.
We could allow the same metadata to be shared for all messages in a session. This allows the metadata to be sent once at the beginning and then omitted for the rest of the communication between two peers. This could be used to stream multiple FIDL objects over one instance of a persistent medium. For example, Fast datagram sockets RFC could avoid adding 8 bytes to each UDP datagram, by first sending the metadata over the socket, followed by multiple objects in encoded form.
In doing so, we take on a constraint that the metadata must be independent of specific messages being encoded, and only dependent on the version of the FIDL runtime compiled into the producer of the message. This also means that a FIDL encoder must not arbitrarily switch wire format representations at runtime, should it support multiple wire format representations.
The 8 bytes additional tax imposed by the metadata does not seem prohibitively expensive. Always including it with the message relieves a lot of extra metadata tracking complexity on the users' end.
For use cases that really desire the raw performance of not including the metadata, we can look towards adding streaming features natively in FIDL. For example, one could imagine defining a transport over a socket that streams values of a single type. Bindings could be implemented to skip sending the metadata where possible and disallow changing the sender/receiver identity (for example, the transport must be re-established every time a new peer is introduced).
@stream protocol UdpSocketPayload over zx.socket { Send(SendMsgPayload); -> Recv(RecvMsgPayload); };
Having the metadata always specific to a message also enables using flags on a per-message basis. For example, we might use one flag to indicate that the body is compressed. It's also conceivable that we could design a more packed wire representation that is less rotated towards the constraints of efficient in-memory IPC (e.g. no need to reserve space for pointers or alignment). The alternative format could be indicated by another bit in the reserved region of the metadata.
The wire format metadata during persistence could be made compatible with the transactional message header.
We would do so by framing persistence as a transport, with a single method that writes the object:
@persistence type Metadata = table { 1: foo int32; 2: bar vector<int64>; }; // desugars to protocol MetadataSink over persistence { // Ordinal is the hash of `mylib/MetadataSink.Metadata`. Metadata(Metadata); };
This approach has some advantages:
This seems an elegant case of transport generalization, but the result will be a very strange protocol that is only one-way and one-shot: the client may only send one kind of value exactly once. There is no opportunity for the receiver to make a response. This wouldn't mesh well with evolution features we are adding at a protocol level, such as open and closed interactions.
As a less adventurous step than alternative 3, we could hash the fully-qualified type name of the persisted message and add that to the wire format metadata to identify the type of the message being persisted, without introducing the idea of a full-fledged transport.
This reduces cross-talk but still has other subtle complexities:
Overall, we believe the security risks prevented by this alternative is not worth the extra complexities required.
The main proposal suggests two kinds of public APIs dealing with the FIDL wire format:
This stems from the observation that the persistence convention is sufficient for all standalone use cases of FIDL today.
A future use case may be better suited to separately transmitting or storing the wire format metadata and the payload. They could reach for the standalone encoding/decoding API, but that has the drawback of allowing handles in the API, which could be unwarranted.
An alternative is to provide three kinds of APIs:
This improves the non-resource guarantee when a use case desires separately transmitting or storing the wire format metadata, but makes for a confusing API, since we end up with two kinds of encoding/decoding API with the only difference being support for resource types. This is compounded by target language limitations where sometimes it can be difficult to hide an API from the public surface.
The main proposal takes the simplification path and merges the first two kinds of APIs in this alternative together.