# [FIDL Tuning Proposal](README.md) 024

Mandatory Source Compatibility
==============================

Field     | Value
----------|--------------------------
Status    | Accepted
Authors   | cramertj@google.com
Submitted | 2019-04-02
Reviewed  | 2019-04-11

## Summary

Establish a source-compatibility standard for FIDL language bindings, as
well as a process for evolving that standard.

## Motivation

Today, there are few written rules for the code generated by language
bindings.
It's expected that they conform to a specific wire ABI, but aside from
that, binding authors are given lots of leeway over how they shape their
APIs.
Any change to a FIDL definition could cause arbitrary changes to the
generated bindings.

In practice, users expect a sort of "common sense" list of things that
should be source-compatible, such as defining a new top-level type.
However, there's no explicit rule saying that this is the case.
While this case seems somewhat absurd, it illustrates how a lack of
specification can ruin users' expectations.
Real examples of this that have occurred in practice include adding fields
to tables, adding a new xunion variant, or adding a new defaulted field to
a struct.
Users could reasonably expect that these changes wouldn't be
source-breaking, but there's no standard specifying this, and all of these
changes cause source-level breakage in one or more language bindings today
(e.g. due to positional initializers in C++ or Go, or struct patterns in
Rust).

Furthermore, there are a number of extremely useful extensions to FIDL
language bindings that have been rejected in the past due to their
interaction with source compatibility.
Examples of this include adding a `copy` or `clone` function to types that
don't contain handles.
Types that contain arbitrary handles cannot be cloned, so adding a handle
to a type would prevent it from offering a `clone` function (or prevent it
from offering a clone function that *worked*, at any rate).
A change to introduce conditional inclusion of a `clone` function to
generated Rust bindings based on the absence of handles has been rejected
multiple times due to its effects on source compatibility.
As a result, Fuchsia developers have had to manually roll their own
`clone` functions and add wrapper types for the FIDL generated types which
`clone` via these hand-rolled methods.
This document proposes a consistent standard against which we can evaluate
functionality like this, hopefully providing a more ergonomic,
user-friendly, and boilerplate-free experience to developers.

## Design

### The Process

This FTP establishes an initial set of source compatibility constraints.
This list will be tracked in a document in the Fuchsia source tree.
Additional source compatibility constraints must be added using the FTP
process.
To facilitate easy addition of source compatibility rules related to new
features, the "Backwards Compatibility" section of the FTP template will
be amended to include a suggestion to introduce new source compatibility
constraints (where applicable).

### Definitions: Source Compatibility and Transitionability

Changes below are required to be either *source-compatible*
(i.e., non-source-breaking) or *transitionable*.

*Source-compatible* changes must not cause source-breakage to any valid
(compiling) usage of the public API of the generated FIDL bindings.
There's some reasonable argument over the definition and feasibility of
restricting what features are part of the "public API" and which aren't,
so for the purposes of this document we consider the "public API" to be
any use of the generated bindings which does not require either
extraordinary language gymnastics (e.g. reflection) or explicit developer
intent to violate privacy (e.g. calling
**__private_dont_use_me_function_2()**).
All other APIs exposed (e.g., positional initialization, pattern-matching,
etc.) must be constrained so that user code cannot be broken by
source-compatible changes to FIDL libraries.

*Transitionable* changes are changes for which it is *possible* to write
code that compiles both before and after the change.
Each transitionable source-compatibility rule must specify exactly what
"use" of the API must be possible during the transition.

### Initial Source Compatibility Constraints

The following are a list of changes that must be source-comptabile:

*   Adding a new top-level item (protocol, type, or constant).
    *   Motivation: users expect that declaring new protocols, types, and
        constants can be done without breaking existing users of the FIDL
        library.
    *   Exemption: usages with "*" or blanket imports from a namespace may
        experience breakage as a result of ambiguities between multiple items
        from different libraries with the same name.
*   Adding a field to a non-strict table.
    *   Motivation: tables are designed for easy extensibility and should
        support additional fields without breakage.
        To opt into breakage, the `strict` modifier can be used.
*   Adding a variant to a non-strict extensible union.
    *   Motivation: extensible unions are designed for easy extensibility
        and should support additional variants without breakage.
        To opt-in-to breakage, the `strict` modifier can be used.
*   Adding a member to a non-strict enum
    *   Motivation: non-strict enums are implicitly opting into expansibility
        and should be expandable without source breakage.
*   Adding a member to a non-strict "bits"
    *   Motivation: non-strict bits are implicitly opting into expansibility
        and should be expandable without source breakage
*   Adding `[Layout = "Simple"]` to an existing protocol
    *   Motivation: `[Layout = "Simple"]` exists in order to enable usage in
        simple C bindings.
        Existing protocols which conform should not require a breaking
        source change in order to specify that they can be used in the simple
        C bindings.
*   Adding `[MaxHandles]` to an existing type
    *   Motivation: `[MaxHandles]` exists to provide extra information about
        a type so that it can be used more permissively.
        It should not require a breaking source change in order to specify
        that a type already contains a fixed maximum number of handles and
        may be assumed to continue containing at most that number of handles.

The following are a list of changes that must be transitionable:

*   Adding `[Transitional]` to a method
    *   Use: it must be possible to implement a protocol and supply an
        implementation of a method using the same source both before and
        after the addition of the `[Transitional]` attribute to that method.
    *   Motivation: it must be possible to gradually add or remove methods
        to protocols so long as all existing implementations can be
        gradually adapted.
*   Adding a new `[Transitional]` method
    *   Use: it must be possible to implement a protocol using the same
        source both before and after the addition of a new `[Transitional]`
        method (though the API need not allow implementation of the method
        during the transition).
    *   Motivation: it must be possible to gradually add or remove methods
        to protocols so long as all existing implementations can be
        gradually adapted.
*   Removing a `[Transitional]` method
    *   Use: it must be possible to implement a protocol using the same
        source both before and after the removal of a `[Transitional]` method
        (though the API need not allow implementation of the method during the
        transition).
    *   Motivation: it must be possible to gradually add or remove methods
        to protocols so long as all existing implementations can be gradually
        adapted.
*   Removing a field of a non-strict table
    *   Use: it must be possible to create a table and access its fields
        (except the one being removed) using the same source both before
        and after the removal of a table field.
    *   Motivation: tables are designed to be evolved easily and should
        support removal without breakage.
        To opt into breakage, the `strict` modifier can be used on the
        table.
*   Removing a variant of a non-strict extensible union
    *   Use: it must be possible to create an xunion and access its
        variants (except the one being removed) using the same source both
        before and after the removal of an xunion variant.
    *   Motivation: xunions are designed to be evolved easily and should
        support removal without breakage.
        To opt into breakage, the `strict` modifier can be used on the
        table.
*   Marking a type as `strict`
    *   Use: it must be possible to access all fields of a table or "bits"
        and all variants of an enum or xunion using the same source both
        before and after `strict` is added.
    *   Motivation: `strict` is intended to be added to a type declaration
        once that type has stabilized, allowing increased reasoning and
        developer tooling.
        However, this is only required as a transitionable change and
        not a non-breaking change because extensible types may wish to
        allow access to unrecognized fields or variants.
        These capabilities don't make sense for a `strict` type, as
        unrecognized fields or variants would be rejected.
*   Adding `[Transitional]` to a member of an enum or bits, field of a
    table, or variant of an extensible union.
    *   Use: it must be possible to access all non-transitional
        members/bits/fields/variants and to construct values of the
        enum/bits/table/extensible union that do not include the
        `[Transitional]` value using the same source both before and
        after the introduction of `[Transitional]`.
    *   Motivation: it must be possible to gradually remove members,
        fields, or variants.
*   Adding a new member of an enum or bits, field of a table, or variant
    of an extensible union marked as `[Transitional]`.
    *   Use: it must be possible to access all non-transitional
        members/bits/fields/variants and to construct values of the
        enum/bits/table/extensible union that do not include the
        `[Transitional]` value using the same source both before and
        after the introduction of the new `[Transitional]` field.
    *   Motivation: it must be possible to gradually add members, fields,
        or variants.
*   Removing a member of an enum or bits, field of a table, or variant of
    an extensible union marked as [Transitional].
    *   Use: it must be possible to access all non-transitional
        members/bits/fields/variants and to construct values of the
        enum/bits/table/extensible union that do not include the
        `[Transitional]` value using the same source both before and
        after the removal of the `[Transitional]` field.
    *   Motivation: it must be possible to gradually remove members,
        fields, or variants.

The following are potential constraints that have been omitted from this
list, including justification as to why they have been omitted:

*   Adding or removing fields (defaulted or not) from structs
    *   This is an ABI-breaking change and would require other significant
        efforts to ensure a compatible transition.
        Making this a non-breaking change requires eliminating anything
        that does "for all fields"-style reasoning about a type,
        including automatic method derivation (e.g. "does this type contain
        any floats"), positional initializers, and exhaustive field matching
        and construction.
*   Adding or removing fields/variants (defaulted or not) from strict
    tables and xunions
    *   `strict` is intended to enable additional developer tooling that
        relies on "for all fields"-style reasoning about a type, including
        automatic method derivation (e.g. "does this type contain any floats"),
        positional initializers, and exhaustive field matching and construction.
        Forcing this to be a non-breaking change would inhibit this
        purpose.
*   Adding handle-containing fields or variants to a type not marked with
    `[MaxHandles]`
    *   Adding fields to a strict type or a struct is already a
        source-breaking change for other reasons, so adding a field with a
        handle is similarly a breaking change and may affect the APIs generated
        as a result.

## Implementation Strategy

This FTP establishes the initial proposed language compatibility standard.
Bugs will be filed and assigned to one author of each language binding to
ensure that their languages bindings are compliant.

## Ergonomics

This change makes FIDL easier to use by setting clear standards for source
compatibility, allowing for automatic checking as well as easier manual
checking of FIDL changes' source-compatibility, as well as offering
bindings authors clearer guidance on source compatibility, allowing them
the freedom to make bindings that are language-idiomatic while still
respecting standard requirements of the project.

## Documentation and Examples

Following the acceptance of this FTP, the process established by the FTP
as well as the source compatibility rules themselves will be published
along with other FIDL reference documentation.

## Backwards Compatibility

Application of the guidance proposed may require changes to bindings and
uses of those bindings, it is up to the respective binding authors to
navigate such changes.

This section (the "backwards compatibility" section) will be amended to
include the following text:

> "If you are introducing a new data type or language feature, consider what
> changes you would expect users to make to FIDL definitions without
> breaking users of the generated code.
> If your feature places any new [source compatibility](ftp-024.md)
> restrictions on the generated language bindings, list those here."

Note that you should include the **source compatibility** text as an actual
link to this FTP, that is:

```md
[source compatibility](/docs/zircon/development/languages/fidl/reference/ftp/ftp-024.md)
```

## Performance

This FTP does not restrict runtime behavior, although the restrictions on
source APIs may cause language binding authors to design more or less
performant APIs.
The feasibility of creating performant bindings in supported languages
should be considered when new source compatibility restrictions are
introduced.

This feature might affect compile-time performance as a result of pushing
towards patterns which require heavier inlining and compiler optimizations
in order to be performant (e.g. optimizing away a complex builder API into
a simple struct initialization).
Bindings authors should strive to make design choices that don't
significantly hamper compile times, but the compile-time-consequence of a
particular language API should not necessarily prevent the introduction of
a new source compatibility restriction.

## Security

This feature does not affect security.

## Testing

Many source compatibility rules are of the form "there cannot exist any
user code that compiled before this change but not after this change."
Unfortunately, these restrictions are difficult or impossible to test
because they would require enumerating every possible usage of the API
before the change.

However, we can (and should) add items to [the FIDL change test
suite][test-suite] to show that there does exist *some* usage of the
API before the change that remains valid after the change.
This is a necessary but not sufficient condition for meeting the source
compatibility requirements.

## Drawbacks, Alternatives, and Unknowns

*   Don't introduce a specification like this.
    Allow bindings authors to choose how breaking or non-breaking they
    want their changes to be.
    This is roughly similar to the current status de jure, but would give
    bindings authors more flexibility than they are granted de facto under the
    current system, in which some source-compatibility-hostile changes have
    received pushback.
*   Create a specification for which changes *are* allowed to be
    source-breaking, rather than which ones are *not* allowed to be
    source-breaking.
    This is tougher to enforce and would require bindings authors to
    anticipate changes under which their bindings must remain
    source-compatible.
*   A slight modification would be to specify both changes that *are* and
    *are not*, with unspecified changes defaulting one way or another -- this
    is essentially the same as either this FTP or the alternative above
    depending on the default, although it sets up a more official expectation
    around documenting the effects of different FIDL changes.

## Prior Art and References

Previous attempts have been made to introduce evolvability restrictions
via the `[MaxHandles]` attribute.
This design and the intended modifications to it have been discussed in
earlier parts of this proposal.

<!-- xrefs -->
[test-suite]: /garnet/tests/fidl-changes/