| # Sysmem overview | |
| [Sysmem][sysmem] is a [FIDL][fidl] service that allocates shared memory for | |
| use by multiple applications and hardware blocks. This document gives an | |
| overview of its major features and what it provides to the system. | |
| ## Motivation | |
| Modern systems have specialized hardware blocks that can perform operations | |
| on data. Some examples of these blocks: | |
| * Rendering computer graphics with a GPU. | |
| * Encoding or decoding video with a hardware codec. | |
| * Capturing high-quality photographs with a camera and DSP. | |
| * Compositing images to a display using the display controller’s overlay | |
| engine. | |
| * Evaluating neural networks with a TPU. | |
| Often, we want to create pipelines passing data between those separate | |
| hardware blocks. Some examples: | |
| * Decoding video with a hardware codec and compositing it to the display | |
| together with user interface content rendered on the GPU. | |
| * Applying a neural network to a live camera feed. | |
| To do this efficiently and avoid copying, drivers and applications must agree | |
| on the format data is in and where the data is located in memory. This | |
| agreement allows the output of one unit to be used directly by the next. | |
| Hardware units have strict constraints on these properties and also | |
| preferences about which data layout will give the best performance. | |
| Sysmem is a global service that can be fed constraints by applications and | |
| allocate buffers such that all the constraints are met. If multiple formats | |
| are supported, it can choose between them based on the expected performance. | |
| Sysmem buffers are often used to represent images and sysmem has special | |
| support for negotiating image formats. However the sysmem interface can also | |
| be used for audio or any other type of data. | |
| Sysmem does not manage the flow of data once buffers have been allocated. | |
| Applications in a pipeline are responsible for coordinating between each | |
| other to ensure synchronization. | |
| ## Protocols: `fuchsia.sysmem2.Allocator` vs. `fuchsia.hardware.sysmem.Sysmem` | |
| Despite the similar names, these two FIDL protocols serve completely different | |
| roles and have different security models: | |
| * **[`fuchsia.sysmem2.Allocator`][Allocator]**: The standard client protocol for | |
| negotiating constraints, allocating buffers, and sharing buffer collections. | |
| **Almost all sysmem clients (including hardware drivers) should use | |
| `fuchsia.sysmem2.Allocator`.** | |
| * **`fuchsia.hardware.sysmem.Sysmem`**: A privileged backend registration | |
| protocol used **only** by: | |
| 1. Drivers that implement and register a custom external sysmem heap | |
| (`RegisterHeap`). | |
| 2. The platform secure memory (TEE) driver that registers physical secure | |
| heaps (`RegisterSecureMem`). | |
| `fuchsia.hardware.sysmem.Sysmem` does **not** provide buffer allocation. Because | |
| sysmem trusts callers on `fuchsia.hardware.sysmem.Sysmem` to back heap | |
| allocations or specify physical secure memory ranges, access is restricted via | |
| Component Framework capability routing: only components/drivers that | |
| specifically and legitimately implement an external heap or secure memory may | |
| declare `use` of `fuchsia.hardware.sysmem.Sysmem` in their component manifests. | |
| ## Allocation process (simplified) | |
| 1. [Participants](#participants) connect to the | |
| [fuchsia.sysmem2.Allocator][Allocator] service | |
| 1. One participant (called the initiator) creates an initial | |
| [buffer collection token](#token). | |
| 1. That participant [duplicates] the token and sends the duplicate to other | |
| participants. | |
| 1. Those participants can also duplicate and send tokens, recursively, until | |
| all participants have received a token. | |
| 1. Each participant [binds][bind] its token to get a buffer collection. | |
| 1. Each participant [sets constraints][SetConstraints] on the buffer collection | |
| 1. Sysmem chooses a format that can satisfy all the constraints. This can | |
| only happen after every participant has bound its token and set constraints | |
| on the collection. | |
| 1. Sysmem allocates a number of [buffers](#Buffers) using that format. | |
| 1. Participants retrieve the buffers from sysmem, as well as information | |
| about the allocated format. | |
| The information returned by sysmem is the set of constraints the image | |
| formats in that buffer must satisfy. | |
| At this point participants can use the buffers with the allocated format, | |
| subject to pipeline-specific constraints on the flow of data between | |
| participants. Since multiple image sizes may be usable from a buffer, | |
| participants must work together to choose an image size before they can | |
| determine the precise image format. This allows a pipeline to switch image | |
| sizes on the fly without needing to reallocate buffers. | |
| If a new participant needs to be added to an existing collection which has | |
| already undergone constraint negotiation and allocation, a new token can be | |
| [attached][AttachToken] to the buffer collection. The new participant's | |
| constraints must be satisfied using the already-allocated buffer collection, | |
| or the logical allocation (from the point of view of the new participant) fails. | |
| To increase the chances that the new participant can be added successfully, one | |
| of the participants present during initial allocation can mark a token | |
| [dispensable][SetDispensable]. This allows the token to be used to specify | |
| stand-in constraints so that the buffer collection will be able to later | |
| accommodate a new participant with the same constraints. | |
| ## Buffer destruction | |
| Buffer destruction is asynchronous. | |
| To wait for old buffers of an old collection to fully delete (optionally with a | |
| deadline), see [AttachLifetimeTracking][AttachLifetimeTracking]. | |
| The buffer destruction sequence begins when the buffer collection fails. In the | |
| normal/success case, this "failure" is typically triggered when a participant | |
| (often the initiator) closes any collection-specific channel without having sent | |
| [`Release`][Release] first. Currently, collection-specific channels are | |
| [`BufferCollectionToken`][BufferCollectionToken], | |
| [`BufferCollection`][BufferCollection], and | |
| [`BufferCollectionTokenGroup`][BufferCollectionTokenGroup] channels. | |
| When the collection enters the failed state, the sysmem server will close the | |
| server end of all the collection-specific channels. All the allocated VMOs | |
| remain usable until participants are done cleaning up (see list below). | |
| At this point all participants are expected to quickly notice (directly or | |
| indirectly via another sufficiently-trusted participant) the collection-specific | |
| channel server-side closure (`ZX_CHANNEL_PEER_CLOSED` signaled at client end). | |
| Each participant then ensures that the list of cleanup conditions below is met | |
| for as many of the buffers as possible. In rare cases keeping a low number of | |
| buffers around for a while longer can make sense, such as for keeping a last | |
| video frame on-screen until a new video frame using a new collection is | |
| available. | |
| All references to a buffer must be removed before sysmem will destroy it and | |
| allow the memory to be reused. These include: | |
| * [Handles][handles] to the VMO. | |
| * [CPU mappings][map] of the VMO. | |
| * [Child VMOs][vmo_create_child] of the VMO. | |
| * [Pins][pmt] of the VMO for use by hardware. | |
| * [Channels][channel] to a Buffer Collection containing that VMOs. The sysmem | |
| server will consider a channel closed whether the close was initiated by the | |
| client or the server. This item does not apply to still-open channels that | |
| sent [`SetWeak`][SetWeak] previously. | |
| Under a non-preferred and less common model, all participants may close all | |
| collection-specific channels (with or without a [`Release`][Release] first) | |
| shortly after allocation, which triggers collection failure early (due to zero | |
| collection-specific channels remaining), but then the participants may continue | |
| to use the VMOs for a while despite the collection having "failed". Then later | |
| they use a separate mechanism to know when they want to clean up the VMOs (see | |
| list above). | |
| The non-preferred model technically works, but it can make the collection | |
| lifetime less clear, and make the sysmem inspect data less clear / less helpful. | |
| Nonetheless, in some rare situations this non-preferred model may be useful to | |
| accommodate constraints imposed by other protocols / interfaces. If considering | |
| this model, first consider having a participant send [`Release`][Release] and | |
| close its collection-specific channel, then learn of collection failure | |
| indirectly via another (sufficiently-trusted) participant; this way the overall | |
| setup still conforms to the preferred model described above. If the only reason | |
| this model is being considered is to allow incremental buffer deallocation, | |
| consider [SetWeak][SetWeak] instead. | |
| ## Advanced usage | |
| ### Fallback constraints and/or optional participant(s) | |
| A [`BufferCollectionTokenGroup`][BufferCollectionTokenGroup] is used in more | |
| complicated situations where an entire participant is optional / best-effort, or | |
| if there are fallback less-strict constraints that differ from preferred | |
| more-strict constraints (such as for performance reasons). | |
| Only one child token directly under the group (along with the sub-tree starting | |
| at that token) is selected during allocation, with the first child token of the | |
| group preferred, etc. | |
| A group under a token under a group is permitted (etc). | |
| The total number of group child selection combinations attempted during | |
| allocation is capped; if hitting this cap, feel free to reach out to discuss | |
| alternatives. | |
| This is advanced usage that's not directly relevant to most participants. See | |
| docs of [`BufferCollectionTokenGroup`][BufferCollectionTokenGroup]. | |
| ### Failure isolation and/or late participant join | |
| The [`SetDispensable`][SetDispensable] and [`AttachToken`][AttachToken] messages | |
| can be used to create a failure-isolated sub-tree (defined by token duplication | |
| parentage) and potentially later replace the sub-tree with a new sub-tree | |
| instance after the original sub-tree instance fails, without failing the root. | |
| The first [token][BufferCollectionToken] created by the initiator is what | |
| creates the root; the [collection channel][BufferCollection] later created from | |
| that token is still considered the root node of the tree (different protocol, | |
| but same "node"). | |
| When replacing the sub-tree, the new sub-tree participants see what looks like a | |
| normal allocation sequence, but really they're "joining" a buffer collection | |
| "already in progress". | |
| Use of [`AttachToken`][AttachToken] doesn't require prior use of | |
| [`SetDispensable`][SetDispensable]. | |
| Currently [`AttachToken`][AttachToken] requires that the collection already have | |
| sufficient buffers to satisfy the newly attached participant instance(s). | |
| This is advanced usage that's not directly relevant to most participants. See | |
| docs of [`SetDispensable`][SetDispensable] and [`AttachToken`][AttachToken]. | |
| ## Glossary | |
| ### Buffers | |
| A buffer represents a single image or other piece of memory an application | |
| will work with. Sysmem currently uses one [VMO][vmo] per buffer. Clients can | |
| [map][map] the memory onto the CPU or [pin][pmt] it to use it with a hardware | |
| block. | |
| ### Participants | |
| A participant is any application or driver that wants to access a buffer. All | |
| participants must connect to sysmem to negotiate memory formats. | |
| ### Image formats | |
| An [image format][ImageFormat] is the entire set of properties needed for a | |
| client to interpret the memory as a set of pixels. For example, it includes | |
| the pixel format, size in width and height, row pitch in bytes between rows, | |
| and color space. | |
| ### Buffer settings | |
| [Buffer settings][SingleBufferSettings] are a complete description of the | |
| properties of a buffer. These include the caching information, and other | |
| properties that participants may need to access the memory. For images the | |
| buffer settings will also have an image format. | |
| Buffer settings do not include the specific memory address, so multiple | |
| different buffers may have the same buffer settings. | |
| ### Heaps | |
| A [heap][Heap] represents one specific type of memory on a system. A | |
| system may have multiple heaps with different performance characteristics | |
| from each other. Some heaps may only be usable from a subset of hardware | |
| devices on a system. | |
| Some heaps may not be accessible from the CPU. For those heaps, the VMO | |
| representing a buffer cannot be used directly but is instead used as a key. | |
| An application wishing to use the buffer must send its VMO handle to the heap | |
| driver and the heap driver can return information about what memory to use. | |
| Example heaps: | |
| * Main system memory. | |
| * VRAM on a discrete GPU. | |
| * A carved-out area of system memory that's only usable by some pieces of | |
| hardware. | |
| ### Constraints | |
| [Constraints][constraints] specify the set of | |
| [BufferSettings][SingleBufferSettings] that a participant is able to use. | |
| Participants often specify multiple potential buffer settings in a single set | |
| of constraints, which gives sysmem the flexibility to pick any of them and | |
| reduces the risk that there are no settings that can satisfy the constraints | |
| from all participants. | |
| ### Negotiation | |
| Negotiation is the process where sysmem looks at all the constraints from | |
| participants and chooses buffer settings that work for all of them. If | |
| multiple settings could work, sysmem can use information about how clients | |
| will use the buffer and the architecture of the system to choose the optimal | |
| settings. | |
| ### Buffer collection | |
| A [buffer collection][BufferCollection] is a set of multiple buffers with all | |
| the same buffer settings. Sysmem allocates an entire buffer collection at | |
| once. Multiple participants may have FIDL channels open to the same buffer | |
| collection. | |
| ### Buffer collection token {#token} | |
| A [token][BufferCollectionToken] is used in the initial stages of the | |
| negotiation process before memory is allocated. Tokens can be duplicated and | |
| passed between processes before finally being [bound][bind] to a buffer | |
| collection. | |
| [vmo]: /docs/reference/kernel_objects/vm_object.md | |
| [sysmem]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2 | |
| [Heap]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#Heap | |
| [ImageFormat]: https://fuchsia.dev/reference/fidl/fuchsia.images2#ImageFormat | |
| [SingleBufferSettings]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#SingleBufferSettings | |
| [duplicates]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollectionToken.Duplicate | |
| [Allocator]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#Allocator | |
| [BufferCollectionToken]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollectionToken | |
| [BufferCollectionTokenGroup]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollectionTokenGroup | |
| [BufferCollection]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollection | |
| [channel]: /docs/reference/kernel_objects/channel.md | |
| [pmt]: /docs/reference/kernel_objects/pinned_memory_token.md | |
| [vmo_create_child]: /reference/syscalls/vmo_create_child.md | |
| [handles]: /docs/concepts/kernel/handles.md | |
| [bind]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#Allocator.BindSharedCollection | |
| [SetConstraints]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollection.SetConstraints | |
| [fidl]: /docs/development/languages/fidl/README.md | |
| [map]: /reference/syscalls/vmar_map.md | |
| [constraints]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollectionConstraints | |
| [AttachToken]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollection.AttachToken | |
| [SetDispensable]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollectionToken.SetDispensable | |
| [AttachLifetimeTracking]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollection.AttachLifetimeTracking | |
| [SetWeak]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollectionToken.SetWeak | |
| [Release]: https://fuchsia.dev/reference/fidl/fuchsia.sysmem2#BufferCollection.Release |