| // Copyright 2018 The Fuchsia Authors. All rights reserved. |
| // Use of this source code is governed by a BSD-style license that can be |
| // found in the LICENSE file. |
| |
| library fuchsia.media; |
| |
| using fuchsia.sysmem; |
| |
| /// StreamBufferConstraints |
| /// |
| /// This struct helps ensure that packet count and buffer space are sufficient |
| /// to avoid major problems. For example, a video decoder needs sufficient |
| /// video frame buffers to hold all potential reference frames concurrently + |
| /// one more video buffer to decode into. Else, the whole video decode pipe can |
| /// easily deadlock. |
| /// |
| /// The secondary purpose of this struct is to help ensure that packet count and |
| /// buffer space are sufficient to achieve reasonably performant operation. |
| /// |
| /// There are separate instances of this struct for stream input and stream |
| /// output. |
| /// |
| /// Notes about fields: |
| /// |
| /// For uncompressed video, separate and complete frames in their |
| /// separate buffers (buffer-per-packet mode) are always a requirement. |
| /// |
| /// per_packet_buffer_bytes.*: These per-packet buffer bytes constraints apply to |
| /// both buffer-per-packet mode and single-buffer mode (see `single_buffer_mode`). |
| /// If buffer-per-packet mode, the constraints apply to each buffer separately. |
| /// If single-buffer mode, the constraints need to be multiplied by the number |
| /// of packets to determine the constraints on the single buffer. |
| // |
| // TODO(dustingreen): Some of the buffer-focused fields in this structure will |
| // go away in favor of using sysmem for those aspects. The packet count fields |
| // will stay. The single-buffer fields will stay. |
| table StreamBufferConstraints { |
| /// This is a version number the server sets on the constraints to allow the |
| /// server to determine when the client has caught up with the latest |
| /// constraints sent by the server. The server won't emit output data until |
| /// the client has configured output settings and buffers with a |
| /// buffer_constraints_version_ordinal >= the latest |
| /// buffer_constraints_version_ordinal that had |
| /// buffer_constraints_action_required true. See |
| /// buffer_constraints_action_required comments for more. |
| /// |
| /// A buffer_constraints_version_ordinal of 0 is not permitted, to simplify |
| /// initial state handling. Other than 0, both odd and even version ordinals |
| /// are allowed (in contrast to the stream_lifetime_ordinal, neither the |
| /// client nor server ever has a reason to consider the latest version to be |
| /// stale, so there would be no benefit to disallowing even values). |
| 1: uint64 buffer_constraints_version_ordinal; |
| |
| /// default_settings |
| /// |
| /// These settings are "default" settings, not "recommended" settings. |
| /// |
| /// These "default" settings can be passed to SetInputBufferSettings() / |
| /// SetOutputBufferSettings() as-is without modification, but a client doing |
| /// that must still obey the semantics of packet_count_for_client, despite |
| /// the stream processor server not having any way to really know the proper |
| /// setting for that field. |
| /// |
| /// For StreamBufferConstraints fields whose names end in "recommended", the |
| /// default_settings will have the corresponding setting field set to that |
| /// recommended value. |
| /// |
| /// The stream processor promises that these default settings as-is (except |
| /// for buffer_lifetime_ordinal) are guaranteed to satisfy the constraints |
| /// indicated by the other fields of StreamBufferConstraints. While |
| /// client-side checking that these settings are within the constraints is |
| /// likely unnecessary in the client, the client should still check that |
| /// these values are within client-side reasonable-ness bounds before using |
| /// these values, to avoid letting a stream processor server cause problems |
| /// for the client. |
| /// |
| /// This structure will always have `single_buffer_mode` false. See |
| /// `single_buffer_mode_allowed` for whether `single_buffer_mode` true is |
| /// allowed. |
| /// |
| /// The client must set the buffer_lifetime_ordinal field to a proper value |
| /// before sending back to the server. The 0 initially in this field will |
| /// be rejected by the server if sent back as-is. See comments on |
| /// StreamBufferSettings.buffer_lifetime_ordinal. |
| 2: StreamBufferSettings default_settings; |
| |
| /// If a client is using buffer per packet mode, each buffer must be at least |
| /// this large. If a client is using single-buffer mode, the one buffer must |
| /// be at least per_packet_buffer_bytes_min * packet_count_for_server_min in |
| /// size. |
| 3: uint32 per_packet_buffer_bytes_min; |
| /// Must be >= per_packet_buffer_bytes_min. Delivering more than |
| /// this per input packet might not perform any better, and in fact might |
| /// perform worse. |
| 4: uint32 per_packet_buffer_bytes_recommended; |
| /// Must be >= per_packet_buffer_bytes_recommended. Can be 0xFFFFFFFF if |
| /// there is no explicitly-enforced limit. |
| 5: uint32 per_packet_buffer_bytes_max; |
| |
| /// Minimum number of packet_count_for_server. |
| /// |
| /// Re. input and output: |
| /// |
| /// This is a strict min for packet_count_for_server, but a client can use |
| /// more packets overall if the client wants to, by using a larger value for |
| /// packet_count_for_server and/or using a non-zero packets_for_client. A |
| /// good reason to do the former would be if the client might occasionally |
| /// deliver a few not-very-full buffers - or to have a few |
| /// extra packets within which to satisfy stream_input_bytes_min. A good |
| /// reason to do the latter would be if a client needs to hold onto some |
| /// packets for any extra duration. |
| /// |
| /// If a client specifies a larger packet_count_for_server value than |
| /// packet_count_for_server_min, a server is permitted (but not encouraged) |
| /// to not make progress until packet_count_for_server are with the server, |
| /// not merely packet_count_for_server_min. |
| /// |
| /// For decoder input and audio encoder input: The |
| /// packet_count_for_server_min may or may not contain enough data to allow |
| /// the stream processor to make progress without copying into an internal |
| /// side buffer. If there isn't enough data delivered in |
| /// packet_count_for_server_min packets to permit progress, the stream |
| /// processor must copy into its own side buffer internally to make |
| /// progress. |
| /// |
| /// If a client intends to use extra packets for client-side purposes, the |
| /// client should specify the extra packets in packets_for_client instead of |
| /// packet_count_for_server, but packet_count_for_server must still be >= |
| /// packet_count_for_server_min. |
| 6: uint32 packet_count_for_server_min; |
| |
| /// This must be at least packet_count_for_server_min and at most |
| /// packet_count_for_server_recommended_max. |
| /// |
| /// This value is likely to be used as-is by most clients, so if having one |
| /// additional packet is a big performance win in a large percentage of |
| /// scenarios, it can be good for the server to include that additional |
| /// packet in this value. |
| 7: uint32 packet_count_for_server_recommended; |
| |
| /// This can be the same as packet_count_for_server_max or can be lower. |
| /// Values above this value and <= packet_count_for_server_max are not |
| /// recommended by the stream processor, but should still work given |
| /// sufficient resources available to both the client and the stream |
| /// processor. |
| 8: uint32 packet_count_for_server_recommended_max; |
| |
| /// This can be 0xFFFFFFFF if there's no stream processor-enforced max, but |
| /// stream processors are encouraged to set a large but still |
| /// plausibly-workable max, and clients are encouraged to request a number |
| /// of packets that isn't excessively large for the client's scenario. |
| 9: uint32 packet_count_for_server_max; |
| |
| /// Normally this would be an implicit 0, but for now we have a min so we can |
| /// force the total number of packets to be a specific number that we know |
| /// works for the moment. |
| 10: uint32 packet_count_for_client_min; |
| |
| /// The client must set packet_count_for_client to be <= |
| /// packet_count_for_client_max. |
| /// |
| /// This value must be at least 1. This can be 0xFFFFFFFF if there's no |
| /// stream-processor-enforced max. Clients are encouraged to request a |
| /// number of packets that isn't excessively large for the client's |
| /// scenario. |
| 11: uint32 packet_count_for_client_max; |
| |
| /// `single_buffer_mode_allowed` false allows a stream processor that's not |
| /// required to support single-buffer mode for a given input or output the |
| /// ability to decline to support single-buffer mode on that input/output. |
| /// |
| /// All encoder output, regardless of audio or video: server support for |
| /// single-buffer mode is optional. |
| /// |
| /// Audio decoder output: server support for single-buffer mode is required. |
| /// |
| /// Video decoder output: There is little reason for a video decoder to |
| /// support single-buffer mode on output. Nearly all video decoders will set |
| /// this to false for their output. |
| /// |
| /// All decoder inputs: Servers must support single-buffer mode on input. |
| /// The client is responsible for managing the input buffer space such that |
| /// filling an input packet doesn't over-write any portion of an input |
| /// packet already in flight to the stream processor. |
| /// |
| /// Encoder inputs: Server support for single-buffer mode on encoder input is |
| /// optional. This is more often useful for audio than for video. |
| /// |
| /// Support for buffer-per-packet mode is always required on both input and |
| /// output, regardless of stream processor type. |
| 12: bool single_buffer_mode_allowed; |
| |
| /// If true, the buffers need to be physically contiguous pages, such as those |
| /// allocated using zx_vmo_create_contiguous(). |
| 13: bool is_physically_contiguous_required; |
| }; |
| |
| /// The stream-processor-controlled output configuration, including both |
| /// StreamBufferConstraints for the output and FormatDetails for the output. |
| // |
| // TODO(dustingreen): Need a better name for this struct, but still short |
| // hopefully. It's stuff the stream processor gets to control, not the client. |
| // It's different than output buffer settings, which the client does get to |
| // control to some extent. It's different than any configurable output |
| // settings the client might specify for output of an encoder. |
| table StreamOutputConstraints { |
| /// A client which always immediately re-configures output buffers on |
| /// receipt of OnOutputConstraints() with buffer_constraints_action_required |
| /// true can safely ignore this field. |
| /// |
| /// A client is permitted to ignore an OnOutputConstraints() message even with |
| /// buffer_constraints_action_required true if the client knows the server |
| /// has already been told to discard the remainder of the stream with the |
| /// same stream_lifetime_ordinal or if this stream_lifetime_ordinal field is |
| /// set to 0. The server is required to re-send needed output config via |
| /// OnOutputConstraints() with new stream_lifetime_ordinal and |
| /// buffer_constraints_action_required true, if the most recent completed |
| /// server-side output config isn't what the server wants/needs yet for the |
| /// new stream. |
| 1: uint64 stream_lifetime_ordinal; |
| |
| /// When the buffer constraints are delivered, they indicate whether action |
| /// is required. A false value here permits delivery of constraints which |
| /// are fresher without forcing a buffer reconfiguration. If this is false, |
| /// a client cannot assume that it's safe to immediately re-configure output |
| /// buffers. If this is true, the client can assume it's safe to |
| /// immediately configure output buffers once. |
| /// |
| /// A client is permitted to ignore buffer constraint versions which have |
| /// buffer_constraints_action_required false. The server is not permitted |
| /// to change buffer_constraints_action_required from false to true for the |
| /// same buffer_constraints_version_ordinal. |
| /// |
| /// For each configuration, a client must use new buffers, never buffers |
| /// that were previously used for anything else, and never buffers |
| /// previously used for any other StreamProcessor purposes. This rule |
| /// exists for multiple good reasons, relevant to both mid-stream changes, |
| /// and changes on stream boundaries. A client should just use new buffers |
| /// each time. |
| /// |
| /// When this is true, the server has already de-refed as many low-level |
| /// output buffers as the server can while still performing efficient |
| /// transition to the new buffers and will de-ref the rest asap. A Sync() |
| /// is not necessary to achieve non-overlap of resource usage to the extent |
| /// efficiently permitted by the formats involved. |
| /// |
| /// If buffer_constraints_action_required is true, the server _must_ not |
| /// deliver more output data until after output buffers have been configured |
| /// (or re-configured) by the client. |
| 2: bool buffer_constraints_action_required; |
| 3: StreamBufferConstraints buffer_constraints; |
| }; |
| |
| table StreamOutputFormat { |
| /// A client is permitted to ignore an OnOutputFormat() message even with |
| /// buffer_constraints_action_required true if the client knows the server |
| /// has already been told to discard the remainder of the stream with the |
| /// same stream_lifetime_ordinal or if this stream_lifetime_ordinal field is |
| /// set to 0. The server is required to re-send needed output config via |
| /// OnOutputConstraints() with new stream_lifetime_ordinal and |
| /// buffer_constraints_action_required true, if the most recent completed |
| /// server-side output config isn't what the server wants/needs yet for the |
| /// new stream. |
| /// |
| /// The server is required to send an OnOutputFormat() before the first |
| /// output packet of a stream. |
| 1: uint64 stream_lifetime_ordinal; |
| |
| /// If format_details.format_details_version_ordinal changes, the client |
| /// should inspect the new format details and determine if it must adjust to |
| /// the new format. The server guarantees that if the format has changed, then |
| /// format_details.format_details_version_ordinal will change, but a change |
| /// to format_details.format_details_version_ordinal does not guarantee that |
| /// the format details actually changed. Servers are strongly encouraged to |
| /// not change format_details.format_details_version_ordinal other than |
| /// before the first output data of a stream unless there is a real |
| /// mid-stream format change in the stream. Unnecessary mid-stream format |
| /// changes can cause simpler clients that have no need to handle mid-stream |
| /// format changes to just close the channel. Format changes before the |
| /// first output data of a stream are not "mid-stream" in this context - |
| /// those can be useful for stream format detection / setup reasons. |
| /// |
| /// Note that in case output buffers don't really need to be re-configured |
| /// despite a format change, a server is encouraged, but not required, to |
| /// set buffer_constraints_action_required false on the message that conveys |
| /// the new format details. Simpler servers may just treat the whole output |
| /// situation as one big thing and demand output buffer reconfiguration on |
| /// any change in the output situation. |
| /// |
| /// A client may or may not actually handle a new buffer_constraints with |
| /// buffer_constraints_action_required false, but the client should always |
| /// track the latest format_details. |
| /// |
| /// An updated format_details is ordered with respect to emitted output |
| /// packets, and applies to all subsequent packets until the next |
| /// format_details with larger version_ordinal. A simple client that does |
| /// not intend to handle mid-stream format changes should still keep track |
| /// of the most recently received format_details until the first output |
| /// packet arrives, then lock down the format details, handle those format |
| /// details, and verify that any |
| /// format_details.format_details_version_ordinal received from the server |
| /// is the same as the locked-down format_details, until the client is done |
| /// with the stream. Even such a simple client must tolerate |
| /// format_details.format_details_version_ordinal changing multiple times |
| /// before the start of data output from a stream (any stream - the first |
| /// stream or a subsequent stream). This allows a stream processor to |
| /// request that output buffers and output format be configured |
| /// speculatively, and for the output config to be optionally adjusted by |
| /// the server before the first data output from a stream after the server |
| /// knows everything it needs to know to fully establish the initial output |
| /// format details. This simplifies stream processor server implementation, |
| /// and allows a clever stream processor server to guess it's output config |
| /// for lower latency before any input data, while still being able to fix |
| /// the output config (including format details) if the guess turns out to |
| /// be wrong. |
| /// |
| /// Whether the format_details.format_details_version_ordinal will actually |
| /// change mid-stream is a per-stream-processor and per-stream detail that |
| /// is not specified in comments here, and in most cases also depends on |
| /// whether the format changes on the input to the stream processor. |
| /// Probably it'll be fairly common for a client to use a format which |
| /// technically supports mid-stream format change, but the client happens to |
| /// know that none of the streams the client intends to process will ever |
| /// have a mid-stream format change. |
| 2: FormatDetails format_details; |
| }; |
| |
| [Deprecated = "Use StreamOutputFormat instead."] |
| table StreamOutputConfig { |
| 1: uint64 stream_lifetime_ordinal; |
| 2: bool buffer_constraints_action_required; |
| 3: StreamBufferConstraints buffer_constraints; |
| 4: FormatDetails format_details; |
| }; |
| |
| /// Default values for input and output |
| /// StreamBufferConstraints.default_settings.packet_count_for_server. |
| /// |
| /// These are defined as "const" in FIDL to avoid all server implementations |
| /// needing to separately define their own values, and these should be |
| /// reasonable as default values, but strictly speaking this is not intended to |
| /// promise that this value won't change from build to build. If a client cares |
| /// about a specific number, the client should separately define what that |
| /// number is and ensure that StreamBufferSettings.packet_count_for_client is |
| /// at least large enough. |
| /// |
| /// In contrast to packet_count_for_client, the packet_count_for_server is much |
| /// more stream-processor-specific, so this file has no numbers for that - each |
| /// stream processor will set those as appropriate for the specific stream |
| /// processor. |
| /// |
| /// These are not "recommended" values, only "default" values, in the sense that |
| /// the stream processor doesn't really know what the correct setting for these |
| /// values is for a given client, and if the default is not appropriate for a |
| /// client, large problems could result such as deadlock. See the comments on |
| /// packet_count_for_client. |
| /// |
| /// Despite these defaults, every client should ideally care about the |
| /// packet_count_for_client setting and should ensure that the setting is at |
| /// least large enough to cover the number of packets the client might ever need |
| /// to camp on for any non-transient duration concurrently. The defaults are |
| /// only intended to be plausible for some clients, not all clients. |
| /// |
| /// One for the client to be filling and one in transit. |
| const uint32 kDefaultInputPacketCountForClient = 2; |
| /// One for the client to be rendering, and one in transit. |
| const uint32 kDefaultOutputPacketCountForClient = 2; |
| |
| /// For input, this is the default on a fairly arbitrary basis. |
| // |
| // TODO(dustingreen): Do we want the default for audio encoding to be |
| // `single_buffer_mode` true instead? If so, we may split this up by audio/video |
| // encoder/decoder. |
| const bool kDefaultInputIsSingleBufferMode = false; |
| const bool kDefaultOutputIsSingleBufferMode = false; |
| |
| /// See relevant corresponding constraints in StreamBufferConstraints. The |
| /// settings must satisfy the constraints. |
| /// |
| /// The client informs the stream processor of these settings and then |
| /// separately informs the stream processor of each buffer. |
| /// |
| /// The total packet count is split into two pieces to disambiguate how many |
| /// packets are allocated for the client to hold onto for whatever reason, |
| /// vs. how many packets are allocated for the server to hold onto for |
| /// whatever reason. |
| /// |
| /// Extra packets to provide slack for performance reasons can be in either |
| /// category, but typically packet_count_for_server_recommended will already |
| /// include any performance-relevant slack for the server's benefit. |
| table StreamBufferSettings { |
| /// The containing message starts a new buffer_lifetime_ordinal. |
| /// |
| /// There is a separate buffer_lifetime_ordinal for input vs. output. |
| /// |
| /// Re-use of the same value is not allowed. Values must be odd. Values |
| /// must only increase (increasing by more than 2 is permitted). |
| /// |
| /// A buffer_lifetime_ordinal lifetime starts at SetInputBufferSettings() or |
| /// SetOutputBufferSettings(), and ends at the earlier of |
| /// CloseCurrentStream() with release_input_buffers/release_output_buffers |
| /// set or SetOutputBufferSettings() with new buffer_lifetime_ordinal in the |
| /// case of mid-stream output config change. |
| 1: uint64 buffer_lifetime_ordinal; |
| |
| /// This value indicates which version of constraints the client is/was aware |
| /// of so far. |
| /// |
| /// For input, this must always be 0 because constraints don't change for |
| /// input (settings can change, but there's no settings vs current |
| /// constraints synchronization issue on input). |
| /// |
| /// For output, this allows the server to know when the client is |
| /// sufficiently caught up before the server will generate any more output. |
| /// |
| /// When there is no active stream, a client is permitted to re-configure |
| /// buffers again using the same buffer_constraints_version_ordinal. |
| 2: uint64 buffer_constraints_version_ordinal; |
| |
| /// How many packets the client is allocating for the stream processor |
| /// server's use. This must be >= |
| /// StreamBufferConstraints.packet_count_for_server_min. If constraints |
| /// change such that this would no longer be true, the server will send an |
| /// OnOutputConstraints() event. |
| /// |
| /// The stream processor server is allowed to demand that all of |
| /// packet_count_for_server become free before making further progress, even |
| /// if packet_count_for_server is > packet_count_for_server_min. |
| /// |
| /// A reasonable value for this is |
| /// StreamBufferConstraints.packet_count_for_server_recommended. |
| 3: uint32 packet_count_for_server; |
| |
| /// This must be at least 1. The server will close the channel if this is 0. |
| /// |
| /// How many packets the client is allocating for the client's use. The |
| /// client may hold onto this many packets for arbitrarily-long duration |
| /// without handing these packets to the stream processor, and despite doing |
| /// so, the stream processor will continue to make progress and function |
| /// normally without getting stuck. The client holding onto additional |
| /// packets transiently is ok, but the client needs to hand those additional |
| /// packets back to the stream processor eventually if the client wants the |
| /// stream processor to make further progress. |
| /// |
| /// In addition to this value needing to include at least as many packets as |
| /// the client ever intends to concurrently camp on indefinitely, any extra |
| /// slack to benefit client-side performance should also be included here. |
| /// |
| /// A typical value for this could be at least 2, but it depends strongly on |
| /// client implementation and overall client buffering goals. It is up to |
| /// the client to determine how many packets are needed in this category by |
| /// any parts of the overall system that will be holding onto packets for |
| /// any reason. Those parts of the system should have a documented and |
| /// possibly queryable defined value to help determine this number. Setting |
| /// this value lower than it actually needs to be can result in the stream |
| /// processor not making progress as it sits waiting for packets, with the |
| /// client unable to recycle any more packets to the stream processor. That |
| /// situation can be difficult to diagnose, while excessively-large values |
| /// here are wasteful, so care is warranted to set this value properly. |
| 4: uint32 packet_count_for_client; |
| |
| /// In buffer-per-packet mode, we require that each buffer have usable bytes |
| /// equal to per_packet_buffer_bytes. Use of differently-sized low-level |
| /// buffers is possible, but the size of the portion used via the |
| /// StreamProcessor interface per StreamBuffer must be the same for all the |
| /// buffers. |
| /// |
| /// In single-buffer mode, we require the portion of the low-level buffer |
| /// used via the StreamProcessor interface to be size |
| /// (packet_count_for_server + packet_count_for_client) * |
| /// per_packet_buffer_bytes. |
| // |
| // TODO(dustingreen): determine if we need to relax these restrictions a bit |
| // for convenience when using gralloc video buffers. |
| 5: uint32 per_packet_buffer_bytes; |
| |
| /// If true there is only one buffer, with index 0, which all packets |
| /// must explicitly refer to with buffer_index == 0. |
| /// |
| /// While it's possible to set up `single_buffer_mode` false with each buffer |
| /// referring to the same underlying VMO, `single_buffer_mode` true is more |
| /// efficient for that case since only one mapping is created. |
| 6: bool single_buffer_mode; |
| }; |
| |
| /// This struct is used instead of StreamBufferSettings when sysmem is used to |
| /// allocate buffers. The settings in StreamBufferSettings that are missing from |
| /// StreamBufferPartialSettings can be conveyed from the client directly to |
| /// sysmem. |
| // |
| // TODO(dustingreen): Consider renaming this to StreamBufferSettings once that's |
| // an option. |
| table StreamBufferPartialSettings { |
| /// The containing message starts a new buffer_lifetime_ordinal. |
| /// |
| /// There is a separate buffer_lifetime_ordinal for input vs. output. |
| /// |
| /// Re-use of the same value is not allowed. Values must be odd. Values |
| /// must only increase (increasing by more than 2 is permitted). |
| /// |
| /// A buffer_lifetime_ordinal lifetime starts at SetInputBufferSettings() or |
| /// SetOutputBufferSettings(), and ends at the earlier of |
| /// CloseCurrentStream() with release_input_buffers/release_output_buffers |
| /// set or SetOutputBufferSettings() with new buffer_lifetime_ordinal in the |
| /// case of mid-stream output config change. |
| 1: uint64 buffer_lifetime_ordinal; |
| |
| /// This value indicates which version of constraints the client is/was aware |
| /// of so far. |
| /// |
| /// For input, this must always be 0 because constraints don't change for |
| /// input (settings can change, but there's no settings vs current |
| /// constraints synchronization issue on input). |
| /// |
| /// For output, this allows the server to know when the client is |
| /// sufficiently caught up before the server will generate any more output. |
| /// |
| /// When there is no active stream, a client is permitted to re-configure |
| /// buffers again using the same buffer_constraints_version_ordinal. |
| 2: uint64 buffer_constraints_version_ordinal; |
| |
| /// If true, there is only one buffer, but still typically more than one |
| /// packet. If false, the # of packets == the number of buffers. |
| /// |
| /// While it's possible to set up `single_buffer_mode` false with each buffer |
| /// referring to the same underlying VMO, `single_buffer_mode` true is more |
| /// efficient for that case since only one mapping is created. |
| /// |
| /// This setting is specified by the client, and influences the constraints |
| /// delivered from the StreamProcessor to sysmem (whether there's more than |
| /// one buffer allocated overall or not). For `single_buffer_mode` true, the |
| /// StreamProcessor is the one to ask sysmem for a buffer - the client should |
| /// refrain from doing so or the StreamProcessor will just fail when more |
| /// than one buffer gets allocated by sysmem. |
| 3: bool single_buffer_mode; |
| |
| /// When `single_buffer_mode` is false: |
| /// |
| /// The actual packet count will be |
| /// max(packet_count_for_server + packet_count_for_client, sysmem_buffers). |
| /// The sysmem_buffers is BufferCollectionInfo.buffer_count from sysmem if |
| /// using sysmem, or 0 if not using sysmem. |
| /// |
| /// When `single_buffer_mode` is true: |
| /// |
| /// The actual packet count is packet_count_for_server + |
| /// packet_count_for_client. |
| /// |
| /// If not using sysmem, or if using `single_buffer_mode`, these fields must be |
| /// set and consistent with correpsonding fields in StreamBufferConstraints. |
| /// |
| /// If `single_buffer_mode` false and using sysmem, these fields can both be |
| /// non-set, or can both be set and consistent with correpsonding fields in |
| /// StreamBufferConstraints. If not set, the value used for the fields in |
| /// the "max" expression above is 0, so buffer_count. |
| // TODO(FIDL-609): Default to 0. |
| 4: uint32 packet_count_for_server; |
| |
| // TODO(FIDL-609): Default to 0. |
| 5: uint32 packet_count_for_client; |
| |
| /// The client end of a BufferCollectionToken channel, which the |
| /// StreamProcessor will use to deliver constraints to sysmem and learn of |
| /// buffers allocated by sysmem. |
| /// |
| /// The client guarantees that the token is already known to sysmem (via |
| /// BufferCollectionToken.Sync(), BufferCollection.Sync(), or |
| /// BufferCollectionEvents.OnDuplicatedTokensKnownByServer()). |
| 6: fuchsia.sysmem.BufferCollectionToken sysmem_token; |
| }; |
| |
| /// The StreamBuffer struct represents a pre-configured buffer. |
| /// |
| /// Both input and output uses StreamBuffer(s), but the two sets of buffers are |
| /// separate. |
| /// |
| /// The client uses SetInputBufferSettings() + AddInputBuffer() * N to inform |
| /// the stream processor about all the input buffers. |
| /// |
| /// The client uses SetOutputBufferSettings() + AddOutputBuffer() * N to inform |
| /// the stream processor about all the output buffers. |
| /// |
| /// When `single_buffer_mode` is true, there is only buffer_index 0 shared by all |
| /// packet(s) of the relevant input or output. |
| table StreamBuffer { |
| /// When using AddOutputBuffer()/AddInputBuffer(), this must match the |
| /// buffer_lifetime_ordinal of the most recent |
| /// SetOutputBufferSettings()/SetInputBufferSettings(). |
| 1: uint64 buffer_lifetime_ordinal; |
| |
| /// Buffers must be added via AddOutputBuffer() / AddInputBuffer() in order |
| /// by buffer_index. The buffer_index is always equal to 0 if |
| /// `single_buffer_mode` is true. |
| 2: uint32 buffer_index; |
| |
| /// For each new buffer_lifetime_ordinal, a client must use new low-level |
| /// buffers. This rule exists for multiple very good reasons, and is |
| /// relevant to mid-stream changes, changes on stream boundaries, and both |
| /// input and output buffers. A new buffer_lifetime_ordinal needs new |
| /// low-level buffers, not just new StreamBuffer(s). If you find yourself |
| /// copying compressed input data into new low-level input buffers solely |
| /// due to this rule, consider asking the source of the data for the ability |
| /// to directly fill new VMOs. The rule exists for good reasons, even for |
| /// input buffers. |
| /// |
| /// The previous paragraph does not prohibit carving up VMOs into sub-pieces |
| /// and using different sub-pieces as different StreamBuffer(s), with some |
| /// VMOs used for more than one StreamBuffer and possibly others used for |
| /// only one StreamBuffer. While this is permitted and enables some |
| /// optimizations, it's not expected to be particularly common. |
| 3: StreamBufferData data; |
| }; |
| |
| /// For the moment, a VMO per buffer is the only type of buffer. |
| /// |
| /// This is extremely likely to change significantly when adding gralloc stuff, |
| /// but the idea with this union is to have a struct per logical way of storing |
| /// the data. Any multi-domain storage within a gralloc buffer will likely be |
| /// only indirectly represented here. |
| union StreamBufferData { |
| 1: StreamBufferDataVmo vmo; |
| |
| // TODO(dustingreen): add the gralloc way |
| }; |
| |
| /// StreamBufferDataVmo |
| /// |
| /// Details for a buffer backed by a VMO. |
| table StreamBufferDataVmo { |
| /// The same VMO can be used by more than one StreamBuffer (only of the same |
| /// buffer_lifetime_ordinal), but each vmo_handle must be a separate handle. |
| 1: handle<vmo> vmo_handle; |
| |
| /// Offset within the VMO of the first usable byte. Must be < the VMO's size |
| /// in bytes. |
| 2: uint64 vmo_usable_start; |
| |
| /// VMO-relative offset that's one past the last usable byte. This can point |
| /// one byte beyond the end of the VMO if desired. In other words, this can |
| /// be equal to the VMO's size, to indicate that the last byte of the VMO is |
| /// usable (and possibly many byte before that, depending on |
| /// vmo_usable_start). |
| 3: uint64 vmo_usable_size; |
| }; |
| |
| /// PacketHeader |
| /// |
| /// When referring to a free packet, we use PacketHeader alone instead of |
| /// Packet, since while a packet is free it doesn't really have meaningful |
| /// offset or length etc. |
| /// |
| /// A populated Packet also has a PacketHeader. |
| table PacketHeader { |
| /// This is which buffer configuration lifetime this header is referring to. |
| /// |
| /// A packet_index is only really meaningful with respect to a particular |
| /// buffer_lifetime_ordinal. |
| /// |
| /// See StreamBufferSettings.buffer_lifetime_ordinal. |
| /// |
| /// For QueueInputPacket(), a server receiving a buffer_lifetime_ordinal that |
| /// isn't the current input buffer_lifetime_ordinal will close the channel. |
| /// |
| /// For OnFreeInputPacket() and RecycleOutputPacket(), the receiver (client |
| /// or server) must ignore a message with stale buffer_lifetime_ordinal. |
| 1: uint64 buffer_lifetime_ordinal; |
| |
| /// The overall set of packet_index values is densely packed from 0..count-1 |
| /// for input and output separately. They can be queued in any order. |
| /// |
| /// Both the client and server should validate the packet_index against the |
| /// known bound and disconnect if it's out of bounds. |
| /// |
| /// When running in single-buffer mode, the buffer index is always 0. |
| /// |
| /// The packet_index values don't imply anything about order of use of |
| /// packets. The client should not expect the ordering to remain the same |
| /// over time - the stream processor is free to hold on to an input or |
| /// output packet for a while during which other packet_index values may be |
| /// used multiple times. |
| /// |
| /// For a given properly-functioning StreamProcessor instance, packet_index |
| /// values will be unique among concurrently-outstanding packets. Servers |
| /// should validate that a client isn't double-using a packet and clients |
| /// should validate as necessary to avoid undefined or unexpected client |
| /// behavior. |
| 2: uint32 packet_index; |
| }; |
| |
| /// A Packet represents a chunk of input or output data to or from a stream |
| /// processor. |
| /// |
| /// stream processor output: |
| /// |
| /// While the Packet is outstanding with the client via OnOutputPacket(), the |
| /// stream processor will avoid modifying the referenced output data. After the |
| /// client calls RecycleOutputPacket(packet_index), the stream processor is |
| /// notified that the client is again ok with the referenced data changing. |
| /// |
| /// stream processor input: |
| /// |
| /// The client initially has all packet_index(es) available to fill, and later |
| /// gets packet_index(s) that are again ready to fill via OnFreeInputPacket(). |
| /// The client must not modify the referenced data in between QueueInputPacket() |
| /// and OnFreeInputPacket(). |
| table Packet { |
| 1: PacketHeader header; |
| |
| /// Which buffer this packet refers to. For single-buffer mode this will |
| /// always be 0, but for multi-buffer mode, a given in-flight interval of a |
| /// packet can refer to any buffer. The packet has an associated buffer only |
| /// while the packet is in-flight, not while the packet is free. |
| /// |
| /// The default value makes accidental inappropriate use of index 0 less |
| /// likely (will tend to complain in an obvious way if not filled out |
| /// instead of a non-obvious data corruption when decoding buffer 0 |
| /// repeatedly instead of the correct buffers). |
| /// |
| /// TODO(dustingreen): Try to make FIDL table defaults have meaning, and not |
| /// complain about !has when accessing the field. For now the default |
| /// specified here does nothing. |
| // TODO(FIDL-609): Default to 0x80000000. |
| 2: uint32 buffer_index; |
| |
| /// The value 1 is the lowest permitted value after stream processor |
| /// creation. Values sent by the client must be odd. Values must only |
| /// increase. |
| /// |
| /// A stream_lifetime_ordinal represents the lifetime of a stream. All |
| /// messages that are specific to a stream have the stream_lifetime_ordinal |
| /// value and the value is the same for all messages relating to a given |
| /// stream. |
| 3: uint64 stream_lifetime_ordinal; |
| |
| /// Which part of the relevant buffer is this packet using. These are valid |
| /// for input data that's in-flight to the stream processor, and are valid |
| /// for output data from the stream processor. |
| /// |
| /// For compressed formats and uncompressed audio, the data in |
| /// [start_offset, start_offset + valid_length_bytes) is the contiguously |
| /// valid data referred to by this packet. |
| /// |
| /// For uncompressed video frames, FormatDetails is the primary means of |
| /// determining which bytes are relevant. The offsets in FormatDetails |
| /// are relative to the start_offset here. The valid_length_bytes must be |
| /// large enough to include the full last line of pixel data, including the |
| /// full line stride of the last line (not just the width in pixels of the |
| /// last line). |
| /// |
| /// Despite these being filled out, some uncompressed video buffers are of |
| /// types that are not readable by the CPU. These fields being here don't |
| /// imply there's any way for the CPU to read an uncompressed frame. |
| // |
| // TODO(dustingreen): Do we have any reason to require that these be filled |
| // out for opaque uncompressed video frames that the CPU can't read? In |
| // that case do we want to require them just so they can be potentially |
| // passed on to a HW renderer in case the HW renderer has any use for them? |
| // Or more likely, it may just be that these tend to refer to the whole |
| // size of an uncompressed buffer, with format_details taking care of |
| // specifying which bytes are actually relevant. |
| 4: uint32 start_offset; |
| |
| /// This must be > 0. |
| /// |
| /// The semantics for valid data per packet vary depending on data type as |
| /// follows. |
| /// |
| /// uncompressed video - A video frame can't be split across packets. Each |
| /// packet is one video frame. |
| /// |
| /// uncompressed audio - Regardless of float or int, linear or uLaw, or |
| /// number of channels, a packet must contain an non-negative number of |
| /// complete audio frames, where a single audio frame consists of data for |
| /// all the channels for the same single point in time. Any |
| /// stream-processor-specific internal details re. lower rate sampling for |
| /// LFE channel or the like should be hidden by the StreamProcessor server |
| /// implementation. |
| /// |
| /// compressed data input - A packet must contain at least one byte of data. |
| /// See also stream_input_bytes_min. Splitting AUs at arbitrary byte |
| /// boundaries is permitted, including at boundaries that are in AU headers. |
| /// |
| /// compressed data output - The stream processor is not required to fully |
| /// fill each output packet's buffer. |
| 5: uint32 valid_length_bytes; |
| |
| /// This value is not strictly speaking a timestamp. It is an arbitrary |
| /// unsigned 64-bit number that, under some circumstances, will be passed by |
| /// a stream processor unmodified from an input packet to the |
| /// exactly-corresponding output packet. |
| /// |
| /// For timestamp_ish values to be propagated from input to output the |
| /// following conditions must be true: |
| /// * promise_separate_access_units_on_input must be true |
| /// * has_timestamp_ish must be true for a given input packet, to have that |
| /// timestamp_ish value (potentially) propagate through to an output |
| /// * the StreamProcessor instance itself decides (async) that the input |
| /// packet generates an output packet - if a given input never generates |
| /// an output packet then the timestamp_ish value on the input will never |
| /// show up on any output packet - depending on the characteristics of the |
| /// input and output formats, and whether a decoder is willing to join |
| /// mid-stream, etc this can be more or less likely to occur, but clients |
| /// should be written to accommodate timestamp_ish values that are fed on |
| /// input but never show up on output, at least to a reasonable degree |
| /// (not crashing, not treating as an error). |
| 6: uint64 timestamp_ish; |
| |
| /// start_access_unit |
| /// |
| /// If promise_separate_access_units_on_input (TODO(dustingreen): or any |
| /// similar mode for output) is true, this bool must be set appropriately |
| /// depending on whether byte 0 _is_ or _is not_ the start of an access |
| /// unit. The client is required to know, and required to set this boolean |
| /// properly. The server is allowed to infer that when this boolean is |
| /// false, byte 0 is the first byte of a continuation of a |
| /// previously-started AU. (The byte at start_offset is "byte 0".) |
| /// |
| /// If promise_separate_access_units_on_input is false, this boolean is |
| /// ignored. |
| 7: bool start_access_unit; |
| |
| /// known_end_access_unit |
| /// |
| /// A client is never required to set this boolean to true. |
| /// |
| /// If promise_separate_access_units_on_input is true, for input data, this |
| /// boolean must be false if the last byte of this packet is not the last |
| /// byte of an AU, and this boolean _may_ be true if the last byte of this |
| /// packet is the last byte of an AU. A client delivering one AU at a time |
| /// that's interested in the lowest possible latency via the decoder should |
| /// set this boolean to true when it can be set to true. |
| /// |
| /// If promise_separate_access_units_on_input is false, this boolean is |
| /// ignored. |
| 8: bool known_end_access_unit; |
| }; |
| |
| /// Overview of operation: |
| /// |
| /// 1. Create |
| /// * create via CodecFactory - see CodecFactory |
| /// * create via LicenseSession - see LicenseSession |
| /// 2. Get input constraints |
| /// * OnInputConstraints() - sent unsolicited by stream processor shortly after |
| /// stream processor creation. |
| /// 3. Provide input buffers |
| /// * SetInputBufferPartialSettings() (current way) |
| /// * SetInputBufferSettings() / AddInputBuffer() (old deprecated way) |
| /// 4. Deliver input data |
| /// * QueueInputPacket() + OnFreeInputPacket(), for as long as it takes, |
| /// possibly working through all input packets repeatedly before... |
| /// 5. Get output constraints and format |
| /// * OnOutputConstraints() |
| /// * This is not sent until after at least one QueueInput* message is sent by |
| /// the client, even if the underlying processor behind the StreamProcessor |
| /// doesn't fundamentally need any input data to determine its output |
| /// constraints. This server behavior prevents clients taking an incorrect |
| /// dependency on the output constraints showing up before input is |
| /// delivered. |
| /// * A client must tolerate this arriving as late as after substantial input |
| /// data has been delivered, including lots of input packet recycling via |
| /// OnFreeInputPacket(). |
| /// * This message can arrive more than once before the first output data. |
| /// 6. Provide output buffers |
| /// * SetOutputBufferPartialSettings() / CompleteOutputBufferPartialSettings() |
| /// (current way) |
| /// * SetOutputBufferSettings() / AddOutputBuffer() |
| /// (old deprecated way) |
| /// 7. Data flows, with optional EndOfStream |
| /// * OnOutputPacket() / RecycleOutputPacket() / QueueInputPacket() / |
| /// OnFreeInputPacket() / QueueInputEndOfStream() / OnOutputEndOfStream() |
| /// |
| /// Semi-trusted StreamProcessor server - SW decoders run in an isolate (with |
| /// very few capabilities) just in case the decoding SW has a vulnerability |
| /// which could be used to take over the StreamProcessor server. Clients of the |
| /// stream processor interface using decoders and processing streams of separate |
| /// security contexts, to a greater extent than some other interfaces, need to |
| /// protect themselves against invalid server behavior, such as double-free of a |
| /// packet_index and any other invalid server behavior. Having fed in |
| /// compressed data of one security context, don't place too much trust in a |
| /// single StreamProcessor instance to not mix data among any buffers that |
| /// StreamProcessor server has ever been told about. Instead, create separate |
| /// StreamProcessor instances for use by security-separate client-side contexts. |
| /// While the picture for HW-based decoders looks somewhat different and is out |
| /// of scope of this paragraph, the client should always use separate |
| /// StreamProcessor instances for security-separate client-side contexts. |
| /// |
| /// Descriptions of actions taken by methods of this protocol and the states of |
| /// things are given as if the methods are synchronously executed by the stream |
| /// processor server, but in reality, as is typical of FIDL interfaces, the |
| /// message processing is async. The states described are to be read as the |
| /// state from the client's point of view unless otherwise stated. Events |
| /// coming back from the server are of course delivered async, and a client that |
| /// processes more than one stream per StreamProcessor instance needs to care |
| /// whether a given event is from the current stream vs. some older |
| /// soon-to-be-gone stream. |
| /// |
| /// The Sync() method's main purpose is to enable the client to robustly prevent |
| /// having both old and new buffers allocated in the system at the same time, |
| /// since media buffers can be significantly large, depending. The Sync() method |
| /// achieves this by only delivering it's response when all previous calls to |
| /// the StreamProcessor protocol have actually taken effect in the |
| /// StreamControl ordering domain. Sync() can also be used to wait for the |
| /// stream processor server to catch up if there's a possibility that a client |
| /// might otherwise get too far ahead of the StreamProcessor server, by for |
| /// example requesting creation of a large number of streams in a row. It can |
| /// also be used during debugging to ensure that a stream processor server |
| /// hasn't gotten stuck. Calling Sync() is entirely optional and never required |
| /// for correctness - only potentially required to de-overlap resource usage. |
| /// |
| /// It's possible to re-use a StreamProcessor instance for another stream, and |
| /// doing so can sometimes skip over re-allocation of buffers. This can be a |
| /// useful thing to do for cases like seeking to a new location - at the |
| /// StreamProcessor interface that can look like switching to a new stream. |
| protocol StreamProcessor { |
| /// Permit the server to use OnStreamFailed() instead of the server just |
| /// closing the whole StreamProcessor channel on stream failure. |
| /// |
| /// If the server hasn't seen this message by the time a stream fails, the |
| /// server will close the StreamProcessor channel instead of sending |
| /// OnStreamFailed(). |
| EnableOnStreamFailed(); |
| |
| /// The stream has failed, but the StreamProcessor instance is still usable |
| /// for a new stream. |
| /// |
| /// This message is only ever sent by the server if the client previously |
| /// sent EnableOnStreamFailed(). If the client didn't send |
| /// EnableOnStreamFailed() then the server closes the StreamProcessor |
| /// channel instead. |
| /// |
| /// StreamProcessor server implementations are encouraged to handle stream |
| /// errors (and ideally to also report them via error_ bools of |
| /// OnOutputPacket() and OnOutputEndOfStream()) without failing the whole |
| /// stream, but if a stream processor server is unable to do that, but still |
| /// can cleanly contain the failure to the stream, the stream processor |
| /// server can (assuming EnableOnStreamFailed() was called) use |
| /// OnStreamFailed() to indicate the stream failure to the client without |
| /// closing the StreamProcessor channel. |
| /// |
| /// An ideal StreamProcessor server handles problems with input data without |
| /// sending this message, but sending this message is preferred vs. closing |
| /// the server end of the StreamProcessor channel if the StreamProcessor |
| /// server can 100% reliably contain the stream failure to the stream, |
| /// without any adverse impact to any later stream. |
| /// |
| /// No further messages will arrive from the server regarding the failed |
| /// stream. This includes any OnOutputEndOfStream() that the client would |
| /// have otherwise expected. |
| -> OnStreamFailed(uint64 stream_lifetime_ordinal, StreamError error); |
| |
| /// The server sends this shortly after StreamProcessor creation to indicate |
| /// input buffer constraints. The "min" and "max" input constraints don't |
| /// change for the life of the StreamProcessor. |
| /// |
| /// The "max" values for buffer size and count are large enough to support |
| /// the most demanding format the server supports on input. The |
| /// "recommended" values should be workable for use with the input |
| /// FormatDetails conveyed during StreamProcessor creation. The |
| /// "recommended" values are not necessarily suitable if the client uses |
| /// QueueInputFormatDetails() to change the input format. In that case it's |
| /// up to the client to determine suitable values, either by creating a new |
| /// StreamProcessor instance instead, or knowing suitable values outside the |
| /// scope of this protocol. |
| /// |
| /// See comments on StreamBufferConstraints. |
| /// |
| /// This message is guaranteed to be sent unsolicited to the StreamProcessor |
| /// client during or shortly after StreamProcessor creation. Clients should |
| /// not depend on this being the very first message to arrive at the client. |
| /// |
| /// The "min" and "max" input constraints are guaranteed not to change for a |
| /// given StreamProcessor instance. The "recommended" values may |
| /// effectively change when the server processes QueueInputFormatDetails(). |
| /// There is not any way in the protocol short of creating a new |
| /// StreamProcessor instance for the client to get those new "recommended" |
| /// values. |
| // |
| // TODO(dustingreen): Maybe provide a way for the client to get updated |
| // "recommended" values for input, maybe only on request rather than via |
| // this event, to keep things simpler for simpler clients. Maybe separate |
| // the recommendations from the constraints. |
| -> OnInputConstraints(StreamBufferConstraints input_constraints); |
| |
| /// Clients should use SetInputBufferPartialSettings() instead. |
| /// |
| /// Configuring input buffers (the old way) consists of calling |
| /// SetInputBufferSettings() followed by a number of calls to |
| /// AddInputBuffer() equal to the number of buffers set via |
| /// SetInputBufferSettings(). In buffer-per-packet mode, this is the same as |
| /// the number of packets. In single-buffer mode, this is 1. |
| /// |
| /// After OnInputConstraints(), the client uses these two methods to set up |
| /// input buffers and packets. |
| /// |
| /// Configuring input buffers is required before QueueInputPacket(). |
| /// |
| /// The client can also re-set-up input buffers any time there is no current |
| /// stream. The client need not wait until all previously-set-up input |
| /// buffers are with the client via OnFreeInputPacket(). The old |
| /// buffer_lifetime_ordinal just ends. |
| /// |
| /// The recommended way to de-overlap resource usage (when/if the client |
| /// wants to) is to send CloseCurrentStream() with release_input_buffers |
| /// true then send Sync() and wait for its response before allocating any |
| /// new buffers. How to cause other parts of the system to release their |
| /// references on low-level buffers is outside the scope of this interface. |
| /// |
| /// This call ends any previous buffer_lifetime_ordinal, and starts a new |
| /// one. |
| [Deprecated = "Use SetInputBufferPartialSettings instead."] |
| SetInputBufferSettings(StreamBufferSettings input_settings); |
| |
| /// The client is required to add all the input buffers before sending any |
| /// message that starts a new stream else the stream processor will close |
| /// the StreamProcessor channel. |
| /// |
| /// When the last buffer is added with this message, all the input packets |
| /// effectively jump from non-existent to free with the client. The |
| /// StreamProcessor will not generate an OnFreeInputPacket() for each new |
| /// input packet. The client can immediately start sending |
| /// QueueInputPacket() after sending the last AddInputBuffer(). |
| [Deprecated = "Use SetInputBufferPartialSettings instead."] |
| AddInputBuffer(StreamBuffer buffer); |
| |
| /// This is the replacement for SetInputBufferSettings(). |
| /// |
| /// When the client is using sysmem to allocate buffers, this message is |
| /// used instead of SetInputBufferSettings()+AddInputBuffer(). Instead, a |
| /// single SetInputBufferPartialSettings() provides the StreamProcessor with |
| /// the client-specified input settings and a BufferCollectionToken which |
| /// the StreamProcessor will use to convey constraints to sysmem. Both the |
| /// client and the StreamProcessor will be informed of the allocated buffers |
| /// directly by sysmem via their BufferCollection channel (not via the |
| /// StreamProcessor channel). |
| /// |
| /// The client must not QueueInput...() until after sysmem informs the client |
| /// that buffer allocation has completed and was successful. |
| /// |
| /// The server should be prepared to see QueueInput...() before the server |
| /// has necessarily heard from sysmem that the buffers are allocated - the |
| /// server must tolerate either ordering, as the QueueInput...() and |
| /// notification of sysmem allocation completion arrive on different |
| /// channels, so the client having heard that allocation is complete doesn't |
| /// mean the server knows that allocation is complete yet. However, the |
| /// server can expect that allocation is in fact complete and can expect to |
| /// get the allocation information from sysmem immediately upon requesting |
| /// the information from sysmem. |
| SetInputBufferPartialSettings(StreamBufferPartialSettings input_settings); |
| |
| /// This event informs the client of new output constraints. |
| /// |
| /// This message is ordered with respect to other output (such as output |
| /// packets, output format, output end-of-stream). |
| /// |
| /// Before the first OnOutputPacket() of a stream, the server guarantees that |
| /// at least one OnOutputConstraints() and exactly one OnOutputFormat() will |
| /// be sent. The server may not set buffer_constraints_action_required true |
| /// in OnOutputConstraints() if the buffer config is already suitable for the |
| /// stream (buffer_constraints_action_required false means the buffer config |
| /// is already fine). The client must tolerate multiple |
| /// OnOutputConstraints() (and 1 OnOutputFormat() message) before the first |
| /// output packet. As long as the client hasn't moved to a new stream, the |
| /// server won't send another OnOutputConstraints() until after the client |
| /// has configured output buffers. |
| /// |
| /// This message can be sent mid-stream by a server. If |
| /// buffer_constraints_action_required false, the message is safe to |
| /// ignore, but a client may choose to stash the new constraints for |
| /// later use the next time the client wants to unilaterally re-configure |
| /// buffers (when allowed). If later the server needs the output config to |
| /// change, the server may send a new OnOutputConstraints() with |
| /// buffer_constraints_action_required true. |
| /// |
| /// On buffer_constraints_action_required true, a client that does not wish |
| /// to fully handle mid-stream output buffer config changes should either |
| /// give up completely on the processing, or at least re-config the output |
| /// as specified before starting a new stream (and possibly re-delivering |
| /// input data, if the client wants). This avoids useless retry with a new |
| /// stream starting from just before the output buffer config change which |
| /// would hit the same mid-stream output config change again. |
| /// |
| /// Similarly, some servers may only partly support mid-stream format |
| /// changes, or only support a mid-stream format change if the buffers are |
| /// already large enough to handle both before and after the format change. |
| /// Such servers should still indicate buffer_constraints_action_required |
| /// true, but then send OnStreamFailed() after the client has re-configured |
| /// output buffers (seamlessly dealing with the mid-stream output config |
| /// change is even better of course, but is not always feasible depending on |
| /// format). When the client retries with a new stream starting from a |
| /// nearby location in the client's logical overall media timeline, the |
| /// output buffers will already be suitable for the larger size output, so |
| /// the new stream will not need any mid-stream output buffer re-config, |
| /// only a mid-stream OnOutputFormat(). This strategy avoids the problem |
| /// that would otherwise occur if a client were to retry with a new stream |
| /// starting just before the mid-stream output buffer config change (the |
| /// retry wouldn't be effective since the same need for an output buffer |
| /// config change would be hit again). Servers are discouraged from sending |
| /// OnStreamFailed() solely due to a mid-stream need for different output |
| /// buffer config without first sending OnOutputConstraints() with |
| /// buffer_constraints_action_required true and waiting for the client to |
| /// re-configure output buffers (to avoid the useless client retry with a |
| /// new stream from a logical location before the config change). |
| /// |
| /// When buffer_constraints_action_required true, the server will not send |
| /// any OnOutputPacket() for this stream until after the client has |
| /// configured/re-configured output buffers. |
| /// |
| /// A client that gives up on processing a stream on any mid-stream |
| /// OnOutputConstraints() or mid-stream OnOutputFormat() should completely |
| /// ignore any OnOutputConstraints() with buffer_constraints_action_required |
| /// false. Otherwise the client may needlessly fail processing, or server |
| /// implementations might not be able to use |
| /// buffer_constraints_action_required false for fear of simpler clients |
| /// just disconnecting. |
| /// |
| /// All clients, even those which don't want to support any mid-stream |
| /// output buffer re-config or mid-stream OnOutputFormat() are required to |
| /// deal with 1..multiple OnOutputConstraints() messages before the first |
| /// output packet, and 1 OnOutputFormat() messages before the first output |
| /// packet. |
| /// |
| /// This message is ordered with respect to output packets, and with respect |
| /// to OnOutputFormat(). |
| -> OnOutputConstraints(StreamOutputConstraints output_config); |
| |
| /// This message is sent by the server before the first output packet of any |
| /// stream, and potentially mid-stream between output packets of the stream, |
| /// ordered with respect to output packets, and ordered with respect to |
| /// OnOutputConstraints(). |
| /// |
| /// The server guarantees that the first packet of every stream will be |
| /// preceeded by an OnOutputFormat(). |
| /// |
| /// The server guarantees that there will be an OnOutputFormat() between an |
| /// OnOutputConstraints() with buffer_constraints_action_required true and an |
| /// OnOutputPacket(). In other words, the client is essentially allowed to |
| /// forget what the output format is on any OnOutputConstraints() with |
| /// buffer_constraints_action_required true, because the server promises a |
| /// subsequent OnOutputFormat() before any OnOutputPacket(). |
| /// |
| /// If the server sets buffer_constraints_action_required true in |
| /// OnOutputConstraints(), the server won't send OnOutputFormat() (and |
| /// therefore also won't send OnOutputPacket()) until the client has |
| /// re-configured output buffers. |
| /// |
| /// The server is allowed to send an OnOutputFormat() mid-stream between two |
| /// output packets. |
| /// |
| /// A server won't send two adjacent OnOutputFormat() messages without any |
| /// output packet in between. However an OnOutputFormat() message doesn't |
| /// guarantee a subsequent packet, because for example the server could send |
| /// OnOutputEndOfStream() or OnStreamFailed() instead. |
| /// |
| /// A client that does not wish to seamlessly handle mid-stream output format |
| /// changes should either ensure that no stream processed by the client |
| /// ever has any mid-stream format change, or the client should ensure that |
| /// any retry of processing starts the new attempt at a point logically at or |
| /// after the point where the old format has ended and the new format starts, |
| /// else the client could just hit the same mid-stream format change again. |
| /// |
| /// An example of this message being sent mid-stream is mid-stream change |
| /// of dimensions of video frames output from a video decoder. |
| /// |
| /// Not all servers will support seamless handling of format change. Those |
| /// that do support seamless handling of format change may require that the |
| /// format change not also require output buffer re-config, in order for the |
| /// handling to be seamless. See the comment block for OnOutputConstraints() |
| /// for more discussion of how servers and clients should behave - in |
| /// particular when they don't seamlessly handle output constraint change |
| /// and/or output format change. |
| /// |
| /// If this message isn't being sent by the server when expected at the |
| /// start of a stream, the most common reason is that a OnOutputConstraints() |
| /// with buffer_constraints_action_required true hasn't been processed by the |
| /// client (by configuring output buffers using |
| /// SetOutputBufferPartialSettings() etc). |
| -> OnOutputFormat(StreamOutputFormat output_format); |
| |
| /// These are not permitted until after the first OnOutputConstraints(). |
| /// |
| /// Roughly speaking, these messages are sent in response to |
| /// OnOutputConstraints() with buffer_constraints_action_required true. |
| /// |
| /// Configuring output buffers consists of calling SetOutputBufferSettings() |
| /// followed by a number of calls to AddOutputBuffer() equal to the number |
| /// of buffers set via SetOutputBufferSettings(). In buffer-per-packet |
| /// mode, this is the same as the number of packets. In single-buffer mode, |
| /// this is 1. |
| /// |
| /// Configuring output buffers is _required_ after OnOutputConstraints() is |
| /// received by the client with buffer_constraints_action_required true and |
| /// stream_lifetime_ordinal equal to the client's current |
| /// stream_lifetime_ordinal (even if there is an active stream), and is |
| /// _permitted_ any time there is no current stream. |
| /// |
| /// Closing the current stream occurs on the StreamControl ordering domain, |
| /// so after a CloseCurrentStream() or FlushEndOfStreamAndCloseStream(), a |
| /// subsequent Sync() completion must be received by the client before the |
| /// client knows that there's no longer a current stream. |
| [Deprecated = "Use SetOutputBufferPartialSettings instead."] |
| SetOutputBufferSettings(StreamBufferSettings output_settings); |
| [Deprecated = "Use SetOutputBufferPartialSettings instead."] |
| AddOutputBuffer(StreamBuffer buffer); |
| |
| /// This is the replacement for SetOutputBufferSettings(). |
| /// |
| /// When the client is using sysmem to allocate buffers, this message is |
| /// used instead of SetOutputBufferSettings()+AddOutputBuffer(). Instead, a |
| /// single SetOutputBufferPartialSettings() provides the StreamProcessor |
| /// with the client-specified output settings and a BufferCollectionToken |
| /// which the StreamProcessor will use to convey constraints to sysmem. |
| /// Both the client and the StreamProcessor will be informed of the |
| /// allocated buffers directly by sysmem via their BufferCollection channel |
| /// (not via the StreamProcessor channel). |
| /// |
| /// Configuring output buffers is _required_ after OnOutputConstraints() is |
| /// received by the client with buffer_constraints_action_required true and |
| /// stream_lifetime_ordinal equal to the client's current |
| /// stream_lifetime_ordinal (even if there is an active stream), and is |
| /// _permitted_ any time there is no current stream. |
| /// |
| /// Closing the current stream occurs on the StreamControl ordering domain, |
| /// so after a CloseCurrentStream() or FlushEndOfStreamAndCloseStream(), a |
| /// subsequent Sync() completion must be received by the client before the |
| /// client knows that there's no longer a current stream. |
| /// |
| /// See also CompleteOutputBufferPartialSettings(). |
| SetOutputBufferPartialSettings(StreamBufferPartialSettings output_settings); |
| |
| /// After SetOutputBufferPartialSettings(), the server won't send |
| /// OnOutputConstraints(), OnOutputFormat(), OnOutputPacket(), or |
| /// OnOutputEndOfStream() until after the client sends |
| /// CompleteOutputBufferPartialSettings(). |
| /// |
| /// Some clients may be able to send |
| /// CompleteOutputBufferPartialSettings() immediately after |
| /// SetOutputBufferPartialSettings() - in that case the client needs to be |
| /// prepared to receive output without knowing the buffer count or packet |
| /// count yet - such clients may internally delay processing the received |
| /// output until the client has heard from sysmem (which is when the client |
| /// will learn the buffer count and packet count). |
| /// |
| /// Other clients may first wait for sysmem to allocate, prepare to receive |
| /// output, and then send CompleteOutputBufferPartialSettings(). |
| CompleteOutputBufferPartialSettings(uint64 buffer_lifetime_ordinal); |
| |
| /// This message is optional. |
| /// |
| /// This message is only valid after QueueInputEndOfStream() for this stream. |
| /// The stream_lifetime_ordinal input parameter must match the |
| /// stream_lifetime_ordinal of the QueueInputEndOfStream(), else the server |
| /// will close the channel. |
| /// |
| /// A client can use this message to flush through (not discard) the last |
| /// input data of a stream so that the stream processor server generates |
| /// corresponding output data for all the input data before the server moves |
| /// on to the next stream, without forcing the client to wait for |
| /// OnOutputEndOfStream() before queueing data of another stream. |
| /// |
| /// The difference between QueueInputEndOfStream() and |
| /// FlushEndOfStreamAndCloseStream(): QueueInputEndOfStream() is a promise |
| /// from the client that there will not be any more input data for the |
| /// stream (and this info is needed by some stream processors for the stream |
| /// processor to ever emit the very last output data). The |
| /// QueueInputEndOfStream() having been sent doesn't prevent the client from |
| /// later completely discarding the rest of the current stream by closing |
| /// the current stream (with or without a stream switch). In contrast, |
| /// FlushEndOfStreamAndCloseStream() is a request from the client that all |
| /// the previously-queued input data be processed including the logical |
| /// "EndOfStream" showing up as OnOutputEndOfStream() (in success case) |
| /// before moving on to any newer stream - this essentially changes the |
| /// close-stream handling from discard to flush-through for this stream |
| /// only. |
| /// |
| /// A client using this message can start providing input data for a new |
| /// stream without that causing discard of old stream data. That's the |
| /// purpose of this message - to allow a client to flush through (not |
| /// discard) the old stream's last data (instead of the default when closing |
| /// or switching streams which is discard). |
| /// |
| /// Because the old stream is not done processing yet and the old stream's |
| /// data is not being discarded, the client must be prepared to continue to |
| /// process OnOutputConstraints() messages until the stream_lifetime_ordinal |
| /// is done. The client will know the stream_lifetime_ordinal is done when |
| /// OnOutputEndOfStream(), OnStreamFailed(), or the StreamProcessor channel |
| /// closes. |
| FlushEndOfStreamAndCloseStream(uint64 stream_lifetime_ordinal); |
| |
| /// This "closes" the current stream, leaving no current stream. In |
| /// addition, this message can optionally release input buffers or output |
| /// buffers. |
| /// |
| /// If there has never been any active stream, the stream_lifetime_ordinal |
| /// must be zero or the server will close the channel. If there has been an |
| /// active stream, the stream_lifetime_ordinal must be the most recent |
| /// active stream whether that stream is still active or not. Else the |
| /// server will close the channel. |
| /// |
| /// Multiple of this message without any new active stream in between is not |
| /// to be considered an error, which allows a client to use this message to |
| /// close the current stream to stop wasting processing power on a stream the |
| /// user no longer cares about, then later decide that buffers should be |
| /// released and send this message again with release_input_buffers and/or |
| /// release_output_buffers true to get the buffers released, if the client is |
| /// interested in trying to avoid overlap in resource usage between old |
| /// buffers and new buffers (not all clients are). |
| /// |
| /// See also Sync(). |
| CloseCurrentStream( |
| uint64 stream_lifetime_ordinal, |
| bool release_input_buffers, |
| bool release_output_buffers); |
| |
| /// On completion, all previous StreamProcessor calls have done what they're |
| /// going to do server-side, _except_ for processing of data queued using |
| /// QueueInputPacket(). |
| /// |
| /// The main purpose of this call is to enable the client to wait until |
| /// CloseCurrentStream() with release_input_buffers and/or |
| /// release_output_buffers set to true to take effect, before the client |
| /// allocates new buffers and re-sets-up input and/or output buffers. This |
| /// de-overlapping of resource usage can be worthwhile for media buffers |
| /// which can consume resource types whose overall pools aren't necessarily |
| /// vast in comparison to resources consumed. Especially if a client is |
| /// reconfiguring buffers multiple times. |
| /// |
| /// Note that Sync() prior to allocating new media buffers is not alone |
| /// sufficient to achieve non-overlap of media buffer resource usage system |
| /// wide, but it can be a useful part of achieving that. |
| /// |
| /// The Sync() transits the Output ordering domain and the StreamControl |
| /// ordering domain, but not the InputData ordering domain. |
| /// |
| /// This request can be used to avoid hitting kMaxInFlightStreams which is |
| /// presently 10. A client that stays <= 8 in-flight streams will |
| /// comfortably stay under the limit of 10. While the protocol permits |
| /// repeated SetInputBufferSettings() and the like, a client that spams the |
| /// channel can expect that the channel will just close if the server or the |
| /// channel itself gets too far behind. |
| Sync() -> (); |
| |
| /// This is how the stream processor emits an output packet to the stream |
| /// processor client. |
| /// |
| /// Order is significant. |
| /// |
| /// The client should eventually call RecycleOutputPacket() (possibly after |
| /// switching streams multiple times), unless the buffer_lifetime_ordinal |
| /// has moved on. A stream change doesn't change which packets are busy |
| /// with the client vs. free with the server. |
| /// |
| /// The relevant buffer is always the one specified in the packet's buffer_index field. |
| /// |
| /// For low-level buffer types that support it, a StreamProcessor is free to |
| /// emit an output packet before the low-level buffer actually has any |
| /// usable data in the buffer, with the mechanism for signalling the |
| /// presence of data separate from the OnOutputPacket() message. For such |
| /// low-level buffer types, downstream consumers of data from the emitted |
| /// packet must participate in the low-level buffer signalling mechanism to |
| /// know when it's safe to consume the data. This is most likely to be |
| /// relevant when using a video decoder and gralloc-style buffers. |
| /// |
| /// The error_ bool(s) allow (but do not require) a StreamProcessor server |
| /// to report errors that happen during an AU or between AUs. |
| /// |
| /// The scope of error_detected_before starts at the end of the last |
| /// delivered output packet on this stream, or the start of stream if there |
| /// were no previous output packets on this stream. The scope ends at the |
| /// start of the output_packet. |
| /// |
| /// The error_detected_before bool is separate so that discontinuities can be |
| /// indicated separately from whether the current packet is damaged. |
| /// |
| /// The scope of error_detected_during is from the start to the end of this |
| /// output_packet. |
| -> OnOutputPacket( |
| Packet output_packet, |
| bool error_detected_before, |
| bool error_detected_during); |
| |
| /// After the client is done with an output packet, the client needs to tell |
| /// the stream processor that the output packet can be re-used for more |
| /// output, via this method. |
| /// |
| /// It's not permitted to recycle an output packet that's already free with |
| /// the stream processor server. It's permitted but discouraged for a |
| /// client to recycle an output packet that has been deallocated by an |
| /// explicit or implicit output buffer de-configuration(). See |
| /// buffer_lifetime_ordinal for more on that. A server must ignore any such |
| /// stale RecycleOutputPacket() calls. |
| RecycleOutputPacket(PacketHeader available_output_packet); |
| |
| /// After QueueInputEndOfStream() is sent by the StreamProcessor client, |
| /// within a reasonable duration the corresponding OnOutputEndOfStream() |
| /// will be sent by the StreamProcessor server. Similar to |
| /// QueueInputEndOfStream(), OnOutputEndOfStream() is sent a maximum of once |
| /// per stream. |
| /// |
| /// No more stream data for this stream will be sent after this message. All |
| /// input data for this stream was processed. |
| /// |
| /// While a StreamProcessor client is not required to |
| /// QueueInputEndOfStream() (unless the client wants to use |
| /// FlushEndOfStreamAndCloseStream()), if a StreamProcessor server receives |
| /// QueueInputEndOfStream(), and the client hasn't closed the stream, the |
| /// StreamProcessor server must generate a corresponding |
| /// OnOutputEndOfStream() if nothing went wrong, or must send |
| /// OnStreamFailed(), or must close the server end of the StreamProcessor |
| /// channel. An ideal StreamProcessor server would handle and report stream |
| /// errors via the error_ flags and complete stream processing without |
| /// sending OnStreamFailed(), but in any case, the above-listed options are |
| /// the only ways that an OnOutputEndOfStream() won't happen after |
| /// QueueInputEndOfStream(). |
| /// |
| /// There will be no more OnOutputPacket() or OnOutputConstraints() messages |
| /// for this stream_lifetime_ordinal after this message - if a server doesn't |
| /// follow this rule, a client should close the StreamProcessor channel. |
| /// |
| /// The error_detected_before bool has the same semantics as the |
| /// error_detected_before bool in OnOutputPacket(). |
| -> OnOutputEndOfStream( |
| uint64 stream_lifetime_ordinal, |
| bool error_detected_before); |
| |
| // TODO(dustingreen): Rename from QueueInputFormatDetails() to |
| // QueueInputFormat(). |
| // |
| /// If the input format details are still the same as specified during |
| /// StreamProcessor creation, this message is unnecessary and does not need |
| /// to be sent. |
| /// |
| /// If the stream doesn't exist yet, this message creates the stream. |
| /// |
| /// The server won't send OnOutputConstraints() until after the client has |
| /// sent at least one QueueInput* message. |
| /// |
| /// All servers must permit QueueInputFormatDetails() at the start of a |
| /// stream without failing, as long as the new format is supported by the |
| /// StreamProcessor instance. Technically this allows for a server to only |
| /// support the exact input format set during StreamProcessor creation, and |
| /// that is by design. A client that tries to switch formats and gets a |
| /// StreamProcessor channel failure should try again one more time with a |
| /// fresh StreamProcessor instance created with CodecFactory using the new |
| /// input format during creation, before giving up. |
| /// |
| /// These format details override the format details specified during stream |
| /// processor creation for this stream only. The next stream will default |
| /// back to the format details set during stream processor creation. |
| /// |
| /// This message is permitted at the start of the first stream (just like at |
| /// the start of any stream). The format specified need not match what was |
| /// specified during stream processor creation, but if it doesn't match, the |
| /// StreamProcessor channel might close as described above. |
| QueueInputFormatDetails( |
| uint64 stream_lifetime_ordinal, FormatDetails format_details); |
| |
| /// This message queues input data to the stream processor for processing. |
| /// |
| /// If the stream doesn't exist yet, this message creates the new stream. |
| /// |
| /// The server won't send OnOutputConstraints() until after the client has |
| /// sent at least one QueueInput* message. |
| /// |
| /// The client must continue to deliver input data via this message even if |
| /// the stream processor has not yet generated the first OnOutputConstraints(), |
| /// and even if the StreamProcessor is generating OnFreeInputPacket() for |
| /// previously-queued input packets. The input data must continue as long |
| /// as there are free packets to be assured that the server will ever |
| /// generate the first OnOutputConstraints(). |
| QueueInputPacket(Packet packet); |
| |
| /// The server sends this message when the stream processor is done |
| /// consuming this packet and the packet can be re-filled by the client. |
| /// |
| /// This is not sent for all packets when a new buffer_lifetime_ordinal |
| /// starts as in that case all the packets are initially free with the |
| /// client. |
| /// |
| /// After receiving the available input buffer via this event, the stream |
| /// processor client can call later call QueueInputBuffer with appropriate |
| /// offset and length set. |
| // |
| // TODO(dustingreen): At the moment, there is no guarantee re. the order of |
| // these messages with respect to the order of QueueInputPacket(), but at |
| // least for decoders, it might be worthwhile to require that servers |
| // preserve the order vs. QueueInputPacket(), to make it easier to feed |
| // input from a ring buffer or similar. For audio encoders it might still |
| // make sense. For video encoders probably not. |
| -> OnFreeInputPacket(PacketHeader free_input_packet); |
| |
| /// Inform the server that all QueueInputPacket() messages for this stream |
| /// have been sent. |
| /// |
| /// If the stream isn't closed first (by the client, or by OnStreamFailed(), |
| /// or StreamProcessor channel closing), there will later be a corresponding |
| /// OnOutputEndOfStream(). |
| /// |
| /// The corresponding OnOutputEndOfStream() message will be generated only if |
| /// the server finishes processing the stream before the server sees the |
| /// client close the stream (such as by starting a new stream). A way to |
| /// force the server to finish the stream before closing is to use |
| /// FlushEndOfStreamAndCloseStream() after QueueInputEndOfStream() before any |
| /// new stream. Another way to force the server to finish the stream before |
| /// closing is to wait for the OnOutputEndOfStream() before taking any action |
| /// that closes the stream. |
| /// |
| /// In addition to serving as an "EndOfStream" marker to make it obvious |
| /// client-side when all input data has been processed, if a client never |
| /// sends QueueInputEndOfStream(), no amount of waiting will necessarily |
| /// result in all input data getting processed through to the output. Some |
| /// stream processors have some internally-delayed data which only gets |
| /// pushed through by additional input data _or_ by this EndOfStream marker. |
| /// In that sense, this message can be viewed as a flush-through at |
| /// InputData domain level, but the flush-through only takes effect if the |
| /// stream processor even gets that far before the stream is just closed at |
| /// StreamControl domain level. This message is not alone sufficient to act |
| /// as an overall flush-through at StreamControl level. For that, send this |
| /// message first and then send FlushEndOfStreamAndCloseStream() (at which |
| /// point it becomes possible to queue input data for a new stream without |
| /// causing discard of this older stream's data), or wait for the |
| /// OnOutputEndOfStream() before closing the current stream. |
| /// |
| /// If a client sends QueueInputPacket(), QueueInputFormatDetails(), |
| /// QueueInputEndOfStream() for this stream after the first |
| /// QueueInputEndOfStream() for this stream, a server should close the |
| /// StreamProcessor channel. |
| QueueInputEndOfStream(uint64 stream_lifetime_ordinal); |
| }; |