blob: cb30447e012b1bb0c99fa32e3024c248be5f0be8 [file] [view]
# Audio Signal Processing
## Overview
The signal processing interface is available to be potentially used by audio composite drivers.
This interface `SignalProcessing` is a FIDL protocol used by the `Composite` protocol to provide
audio signal processing capabilities.
The `SignalProcessing` protocol is defined to control signal processing hardware and their
topologies. We define processing elements (PEs) as a logical unit of audio data processing provided
by an audio driver, and we define topologies as the arrangement of PEs in [pipelines][pipeline] and
controls associated with them.
The `SignalProcessing` protocol allows hardware vendors to implement drivers with stable
application binary interfaces (ABIs), and allow system integrators to configure drivers to perform
differently based on system or product requirements using these interfaces for run-time
configurations.
The `SignalProcessing` protocol composes the `Reader` signal processing protocol. Signal processing
methods that only retrieve information are part of the `Reader` protocol, the rest are part of the
`SignalProcessing` protocol itself. This separation allows clients of this interface to compose the
`Reader` signal processing protocol into their own protocol if they require providing a read only
subset of functionality to their own clients.
The `SignalProcessing` protocol and associated definitions are part of the
[fuchsia.hardware.audio.signalprocessing](/sdk/fidl/fuchsia.hardware.audio.signalprocessing)
FIDL library.
### Topologies
Each driver can have its own topology. Drivers can abstract from applications the topologies
exposed by other drivers as needed for a particular configuration or product. Note that it is
possible although not required to expose topologies to applications, in particular to `audio_core`.
Notes:
* Topologies are not meant to fully describe the audio pipeline state/format/configuration
in and out of every PE. The intent is to describe what can be changed/rearranged by the client
based on its knowledge, configuration (for instance from metadata) and specific business logic.
* Topologies include one or more processing elements of type `RingBuffer`, if (and only if) ring
buffers are supported by the driver.
* Topologies include one or more processing elements of type `DaiInterconnect`, if (and only if)
DAI interconnects are supported by the driver.
*Topologies include one or more processing elements of type `PackeetStream`, if (and only if)
packet streams are supported by the driver.
* Topologies must include at least one of these element types.
### Processing Elements
A PE (defined in the `fuchsia.hardware.audio.signalprocessing` FIDL library as `Element`) is
expected to be hardware-provided functionality managed by a particular driver (but it could be
emulated in software, as any other driver functionality). A pipeline is composed of one or more PEs
and a topology is composed of one or more pipelines.
We refer to the server as the driver that is providing the signal processing protocol.
We refer to the client as the user of the functionality, e.g. an application such as `audio_core`.
## Basic operation
The client is responsible for requesting and then configuring any signal processing capabilities.
Once the server provides its PEs by replying to a client's `GetElements`, the client may
issue `WatchElementState` calls (see [hanging get pattern][hanging-get]) to retrieve
PE state and `SetElementState` to dynamically control the PEs parameters as needed. For
instance, to retrieve the `gain` of a PE of `type` `GAIN`, the client issues
`WatchElementState` calls, one to retrieve the initial state (the driver will reply to the
first `WatchElementState` sent by the client), and subsequent ones to get notified of updates
to the `ElementState` that includes the `gain`. Similarly, to retrieve the state of a PE
of `type` `EQUALIZER`, which is composed of multiple bands in its `bands_state`, a client would
issue a `WatchElementState` that would retrieve the initial state (the driver will reply to the
first `WatchElementState` sent by the client) including for instance `frequency` fields for
each band.
Also after the server provides its PEs by replying to a client's `GetElements`, the client
may request available topologies with the `GetTopologies` method. If more than one topology is
returned by `GetTopologies`, then `SetTopology` can be used to pick the topology to use, and
`WatchTopology` can be used to observe updates to the active topology.
### GetElements
`GetElements` allows to optionally get a list of all PEs. For instance this method may
be called by a client on a driver abstracting a hardware codec. Once the list of PEs is known to
the client, the client may configure the PEs based on the parameters exposed by the PE types.
### SetElementState
`SetElementState` allows a client to control the state of a PE using an id returned by
`GetElements`. PEs of different types may have different state exposed to clients, the
`SetElementState` parameter `state` has a different type depending on the type of PE.
### WatchElementState
`WatchElementState` allows a client to monitor the state of a PE using an id returned by
`GetElements`. PEs of different types may have different state exposed to clients, the
`WatchElementState` parameter `state` has a different type depending on the type of PE.
The `state` of a PE is composed of values that may be changed directly by the client via a call to
`SetElementState`, or indirectly for instance by calling `SetElementState` on a
different PE, or independent of the client for instance due to a plug detect change.
### GetTopologies
`GetTopologies` allows to optionally get a list of topologies. For instance this method may be
called by a client on a driver abstracting a hardware codec. Once the list of topologies is known to
the client, the client may configure the server to use a particular topology.
### SetTopology
`SetTopology` allows a client to control which topology is used by the server. Only one
topology can be selected at any time.
### WatchTopology
`WatchTopology` allows a client to observe the currently active topology id via a hanging get,
and to be notified asynchronously when the topology changes.
## Processing elements types
The PEs returned by `GetElements` support a number of different types of signal processing
defined by the PE types and parameters. PE types define standard signal processing (e.g. `GAIN`,
`DELAY`, `EQUALIZER`, etc), vendor specific signal processing (`VENDOR_SPECIFIC` e.g. a type not
defined in the `SignalProcessing` protocol), and `CONNECTION_POINT`s used to construct
multi-pipeline topologies (allow for routing and mixing definitions, see
[Connection points](#connection-points) below).
Each individual PE may have one or more inputs and one or more output channels. For routing and
mixing, PEs may make the number of output channels different from the number of input channels.
Data in each channel (a.k.a. the signal that is processed) may be altered by the PE. For instance
if there is a single PE of type `AGL` in a pipeline that includes a `DAI_INTERCONNECT`
element with `DaiFormat` `number_of_channels` set to 2, then AGL (Automatic Gain
Limiting) can be started or stopped for these 2 channels by a client calling `SetElementState`
with `state` `started` set to true (or false if the AGL `Element`'s `can_stop` was true).
If optional fields in the different PE types are not included, then the state of the processing
element is not changed with respect to the particular field. For instance, if an
`EqualizerBandState` in a `SetElementState` call does not include an optional `frequency` then the
equalizer's band frequency state is not changed.
## Vendor specific data
`ElementState` `vendor_specific_data` is an optional parameter that can be specified for any
processing element. This allows processing elements to specify an opaque object to be either sent
to the driver as part of a `SetElementState` or received from a driver as part of a
`WatchElementState`.
In addition to opaque data for any type, a processing element of type `VENDOR_SPECIFIC` allows
drivers to specify a type that is not defined in the `SignalProcessing` protocol, for instance
something that is not standard yet or is not meant to be standardized and provided only by a
specific vendor. A processing element of type `VENDOR_SPECIFIC` specifies a `vendor_specific`
variant in `TypeSpecificElement` and `TypeSpecificElementState`. In addition, it may specify
opaque data to be sent to or received from the driver using the `ElementState`
`vendor_specific_data` parameter, just like any other processing element type.
## Topologies {#topologies}
The topologies returned by `GetTopologies` support different arrangements for the PEs returned by
`GetElements`. `GetTopologies` may advertise one or multiple topologies.
### One topology
If one topology is advertised, i.e. `GetTopologies` returns a vector with one element, then all PEs
are part of this explicit single pipeline. Ordering in this case is explicit. For instance, if
`GetElements` returns 2 PEs:
1. `Element`: id = 1, type = `AUTOMATIC_GAIN_LIMITER` (AGL)
1. `Element`: id = 2, type = `EQUALIZER` (EQ)
The one `Topology` element returned by `GetTopologies` will list an `id` and a
`processing_elements_edge_pairs` vector explicitly advertising the order in which signal processing
is performed, in this example:
1. `Topology`: id = 1, `processing_elements_edge_pairs` = vector with one element with
`processing_element_id_from` = 1 and `processing_element_id_to` = 2.
This advertises this one topology with one pipeline:
+-------+ +-------+
Input signal -> | AGL | -> | EQ | -> Output signal
+-------+ +-------+
In this topology the beginning (where the input signal is input into the pipeline) and the end of
the pipeline (where the output signal is output from the pipeline) are implicit.
If only one topology is advertised, then the contents are informational only since the client can't
change the use of one and only topology.
### Multiple topologies {#multiple-topologies}
If multiple topologies are advertised, i.e. `GetTopologies` returns a vector with multiple
elements, then PEs may be used in multiple configurations, i.e. topologies. Each topology
explicitly lists a number of PEs and their ordering, i.e. ordering in this case is explicit.
The arrangement and ordering of PEs define a pipeline.
By listing only the specific arrangements and ordering of PEs supported, servers restrict what
combinations of pipelines are valid.
For instance, if `GetElements` returns 6 PEs:
1. `Element`: id = 1, type = `AUTOMATIC_GAIN_LIMITER` (AGL)
1. `Element`: id = 2, type = `EQUALIZER` (EQ)
1. `Element`: id = 3, type = `SAMPLE_RATE_CONVERSION` (SRC)
1. `Element`: id = 4, type = `GAIN`
1. `Element`: id = 5, type = `DYNAMICS` (DRC1)
1. `Element`: id = 6, type = `DYNAMICS` (DRC2) parameters different from
DRC1 parameters.
The `Topology` elements returned by `GetTopologies` will list an `id` and a
`processing_elements_edge_pairs` for each topology, in this example:
1. `Topology`: id = 1, `processing_elements_edge_pairs` =
* `processing_element_id_from` = 3 and `processing_element_id_to` = 2.
* `processing_element_id_from` = 2 and `processing_element_id_to` = 4.
* `processing_element_id_from` = 4 and `processing_element_id_to` = 5.
* `processing_element_id_from` = 5 and `processing_element_id_to` = 1.
1. `Topology`: id = 2, `processing_elements_edge_pairs` =
* `processing_element_id_from` = 2 and `processing_element_id_to` = 4.
* `processing_element_id_from` = 4 and `processing_element_id_to` = 6.
This advertises two topologies with one pipeline each:
+-------+ +-------+ +-------+ +-------+ +-------+
Input signal -> | SRC | -> | EQ | -> | GAIN | -> | DRC1 | -> | AGL | -> Output signal
+-------+ +-------+ +-------+ +-------+ +-------+
+-------+ +-------+ +-------+
Input signal -> | EQ | -> | GAIN | -> | DRC2 | -> Output signal
+-------+ +-------+ +-------+
## Connection points {#connection-points}
The PEs of type `CONNECTION_POINT` allow for:
1. Mixing multiple channels within a single pipeline.
1. Mixing multiple channels from different pipelines.
1. Repeating channels.
1. Expanding a single pipeline into multiple pipelines ones (scatter).
{% comment %}
// TODO(https://fxbug.dev/42143529): Add extra context for multi-pipeline construction.
{% endcomment %}
<!-- Reference links -->
[pipeline]: https://en.wikipedia.org/wiki/Pipeline_(computing)
[hanging-get]: /docs/development/api/fidl.md#hanging-get