The device effect (DFX) interface enables device makers and system builders to add custom audio processing at a location in the audio pipeline that is unavailable to normal audio clients. This processing can be performed on the final mixed output audio stream right before it is consumed by playback hardware, and/or on an incoming stream immediately after it is produced by input hardware before it is provided to clients of the audio capture API.
The following interfaces, structures and constants are specified in the audio_device_fx.h file.
The Fuchsia audio system calls these device effects using the “C” ABI specified by the following nine interface calls and their related structs and consts. The system dynamically loads the library audio_dfx.so by name and calls the synchronous functions described below, to query and configure the effect, and subsequently to process audio while it is actively streaming.
The DFX library can contain numerous different effects (such as an equalizer, a volume limiter, and a reverb); these are called effect types. For a given effect type, at any time there may be multiple independent copies of the effect that are active; each is called an effect instance. This is not unlike an object hierarchy; effect types equate to classes, and effect instances are analogous to instantiated objects of that class.
Additionally, an effect can be configured by settings that govern its behavior (such as EQ parameters or reverb decay time); these parameters, and their values, are called effect controls and control values respectively. Mirroring the object-oriented design mentioned above, controls are defined across a given effect type, but each instance of that effect has an independent set of values for those settings.
This low-level interface does not provide effects with any knowledge of where they are being inserted in the processing chain. This allows the system to be flexibly configured. However, a settings file that accompanies the audio_dfx.so (discussed elsewhere, TBD) is used to instruct the system which effects should be used, in which topology, with which control values.
The settings file, and the FIDL interface to control the enabling, ordering and configuration of specific effects on specific devices, are described in separate README (TBD) for that higher level API.
Below are input parameters, return values and usage notes for the Audio DFX (device effects) APIs. Each of these nine functions must be implemented in the audio_dfx.so shared library.
bool fuchsia_audio_dfx_get_num_effects(
uint32_t* num_effects_out
);
*num_effects_out contains the number of device effects available.*num_effects_out is undefined.This interface returns the number of effect types found in this audio_dfx lib. The value returned from this interface is important for subsequent APIs, where the required effect_id value must be less than the reported number of effect types (per zero-based indexing).
bool fuchsia_audio_dfx_get_info(
uint32_t effect_id,
fuchsia_audio_dfx_description* device_fx_desc
);
*device_fx_desc contains information about the specified effect type.*device_fx_desc are undefined.This interface returns information about this type of effect, including number of controls. The num_controls value returned from this interface is important for subsequent APIs, where the required control_num value must be less than the reported number of controls for this effect type (per zero-based indexing).
bool fuchsia_audio_dfx_get_control_info(
uint32_t effect_id,
uint16_t control_num,
fuchsia_audio_dfx_control_description* device_fx_control_desc
);
*device_fx_control_desc contains information about the specified effect control.*device_fx_control_desc are undefined.This interface returns information related to the identified effect type. Specifically, it returns details about a particular control on that effect.
fx_token_t fuchsia_audio_dfx_create(
uint32_t effect_id,
uint32_t frame_rate,
uint16_t channels_in,
uint16_t channels_out
);
fx_token_t is a token representing the just-created active instance of this effect type.The value that this call returns will be used as dfx_token, in subsequent API calls that reference a particular effect instance.
When the values of channels_in and channels_out are equal, the system will require the created device effect to process audio in-place.
bool fuchsia_audio_dfx_delete(
fx_token_t dfx_token
);
bool fuchsia_audio_dfx_get_parameters(
fx_token_t dfx_token,
fuchsia_audio_dfx_parameters* device_fx_params
);
fx_token_t representing an active effect instance.*device_fx_params contains information about the specified effect instance.*device_fx_params are undefined.This interface returns the operational parameters for this instance of the device effect. These parameters are invariant for the lifetime of this effect, based on the initial values provided when the client created the effect.
bool fuchsia_audio_dfx_get_control_value(
fx_token_t dfx_token,
uint16_t control_num,
float* value_out
);
fx_token_t representing an active effect instance.uint16_t representing a control of this effect instance.float allocated by the system, into which the DFX implementation should copy the current value of this control.*value_out contains the current value of this control.*value_out is undefined.This interface returns the value for the specified control on this active effect.
bool fuchsia_audio_dfx_set_control_value(
fx_token_t dfx_token,
uint16_t control_num,
float value
);
fx_token_t representing an active effect instance.uint16_t representing a control of this effect instance.float containing the value to which this control should be set.This interface sets the value of the specified control, on this active effect.
bool fuchsia_audio_dfx_reset(
fx_token_t dfx_token
);
bool fuchsia_audio_dfx_process_inplace(
fx_token_t dfx_token,
uint32_t num_frames,
float* audio_buff_in_out
);
fx_token_t representing an active effect instance.uint32_t containing the number of frames that the specified effect instance must process. This value cannot exceed the previously-set frame rate for this effect instance.float.num_frames of audio samples, in-place within the buffer specified by audio_buff_in_out.audio_buff_in_out are undefined.This interface synchronously processes the buffer of num_frames audio data in-place. The total number of samples processed is equal to num_frames multiplied by the value of channels_out, from the earlier fuchsia_audio_dfx_create call.
fuchsia_audio_dfx_process_inplace is used in all cases except where rechannelization occurs. It is preferred over fuchsia_audio_dfx_process because of its reduced memory consumption and related system impact.
The upper limit on the value of num_frames guarantees that no fuchsia_audio_dfx_process_inplace call will ever entail more than one second of audio.
bool fuchsia_audio_dfx_process(
fx_token_t dfx_token,
uint32_t num_frames,
const float* audio_buff_in,
float* audio_buff_out
);
fx_token_t representing an active effect instance.uint32_t containing the number of frames that the specified effect instance must process. This value cannot exceed the previously-set frame rate for this effect instance.float, from which num_frames audio frames must be ingested by the effect instance.num_frames audio frames of type float must be produced, based on the data ingested from audio_buff_in.num_frames of audio samples from the buffer specified by audio_buff_in, to the buffer specified by audio_buff_out.audio_buff_out are undefined.This interface synchronously processes ‘num_frames’ of audio data in type float, from audio_buff_in to audio_buff_out. The total number of input samples ingested is equal to num_frames multiplied by the value of channels_in from the earlier fuchsia_audio_dfx_create call. The total number of output samples generated and populated into audio_buff_out is equal to num_frames multiplied by the value of channels_out from that same fuchsia_audio_dfx_create call.
This interface is necessary for cases where rechannelization occurs; it is not used otherwise. Where possible, the system instead uses fuchsia_audio_dfx_process_inplace, because of its reduced memory consumption and related system impact.
The upper limit on the value of num_frames guarantees that no fuchsia_audio_dfx_process call will ever entail more than one second of audio.
bool fuchsia_audio_dfx_flush(
fx_token_t dfx_token
);
Flushing an effect instance does not return its settings to their initial state. To do that, fuchsia_audio_dfx_reset must be called.
Below are fields and usages for the Media Client structs:
typedef void* fx_token_t;
This token represents an active instance of an effect. They are guaranteed to be unique, although new values might recycle those of previously-deleted effect instances.
typedef struct {
char name[FUCHSIA_AUDIO_DFX_MAX_NAME_LENGTH];
uint16_t num_controls;
uint16_t incoming_channels;
uint16_t outgoing_channels;
} fuchsia_audio_dfx_description;
This string is a descriptive (user-facing) name for the specified audio device effect type.
This indicates the number of settable controls on this effect type -- in a way, the “number of knobs and buttons” for this audio processor. For each of these controls, the system can retrieve a corresponding fuchsia_audio_dfx_control_description struct (see below).
typedef struct {
char name[FUCHSIA_AUDIO_DFX_MAX_NAME_LENGTH];
float max_val;
float min_val;
float initial_val;
} fuchsia_audio_dfx_control_description;
This string is a descriptive (user-facing) name for the specified control on this audio device effect.
This is the maximum value that can be set for this control.
This is the minimum value that can be set for this control.
This is the value of the specified control, at the time this effect instance was created.
typedef struct {
uint32_t frame_rate;
uint16_t channels_in;
uint16_t channels_out;
uint32_t signal_latency_frames;
uint32_t suggested_frames_per_buffer;
} fuchsia_audio_dfx_parameters;
This is the sampling rate (in Hertz) of the incoming and outgoing frames of audio data.
This is the number of channels in the incoming stream that is passed to the device effect instance via fuchsia_audio_dfx_process or fuchsia_audio_dfx_process_inplace calls. This value is important for calculating the correct number of samples to ingest (specifically, multiply this value by num_frames). As specified in media_types.fidl, the maximum number of input channels currently supported is 8.
This is the number of channels in the outgoing stream produced by the device effect instance via fuchsia_audio_dfx_process or fuchsia_audio_dfx_process_inplace calls. This value is important for calculating the correct number of samples to generate and populate (specifically, multiply this value by num_frames). As specified in media_types.fidl, the maximum number of input channels currently supported is 8.
Note that channel order is not explicitly specified here; where appropriate, the generally-accepted channel-ordering will be assumed (such as 5.1 ordering if 6 channels is specified). We acknowledge that this is inadequate specification for many use cases, but suggest that for now it may be adequate.
This is the group delay (in frames) caused by the audio processing algorithm itself, considering the parameters specified by fuchsia_audio_dfx_create (most notably frame_rate) when this effect instance was created. As an example, a basic volume effect should report a value of 0, since it will adds no “right-shift” of any kind when considering the incoming audio signal within the audio buffer; restated, this example effect needs no additional information beyond the first input sample itself, in order to produce the first output sample (hence a reported value of 0 frames of delay added). As another example, an effect that operates in the frequency domain would set this value based on the window size it uses (512, for example).
The effect uses this value to indicate a num_frames value at which it most effectively executes the calls fuchsia_audio_dfx_process or fuchsia_audio_dfx_process_inplace.
In general, this value represents the effect-creator's suggested balance between latency (which is minimized by smaller values) and efficiency (which is maximized by larger values). Effects that use frequency-domain processing can use this parameter to indicate the optimal window size for their algorithm. With this in mind, this value indicates not merely the ideal size of any single process call, but rather the ongoing alignment stride from the very start of processing.
This value is advisory-only. The system is not obligated to honor this request (indeed, in the case of multiple effects chained in series, this may not even be possible). Rather, the system makes decisions of this kind with the goal of optimizing overall system behavior, including responsiveness, fidelity, throughput, power consumption and other factors.
Below are constant values used with the DFX interfaces and structs:
const fx_token_t FUCHSIA_AUDIO_DFX_INVALID_TOKEN = 0;
This value is returned by fuchsia_audio_dfx_create, if an effect instance with the specified parameters cannot be created.
const uint16_t FUCHSIA_AUDIO_DFX_CHANNELS_ANY =
std::numeric_limits<uint16_t>::max();
This value can be returned in a fuchsia_audio_dfx_description struct, to indicate that an effect can potentially accommodate any number of channels. This value may NOT be used in fuchsia_audio_dfx_create calls or fuchsia_audio_dfx_parameters structs, as channels_in and channels_out in those contexts indicate the actual number of channels in the effect instance, rather than the possible values.
const uint16_t FUCHSIA_AUDIO_DFX_CHANNELS_SAME_AS_IN =
std::numeric_limits<uint16_t>::max() - 1;
This value can be returned in a fuchsia_audio_dfx_description struct for the value of outgoing_channels, to indicate that the number of output channels must equal the number of input channels. An effect should only use this if it reports its incoming_channels as FUCHSIA_AUDIO_DFX_CHANNELS_ANY.
const uint16_t FUCHSIA_AUDIO_DFX_CHANNELS_MAX = 256;
This is the maximum value allowed in fuchsia_audio_dfx_create calls or in the fuchsia_audio_dfx_parameters struct. It can also be used in fuchsia_audio_dfx_description structs, but note that in practice, effects are constrained by the number of channels supported by actual audio hardware. Currently the Fuchsia audio system limits this to MAX_PCM_CHANNEL_COUNT, which is less than this value.
const size_t FUCHSIA_AUDIO_DFX_MAX_NAME_LENGTH = 255;
This value is used as an outer limit on the lengths of names of effects and their controls. This const is important because it determines the effective size of the structs fuchsia_audio_dfx_description and fuchsia_audio_dfx_control_description (see above).