The audio effects interface enables device makers and system builders to add custom audio processing at a location in the audio pipeline that is unavailable to normal audio clients. This processing can be performed on the final mixed output audio stream right before it is consumed by playback hardware, and/or on an incoming stream immediately after it is produced by input hardware before it is provided to clients of the audio capture API.
The following interfaces, structures and constants are specified in the audio_effects.h file.
The Fuchsia audio system calls these device effects using the “C” ABI specified by the following interface calls and their related structs and consts. The system dynamically loads the library calls the synchronous functions described below to query and configure the effect, and subsequently to process audio while it is actively streaming.
The audio effects library can contain numerous different effects (such as an equalizer, a volume limiter, or a reverb); these are called effect types. For a given effect type, at any time there may be multiple independent copies of the effect that are active; each is called an effect instance. This is not unlike an object hierarchy; effect types equate to classes, and effect instances are analogous to instantiated objects of that class.
Additionally, an effect can be configured by settings that govern its behavior (such as EQ parameters or reverb decay time). In aggregate, these settings are referred to as the effects configuration. This configuration is provided to the effect in the form of a string, the schema of which is determined by each effect internally. Mirroring the object-oriented design mentioned above, the configuration and schema are defined across a given effect type, but each instance of that effect has an independent configuration values.
This low-level interface does not provide effects with any knowledge of where they are being inserted in the processing chain. This allows the system to be flexibly configured. However, a settings file (discussed elsewhere, TBD) is used to instruct the system which effects should be used, in which topology, with which configuration blobs.
Below are input parameters, return values and usage notes for the audio effects APIs. The library must export a fuchsia_audio_effects_module_v1 structure that contains data and function pointers that comprise that modules implementation:
typedef struct {
uint32_t num_effects;
bool (*get_info)(uint32_t effect_id, fuchsia_audio_effects_description* effect_desc);
fuchsia_audio_effects_handle_t (*create_effect)(uint32_t effect_id, uint32_t frame_rate,
uint16_t channels_in, uint16_t channels_out,
const char* config, size_t config_length);
bool (*update_effect_configuration)(fuchsia_audio_effects_handle_t effects_handle,
const char* config, size_t config_length);
bool (*delete_effect)(fuchsia_audio_effects_handle_t effects_handle);
bool (*get_parameters)(fuchsia_audio_effects_handle_t effects_handle,
fuchsia_audio_effects_parameters* effect_params);
bool (*process_inplace)(fuchsia_audio_effects_handle_t effects_handle, uint32_t num_frames,
float* audio_buff_in_out);
bool (*process)(fuchsia_audio_effects_handle_t effects_handle, uint32_t num_frames,
const float* audio_buff_in, float* audio_buff_out);
bool (*flush)(fuchsia_audio_effects_handle_t effects_handle);
} fuchsia_audio_effects_module_v1;
bool (*get_info)(
uint32_t effect_id,
fuchsia_audio_effects_description* effect_desc
);
*effect_desc contains information about the specified effect type.*effect_desc are undefined.fuchsia_audio_effects_handle_t (*create_effect)(
uint32_t effect_id,
uint32_t frame_rate,
uint16_t channels_in,
uint16_t channels_out,
const char* config,
size_t config_length
);
fuchsia_audio_effects_handle_t is a unique handle representing the just-created active instance of this effect type.The value that this call returns will be used as fuchsia_audio_effects_handle_t, in subsequent API calls that reference a particular effect instance.
When the values of channels_in and channels_out are equal, the system will require the created device effect to process audio in-place.
bool (*delete_effect)(
fuchsia_audio_effects_handle_t effects_handle
);
bool (*get_parameters)(
fuchsia_audio_effects_handle_t effects_handle,
fuchsia_audio_effects_parameters* effect_params
);
fuchsia_audio_effects_handle_t representing an active effect instance.*effect_params contains information about the specified effect instance.*effect_params are undefined.This interface returns the operational parameters for this instance of the device effect. These parameters are invariant for the lifetime of this effect, based on the initial values provided when the client created the effect.
bool (*process_inplace)(
fuchsia_audio_effects_handle_t effects_handle,
uint32_t num_frames,
float* audio_buff_in_out
);
fuchsia_audio_effects_handle_t representing an active effect instance.uint32_t containing the number of frames that the specified effect instance must process. This value cannot exceed the previously-set frame rate for this effect instance.float.num_frames of audio samples, in-place within the buffer specified by audio_buff_in_out.audio_buff_in_out are undefined.This interface synchronously processes the buffer of num_frames audio data in-place. The total number of samples processed is equal to num_frames multiplied by the value of channels_out, from the earlier fuchsia_audio_effects_module_v1::create call.
fuchsia_audio_effects_module_v1::process_inplace is used in all cases except where rechannelization occurs. It is preferred over fuchsia_audio_effects_module_v1::process because of its reduced memory consumption and related system impact.
The upper limit on the value of num_frames guarantees that no fuchsia_audio_effects_module_v1::process_inplace call will ever entail more than one second of audio.
bool (*process)(
fuchsia_audio_effects_handle_t effects_handle,
uint32_t num_frames,
const float* audio_buff_in,
float* audio_buff_out
);
fuchsia_audio_effects_handle_t representing an active effect instance.uint32_t containing the number of frames that the specified effect instance must process. This value cannot exceed the previously-set frame rate for this effect instance.float, from which num_frames audio frames must be ingested by the effect instance.num_frames audio frames of type float must be produced, based on the data ingested from audio_buff_in.num_frames of audio samples from the buffer specified by audio_buff_in, to the buffer specified by audio_buff_out.audio_buff_out are undefined.This interface synchronously processes ‘num_frames’ of audio data in type float, from audio_buff_in to audio_buff_out. The total number of input samples ingested is equal to num_frames multiplied by the value of channels_in from the earlier fuchsia_audio_effects_module_v1::create call. The total number of output samples generated and populated into audio_buff_out is equal to num_frames multiplied by the value of channels_out from that same fuchsia_audio_effects_module_v1::create call.
This interface is necessary for cases where rechannelization occurs; it is not used otherwise. Where possible, the system instead uses fuchsia_audio_effects_module_v1::process_inplace, because of its reduced memory consumption and related system impact.
The upper limit on the value of num_frames guarantees that no fuchsia_audio_effects_module_v1::process call will ever entail more than one second of audio.
Below are fields and usages for the audio effects structs:
typedef void* fuchsia_audio_effects_handle_t;
This handle represents an active instance of an effect. They are guaranteed to be unique, although new values might recycle those of previously-deleted effect instances.
typedef struct {
char name[FUCHSIA_AUDIO_EFFECTS_MAX_NAME_LENGTH];
uint16_t incoming_channels;
uint16_t outgoing_channels;
} fuchsia_audio_effects_description;
This string is a descriptive (user-facing) name for the specified audio device effect type.
typedef struct {
uint32_t frame_rate;
uint16_t channels_in;
uint16_t channels_out;
uint32_t signal_latency_frames;
uint32_t suggested_frames_per_buffer;
} fuchsia_audio_effects_parameters;
This is the sampling rate (in Hertz) of the incoming and outgoing frames of audio data.
This is the number of channels in the incoming stream that is passed to the device effect instance via fuchsia_audio_effects_module_v1::process or fuchsia_audio_effects_module_v1::process_inplace calls. This value is important for calculating the correct number of samples to ingest (specifically, multiply this value by num_frames). See audio.fidl for the current maximum number of input channels currently supported.
This is the number of channels in the outgoing stream produced by the device effect instance via fuchsia_audio_effects_module_v1::process or fuchsia_audio_effects_module_v1::process_inplace calls. This value is important for calculating the correct number of samples to generate and populate (specifically, multiply this value by num_frames).
Note that channel order is not explicitly specified here; where appropriate, the generally-accepted channel-ordering will be assumed (such as 5.1 ordering if 6 channels is specified). We acknowledge that this is inadequate specification for many use cases, but suggest that for now it may be adequate.
This is the group delay (in frames) caused by the audio processing algorithm itself, considering the parameters specified by fuchsia_audio_effects_module_v1::create (most notably frame_rate) when this effect instance was created. As an example, a basic volume effect should report a value of 0, since it will adds no “right-shift” of any kind when considering the incoming audio signal within the audio buffer; restated, this example effect needs no additional information beyond the first input sample itself, in order to produce the first output sample (hence a reported value of 0 frames of delay added). As another example, an effect that operates in the frequency domain would set this value based on the window size it uses (512, for example).
The effect uses this value to indicate a num_frames value at which it most effectively executes the calls fuchsia_audio_effects_module_v1::process or fuchsia_audio_effects_module_v1::process_inplace.
In general, this value represents the effect-creator's suggested balance between latency (which is minimized by smaller values) and efficiency (which is maximized by larger values). Effects that use frequency-domain processing can use this parameter to indicate the optimal window size for their algorithm. With this in mind, this value indicates not merely the ideal size of any single process call, but rather the ongoing alignment stride from the very start of processing.
This value is advisory-only. The system is not obligated to honor this request (indeed, in the case of multiple effects chained in series, this may not even be possible). Rather, the system makes decisions of this kind with the goal of optimizing overall system behavior, including responsiveness, fidelity, throughput, power consumption and other factors.
Below are constant values used with the audio effects interfaces and structs:
const fuchsia_audio_effects_handle_t FUCHSIA_AUDIO_EFFECTS_INVALID_HANDLE = 0;
This value is returned by fuchsia_audio_effects_module_v1::create, if an effect instance with the specified parameters cannot be created.
const uint16_t FUCHSIA_AUDIO_EFFECTS_CHANNELS_ANY =
std::numeric_limits<uint16_t>::max();
This value can be returned in a fuchsia_audio_effects_description struct, to indicate that an effect can potentially accommodate any number of channels. This value may NOT be used in fuchsia_audio_effects_module_v1::create calls or fuchsia_audio_effects_parameters structs, as channels_in and channels_out in those contexts indicate the actual number of channels in the effect instance, rather than the possible values.
const uint16_t FUCHSIA_AUDIO_EFFECTS_CHANNELS_SAME_AS_IN =
std::numeric_limits<uint16_t>::max() - 1;
This value can be returned in a fuchsia_audio_effects_description struct for the value of outgoing_channels, to indicate that the number of output channels must equal the number of input channels. An effect should only use this if it reports its incoming_channels as FUCHSIA_AUDIO_EFFECTS_CHANNELS_ANY.
const uint16_t FUCHSIA_AUDIO_EFFECTS_CHANNELS_MAX = 256;
This is the maximum value allowed in fuchsia_audio_effects_create calls or in the fuchsia_audio_effects_parameters struct. It can also be used in fuchsia_audio_effects_description structs, but note that in practice, effects are constrained by the number of channels supported by actual audio hardware. Currently the Fuchsia audio system limits this to MAX_PCM_CHANNEL_COUNT, which is less than this value.
const size_t FUCHSIA_AUDIO_EFFECTS_MAX_NAME_LENGTH = 255;
This value is used as an outer limit on the lengths of names of effects. This constant is important because it determines the effective size of the struct fuchsia_audio_effects_description.