[Codec] VP9 packet != buffer; stage 1
VP9 has show_existing_frame, which AFAICT forces some codec models to
copy uncompressed output video frames, as they aren't capable of having
the same buffer in the output queue more than once (roughly speaking).
We don't want our codec model to be one of those codec models, so this
commit starts de-conflating packet_index and buffer_index.
With this commit, VP9 with show_existing_frame decodes and displays
with use_media_decoder. H264 continues to work, but for now
intentionally preserves packet == buffer (temporarily) to avoid breaking
code that hasn't yet been updated to allow packet != buffer.
(mediaplayer_tests getting VP9 is after SAR is plumbed from VP9 + some
changes to allow packet != buffer.)
This also tweaks some video_decoder_lock_ hold intervals, adds some
comments re. a potential sharp edge re. VideoFrame shared_ptr<> and
weak_ptr<> and video_decoder_lock_, and fences out any in-flight
CoreCodecRecycleOutputPacket() before packets go away.
Tested: use_media_decoder --vp9 (show_existing_frame/normal) / --h264, mediaplayer_tests (h264), use_aac_decoder_test
Change-Id: I8d967cd8ec2c5cd30f41542a0871f7d8e0abf844
diff --git a/bin/media/codecs/sw/omx/codec_runner_sw_omx/omx_codec_runner.cc b/bin/media/codecs/sw/omx/codec_runner_sw_omx/omx_codec_runner.cc
index d0fac7a..075a272 100644
--- a/bin/media/codecs/sw/omx/codec_runner_sw_omx/omx_codec_runner.cc
+++ b/bin/media/codecs/sw/omx/codec_runner_sw_omx/omx_codec_runner.cc
@@ -1627,6 +1627,13 @@
}
assert(packet.stream_lifetime_ordinal == stream_lifetime_ordinal_);
+ if (packet.header.packet_index >= all_packets_[kInput].size()) {
+ Exit("client QueueInputPacket() with packet_index out of range");
+ }
+ if (packet.buffer_index >= all_buffers_[kInput].size()) {
+ Exit("client QueueInputPacket() with buffer_index out of range");
+ }
+
// Protocol check re. free/busy coherency.
if (!packet_free_bits_[kInput][packet.header.packet_index]) {
Exit("client QueueInputPacket() with packet_index !free - exiting\n");
@@ -3084,6 +3091,7 @@
.header.buffer_lifetime_ordinal =
packet->buffer_lifetime_ordinal(),
.header.packet_index = packet->packet_index(),
+ .buffer_index = packet->buffer().buffer_index(),
.stream_lifetime_ordinal = stream_lifetime_ordinal_,
.start_offset = pBuffer->nOffset,
.valid_length_bytes = pBuffer->nFilledLen,
diff --git a/bin/mediaplayer/fidl/fidl_decoder.cc b/bin/mediaplayer/fidl/fidl_decoder.cc
index e4d18ae..62df2a0 100644
--- a/bin/mediaplayer/fidl/fidl_decoder.cc
+++ b/bin/mediaplayer/fidl/fidl_decoder.cc
@@ -235,6 +235,7 @@
codec_packet.header.buffer_lifetime_ordinal =
current_set.lifetime_ordinal();
codec_packet.header.packet_index = packet->payload_buffer()->id();
+ codec_packet.buffer_index = packet->payload_buffer()->id();
codec_packet.stream_lifetime_ordinal = stream_lifetime_ordinal_;
codec_packet.start_offset = 0;
codec_packet.valid_length_bytes = packet->size();
@@ -535,7 +536,14 @@
FXL_DCHECK_CREATION_THREAD_IS_CURRENT(thread_checker_);
uint64_t buffer_lifetime_ordinal = packet.header.buffer_lifetime_ordinal;
- uint32_t buffer_index = packet.header.packet_index;
+ uint32_t packet_index = packet.header.packet_index;
+ uint32_t buffer_index = packet.buffer_index;
+ FXL_DCHECK(buffer_index != 0x80000000);
+
+ // TODO(dustingreen): separate buffer_index from packet_index in FidlDecoder.
+ // Until then, this will work for h264, but won't handle VP9 with its
+ // show_existing_frame that can happen repeatedly for the same buffer.
+ FXL_CHECK(packet_index == buffer_index);
if (error_detected_before) {
FXL_LOG(WARNING) << "OnOutputPacket: error_detected_before";
diff --git a/drivers/video/amlogic-decoder/codec_adapter_h264.cc b/drivers/video/amlogic-decoder/codec_adapter_h264.cc
index 118f1a1..a821259 100644
--- a/drivers/video/amlogic-decoder/codec_adapter_h264.cc
+++ b/drivers/video/amlogic-decoder/codec_adapter_h264.cc
@@ -167,9 +167,29 @@
io_buffer_cache_flush_invalidate(&frame->buffer, frame->uv_plane_offset,
frame->stride * frame->height / 2);
- CodecPacket* packet = frame->codec_packet;
+ const CodecBuffer* buffer = frame->codec_buffer;
+ ZX_DEBUG_ASSERT(buffer);
+
+ // We intentionally _don't_ use the packet with same index as the buffer (in
+ // general - it's fine that they sometimes match), to avoid clients building
+ // up inappropriate dependency on buffer index being the same as packet
+ // index (as nice as that would be, VP9, and maybe others, don't get along
+ // with that in general, so ... force clients to treat packet index and
+ // buffer index as separate things).
+ //
+ // TODO(dustingreen): Make the previous paragraph fully true. For the
+ // moment for h264 we let the packet_index and buffer_index be equal, just
+ // to split up the separation into smaller steps.
+ CodecPacket* packet = GetFreePacket(buffer->buffer_index());
+ // With h.264, we know that an emitted buffer implies an available output
+ // packet, because h.264 doesn't put the same output buffer in flight more
+ // than once concurrently, and we have as many output packets as buffers.
+ // This contrasts with VP9 which has unbounded show_existing_frame.
ZX_DEBUG_ASSERT(packet);
+ // Associate the packet with the buffer while the packet is in-flight.
+ packet->SetBuffer(buffer);
+
packet->SetStartOffset(0);
uint64_t total_size_bytes = frame->stride * frame->height * 3 / 2;
packet->SetValidLengthBytes(total_size_bytes);
@@ -223,7 +243,7 @@
CodecInputItem::FormatDetails(per_stream_override_format_details));
}
-void CodecAdapterH264::CoreCodecQueueInputPacket(const CodecPacket* packet) {
+void CodecAdapterH264::CoreCodecQueueInputPacket(CodecPacket* packet) {
QueueInputItem(CodecInputItem::Packet(packet));
}
@@ -322,16 +342,30 @@
}
ZX_DEBUG_ASSERT(!packet->is_new());
- std::shared_ptr<VideoFrame> frame = packet->video_frame().lock();
- if (!frame) {
- // EndOfStream seen at the output, or a new InitializeFrames(), can cause
- // !frame, which is fine. In that case, any new stream will request
- // allocation of new frames.
- return;
- }
+ // A recycled packet will have a buffer set because the packet is in-flight
+ // until put on the free list, and has a buffer associated while in-flight.
+ const CodecBuffer* buffer = packet->buffer();
+ ZX_DEBUG_ASSERT(buffer);
+
+ // Getting the buffer is all we needed the packet for. The packet won't get
+ // re-used until after ReturnFrame() below.
+ packet->SetBuffer(nullptr);
+
+ // TODO(dustingreen): Use scrambled free_packet_list_ like for VP9, and put
+ // packet back on the free list here (and update comment above re. when re-use
+ // of the paket can happen). For now for this codec we let packet_index ==
+ // buffer_index to split the separation of buffer_index from packet_index into
+ // smaller changes.
{ // scope lock
std::lock_guard<std::mutex> lock(*video_->video_decoder_lock());
+ std::shared_ptr<VideoFrame> frame = buffer->video_frame().lock();
+ if (!frame) {
+ // EndOfStream seen at the output, or a new InitializeFrames(), can cause
+ // !frame, which is fine. In that case, any new stream will request
+ // allocation of new frames.
+ return;
+ }
// Recycle can happen while stopped, but this CodecAdapater has no way yet
// to return frames while stopped, or to re-use buffers/frames across a
// stream switch. Any new stream will request allocation of new frames.
@@ -526,8 +560,9 @@
ZX_DEBUG_ASSERT(all_output_buffers_[i]->buffer_index() == i);
ZX_DEBUG_ASSERT(all_output_buffers_[i]->codec_buffer().buffer_index == i);
frames.emplace_back(CodecFrame{
- .codec_buffer = fidl::Clone(all_output_buffers_[i]->codec_buffer()),
- .codec_packet = all_output_packets_[i],
+ .codec_buffer_spec =
+ fidl::Clone(all_output_buffers_[i]->codec_buffer()),
+ .codec_buffer_ptr = all_output_buffers_[i],
});
}
width = width_;
@@ -643,7 +678,7 @@
}
uint8_t* data =
- item.packet()->buffer().buffer_base() + item.packet()->start_offset();
+ item.packet()->buffer()->buffer_base() + item.packet()->start_offset();
uint32_t len = item.packet()->valid_length_bytes();
video_->pts_manager()->InsertPts(parsed_video_size_,
@@ -1043,3 +1078,11 @@
}
events_->onCoreCodecFailStream();
}
+
+CodecPacket* CodecAdapterH264::GetFreePacket(uint32_t buffer_index) {
+ // TODO(dustingreen): Intentionally don't always have packet_index ==
+ // buffer_index. However, for the moment, for h.264, always have packet_index
+ // == buffer_index, as a temporary measure to do the separation in smaller
+ // changes.
+ return all_output_packets_[buffer_index];
+}
diff --git a/drivers/video/amlogic-decoder/codec_adapter_h264.h b/drivers/video/amlogic-decoder/codec_adapter_h264.h
index 3e49bb2..f6faa14 100644
--- a/drivers/video/amlogic-decoder/codec_adapter_h264.h
+++ b/drivers/video/amlogic-decoder/codec_adapter_h264.h
@@ -28,7 +28,7 @@
void CoreCodecQueueInputFormatDetails(
const fuchsia::mediacodec::CodecFormatDetails&
per_stream_override_format_details) override;
- void CoreCodecQueueInputPacket(const CodecPacket* packet) override;
+ void CoreCodecQueueInputPacket(CodecPacket* packet) override;
void CoreCodecQueueInputEndOfStream() override;
void CoreCodecStopStream() override;
void CoreCodecAddBuffer(CodecPort port, const CodecBuffer* buffer) override;
@@ -68,6 +68,7 @@
uint32_t sar_width, uint32_t sar_height);
void OnCoreCodecFailStream();
+ CodecPacket* GetFreePacket(uint32_t buffer_index);
DeviceCtx* device_ = nullptr;
AmlogicVideo* video_ = nullptr;
diff --git a/drivers/video/amlogic-decoder/codec_adapter_mpeg2.cc b/drivers/video/amlogic-decoder/codec_adapter_mpeg2.cc
index e5dc3d2..4a3c764 100644
--- a/drivers/video/amlogic-decoder/codec_adapter_mpeg2.cc
+++ b/drivers/video/amlogic-decoder/codec_adapter_mpeg2.cc
@@ -43,7 +43,7 @@
ZX_DEBUG_ASSERT_MSG(false, "not yet implemented");
}
-void CodecAdapterMpeg2::CoreCodecQueueInputPacket(const CodecPacket* packet) {
+void CodecAdapterMpeg2::CoreCodecQueueInputPacket(CodecPacket* packet) {
ZX_DEBUG_ASSERT_MSG(false, "not yet implemented");
}
diff --git a/drivers/video/amlogic-decoder/codec_adapter_mpeg2.h b/drivers/video/amlogic-decoder/codec_adapter_mpeg2.h
index a95c688..cacd4be 100644
--- a/drivers/video/amlogic-decoder/codec_adapter_mpeg2.h
+++ b/drivers/video/amlogic-decoder/codec_adapter_mpeg2.h
@@ -24,7 +24,7 @@
void CoreCodecQueueInputFormatDetails(
const fuchsia::mediacodec::CodecFormatDetails&
per_stream_override_format_details) override;
- void CoreCodecQueueInputPacket(const CodecPacket* packet) override;
+ void CoreCodecQueueInputPacket(CodecPacket* packet) override;
void CoreCodecQueueInputEndOfStream() override;
void CoreCodecStopStream() override;
void CoreCodecAddBuffer(CodecPort port, const CodecBuffer* buffer) override;
diff --git a/drivers/video/amlogic-decoder/codec_adapter_vp9.cc b/drivers/video/amlogic-decoder/codec_adapter_vp9.cc
index b748c16..abb1418 100644
--- a/drivers/video/amlogic-decoder/codec_adapter_vp9.cc
+++ b/drivers/video/amlogic-decoder/codec_adapter_vp9.cc
@@ -157,9 +157,14 @@
io_buffer_cache_flush_invalidate(&frame->buffer, frame->uv_plane_offset,
frame->stride * frame->height / 2);
- CodecPacket* packet = frame->codec_packet;
+ const CodecBuffer* buffer = frame->codec_buffer;
+ ZX_DEBUG_ASSERT(buffer);
+
+ CodecPacket* packet = GetFreePacket();
+ // We know there will be a free packet thanks to SetCheckOutputReady().
ZX_DEBUG_ASSERT(packet);
+ packet->SetBuffer(buffer);
packet->SetStartOffset(0);
uint64_t total_size_bytes = frame->stride * frame->height * 3 / 2;
packet->SetValidLengthBytes(total_size_bytes);
@@ -175,6 +180,13 @@
decoder->SetInitializeFramesHandler(
fit::bind_member(this, &CodecAdapterVp9::InitializeFramesHandler));
decoder->SetErrorHandler([this] { OnCoreCodecFailStream(); });
+ decoder->SetCheckOutputReady([this] {
+ std::lock_guard<std::mutex> lock(lock_);
+ // We're ready if output hasn't been configured yet, or if we have free
+ // output packets. This way the decoder can swap in when there's no output
+ // config yet, but will stop trying to run when we're out of output packets.
+ return all_output_packets_.empty() || !free_output_packets_.empty();
+ });
{ // scope lock
std::lock_guard<std::mutex> lock(*video_->video_decoder_lock());
@@ -212,7 +224,7 @@
CodecInputItem::FormatDetails(per_stream_override_format_details));
}
-void CodecAdapterVp9::CoreCodecQueueInputPacket(const CodecPacket* packet) {
+void CodecAdapterVp9::CoreCodecQueueInputPacket(CodecPacket* packet) {
QueueInputItem(CodecInputItem::Packet(packet));
}
@@ -292,7 +304,13 @@
ZX_DEBUG_ASSERT(all_output_buffers_.size() == packets.size());
for (auto& packet : packets) {
all_output_packets_.push_back(packet.get());
+ free_output_packets_.push_back(packet.get()->packet_index());
}
+ // This should prevent any inadvetent dependence by clients on the ordering
+ // of packet_index values in the output stream or any assumptions re. the
+ // relationship between packet_index and buffer_index.
+ std::shuffle(free_output_packets_.begin(), free_output_packets_.end(),
+ not_for_security_prng_);
}
}
@@ -303,16 +321,27 @@
}
ZX_DEBUG_ASSERT(!packet->is_new());
- std::shared_ptr<VideoFrame> frame = packet->video_frame().lock();
- if (!frame) {
- // EndOfStream seen at the output, or a new InitializeFrames(), can cause
- // !frame, which is fine. In that case, any new stream will request
- // allocation of new frames.
- return;
- }
+ const CodecBuffer* buffer = packet->buffer();
+ packet->SetBuffer(nullptr);
+
+ // Getting the buffer is all we needed the packet for, so note that the packet
+ // is free fairly early, to side-step any issues with early returns. The
+ // CodecImpl already considers the packet free, but it won't actually get
+ // re-used until after it goes on the free list here.
+ { // scope lock
+ std::lock_guard<std::mutex> lock(lock_);
+ free_output_packets_.push_back(packet->packet_index());
+ } // ~lock
{ // scope lock
std::lock_guard<std::mutex> lock(*video_->video_decoder_lock());
+ std::shared_ptr<VideoFrame> frame = buffer->video_frame().lock();
+ if (!frame) {
+ // EndOfStream seen at the output, or a new InitializeFrames(), can cause
+ // !frame, which is fine. In that case, any new stream will request
+ // allocation of new frames.
+ return;
+ }
// Recycle can happen while stopped, but this CodecAdapater has no way yet
// to return frames while stopped, or to re-use buffers/frames across a
// stream switch. Any new stream will request allocation of new frames.
@@ -508,8 +537,9 @@
ZX_DEBUG_ASSERT(all_output_buffers_[i]->buffer_index() == i);
ZX_DEBUG_ASSERT(all_output_buffers_[i]->codec_buffer().buffer_index == i);
frames.emplace_back(CodecFrame{
- .codec_buffer = fidl::Clone(all_output_buffers_[i]->codec_buffer()),
- .codec_packet = all_output_packets_[i],
+ .codec_buffer_spec =
+ fidl::Clone(all_output_buffers_[i]->codec_buffer()),
+ .codec_buffer_ptr = all_output_buffers_[i],
});
}
width = width_;
@@ -667,7 +697,7 @@
ZX_DEBUG_ASSERT(item.is_packet());
uint8_t* data =
- item.packet()->buffer().buffer_base() + item.packet()->start_offset();
+ item.packet()->buffer()->buffer_base() + item.packet()->start_offset();
uint32_t len = item.packet()->valid_length_bytes();
video_->pts_manager()->InsertPts(parsed_video_size_,
@@ -808,3 +838,10 @@
}
events_->onCoreCodecFailStream();
}
+
+CodecPacket* CodecAdapterVp9::GetFreePacket() {
+ std::lock_guard<std::mutex> lock(lock_);
+ uint32_t free_index = free_output_packets_.back();
+ free_output_packets_.pop_back();
+ return all_output_packets_[free_index];
+}
diff --git a/drivers/video/amlogic-decoder/codec_adapter_vp9.h b/drivers/video/amlogic-decoder/codec_adapter_vp9.h
index 4c94108..235f868 100644
--- a/drivers/video/amlogic-decoder/codec_adapter_vp9.h
+++ b/drivers/video/amlogic-decoder/codec_adapter_vp9.h
@@ -12,6 +12,8 @@
#include "vp9_decoder.h"
+#include <random>
+
class AmlogicVideo;
struct CodecFrame;
class DeviceCtx;
@@ -32,7 +34,7 @@
void CoreCodecQueueInputFormatDetails(
const fuchsia::mediacodec::CodecFormatDetails&
per_stream_override_format_details) override;
- void CoreCodecQueueInputPacket(const CodecPacket* packet) override;
+ void CoreCodecQueueInputPacket(CodecPacket* packet) override;
void CoreCodecQueueInputEndOfStream() override;
void CoreCodecStopStream() override;
void CoreCodecAddBuffer(CodecPort port, const CodecBuffer* buffer) override;
@@ -68,6 +70,7 @@
uint32_t sar_width, uint32_t sar_height);
void OnCoreCodecFailStream();
+ CodecPacket* GetFreePacket();
DeviceCtx* device_ = nullptr;
AmlogicVideo* video_ = nullptr;
@@ -92,6 +95,7 @@
std::vector<const CodecBuffer*> all_output_buffers_;
std::vector<CodecPacket*> all_output_packets_;
+ std::vector<uint32_t> free_output_packets_;
uint32_t packet_count_total_ = 0;
uint32_t width_ = 0;
diff --git a/drivers/video/amlogic-decoder/h264_decoder.cc b/drivers/video/amlogic-decoder/h264_decoder.cc
index eff29da..d747013 100644
--- a/drivers/video/amlogic-decoder/h264_decoder.cc
+++ b/drivers/video/amlogic-decoder/h264_decoder.cc
@@ -4,6 +4,7 @@
#include "h264_decoder.h"
+#include <lib/media/codec_impl/codec_buffer.h>
#include <lib/media/codec_impl/codec_frame.h>
#include <lib/media/codec_impl/codec_packet.h>
#include <zx/vmo.h>
@@ -400,7 +401,8 @@
// owner_->ConfigureCanvas() below.
zx_status_t status = io_buffer_init_vmo(
&frame->buffer, owner_->bti(),
- frames[i].codec_buffer.data.vmo().vmo_handle.get(), 0, IO_BUFFER_RW);
+ frames[i].codec_buffer_spec.data.vmo().vmo_handle.get(), 0,
+ IO_BUFFER_RW);
if (status != ZX_OK) {
DECODE_ERROR("Failed to io_buffer_init_vmo() for frame - status: %d\n",
status);
@@ -420,9 +422,9 @@
frame->index = i;
// can be nullptr
- frame->codec_packet = frames[i].codec_packet;
- if (frames[i].codec_packet) {
- frames[i].codec_packet->SetVideoFrame(frame);
+ frame->codec_buffer = frames[i].codec_buffer_ptr;
+ if (frames[i].codec_buffer_ptr) {
+ frames[i].codec_buffer_ptr->SetVideoFrame(frame);
}
// The ConfigureCanvas() calls validate that the VMO is physically
@@ -505,14 +507,14 @@
.vmo_usable_size = frame_vmo_bytes,
});
frames.emplace_back(CodecFrame{
- .codec_buffer =
+ .codec_buffer_spec =
fuchsia::mediacodec::CodecBuffer{
.buffer_lifetime_ordinal =
next_non_codec_buffer_lifetime_ordinal_,
.buffer_index = i,
.data = std::move(codec_buffer_data),
},
- .codec_packet = nullptr,
+ .codec_buffer_ptr = nullptr,
});
}
next_non_codec_buffer_lifetime_ordinal_++;
diff --git a/drivers/video/amlogic-decoder/video_decoder.h b/drivers/video/amlogic-decoder/video_decoder.h
index 142a459..3a5de31 100644
--- a/drivers/video/amlogic-decoder/video_decoder.h
+++ b/drivers/video/amlogic-decoder/video_decoder.h
@@ -60,6 +60,9 @@
class CodecPacket;
class VideoDecoder {
public:
+ // In actual operation, the FrameReadyNotifier must not keep a reference on
+ // the frame shared_ptr<>, as that would interfere with muting calls to
+ // ReturnFrame(). See comment on Vp9Decoder::Frame::frame field.
using FrameReadyNotifier = std::function<void(std::shared_ptr<VideoFrame>)>;
using InitializeFramesHandler =
std::function<zx_status_t(::zx::bti,
@@ -73,6 +76,7 @@
uint32_t, // sar_width
uint32_t // sar_height
)>;
+ using CheckOutputReady = fit::function<bool()>;
class Owner {
public:
virtual __WARN_UNUSED_RESULT DosRegisterIo* dosbus() = 0;
@@ -97,13 +101,16 @@
return ZX_ERR_NOT_SUPPORTED;
}
virtual void HandleInterrupt() = 0;
- virtual void SetFrameReadyNotifier(FrameReadyNotifier notifier) {}
+ virtual void SetFrameReadyNotifier(FrameReadyNotifier notifier) = 0;
virtual void SetInitializeFramesHandler(InitializeFramesHandler handler) {
ZX_ASSERT_MSG(false, "not yet implemented");
}
virtual void SetErrorHandler(fit::closure error_handler) {
ZX_ASSERT_MSG(false, "not yet implemented");
}
+ virtual void SetCheckOutputReady(CheckOutputReady checkOutputReady) {
+ ZX_ASSERT_MSG(false, "not yet implemented");
+ };
virtual void ReturnFrame(std::shared_ptr<VideoFrame> frame) = 0;
virtual void InitializedFrames(std::vector<CodecFrame> frames, uint32_t width,
uint32_t height, uint32_t stride) = 0;
diff --git a/drivers/video/amlogic-decoder/video_frame.h b/drivers/video/amlogic-decoder/video_frame.h
index 63a633f..0c4846d 100644
--- a/drivers/video/amlogic-decoder/video_frame.h
+++ b/drivers/video/amlogic-decoder/video_frame.h
@@ -9,7 +9,7 @@
#include <cstdint>
-class CodecPacket;
+class CodecBuffer;
struct VideoFrame {
~VideoFrame() { io_buffer_release(&buffer); }
@@ -25,7 +25,7 @@
bool has_pts = false;
uint64_t pts = 0;
- CodecPacket* codec_packet = nullptr;
+ const CodecBuffer* codec_buffer = nullptr;
};
// The video frame must be in NV12 format. The output file can be read using
diff --git a/drivers/video/amlogic-decoder/vp9_decoder.cc b/drivers/video/amlogic-decoder/vp9_decoder.cc
index 4058975..0c7c1d3 100644
--- a/drivers/video/amlogic-decoder/vp9_decoder.cc
+++ b/drivers/video/amlogic-decoder/vp9_decoder.cc
@@ -4,6 +4,7 @@
#include "vp9_decoder.h"
+#include <lib/media/codec_impl/codec_buffer.h>
#include <lib/media/codec_impl/codec_packet.h>
#include "firmware_blob.h"
@@ -446,15 +447,16 @@
video_frame->uv_plane_offset = video_frame->stride * video_frame->height;
video_frame->index = i;
- video_frame->codec_packet = frames[i].codec_packet;
- if (frames[i].codec_packet) {
- frames[i].codec_packet->SetVideoFrame(video_frame);
+ video_frame->codec_buffer = frames[i].codec_buffer_ptr;
+ if (frames[i].codec_buffer_ptr) {
+ frames[i].codec_buffer_ptr->SetVideoFrame(video_frame);
}
assert(video_frame->height % 2 == 0);
zx_status_t status = io_buffer_init_vmo(
&video_frame->buffer, owner_->bti(),
- frames[i].codec_buffer.data.vmo().vmo_handle.get(), 0, IO_BUFFER_RW);
+ frames[i].codec_buffer_spec.data.vmo().vmo_handle.get(), 0,
+ IO_BUFFER_RW);
if (status != ZX_OK) {
DECODE_ERROR("Failed to io_buffer_init_vmo() for frame - status: %d\n",
status);
@@ -501,7 +503,13 @@
ref_frame->refcount--;
assert(ref_frame->refcount >= 0);
- if (waiting_for_empty_frames_) {
+ // If either of these bools is true, we know the decoder isn't running. It's
+ // fine that we don't check here that there's a frame with refcount 0 or check
+ // here that the output is ready, because PrepareNewFrame() will re-check
+ // both those things, and set the appropriate waiting bool back to true if we
+ // still need to wait.
+ if (waiting_for_output_ready_ || waiting_for_empty_frames_) {
+ waiting_for_output_ready_ = false;
waiting_for_empty_frames_ = false;
PrepareNewFrame();
}
@@ -847,6 +855,10 @@
if (!has_available_output_frames)
return false;
+ if (check_output_ready_ && !check_output_ready_()) {
+ return false;
+ }
+
return frame_data_provider_->HasMoreInputData();
}
@@ -888,6 +900,15 @@
}
void Vp9Decoder::PrepareNewFrame() {
+ if (check_output_ready_ && !check_output_ready_()) {
+ // Becomes false when ReturnFrame() gets called, at which point
+ // PrepareNewFrame() gets another chance to check again and set back to true
+ // as necessary. This bool needs to exist only so that ReturnFrame() can
+ // know whether the decoder is currently needing PrepareNewFrame().
+ waiting_for_output_ready_ = true;
+ return;
+ }
+
HardwareRenderParams params;
BarrierBeforeInvalidate();
io_buffer_cache_flush_invalidate(&working_buffers_.rpm.buffer(), 0,
@@ -966,6 +987,10 @@
notifier_ = notifier;
}
+void Vp9Decoder::SetCheckOutputReady(CheckOutputReady check_output_ready) {
+ check_output_ready_ = std::move(check_output_ready);
+}
+
Vp9Decoder::Frame::~Frame() {
io_buffer_release(&compressed_header);
io_buffer_release(&compressed_data);
@@ -1001,8 +1026,45 @@
(new_frame->frame->height != params->height)) {
BarrierBeforeRelease();
// It's simplest to allocate all frames at once on resize, though that can
- // cause problems if show_existing_frame happens after the resize.
+ // cause frames that should have been output to not be output if a
+ // show_existing_frame after the resize wants to show a pre-resize frame, or
+ // if the reallocate leads to reference frames that aren't available to use
+ // for constructing a frame.
+ //
+ // We care that the decoder doesn't crash across buffer reallocation, and
+ // that it re-synchronizes with the stream after a while (doesn't refuse to
+ // deliver output frames forever), but we don't (so far) care that frames
+ // can be dropped when resolution switching also involves re-allocating
+ // buffers.
+ //
+ // TODO(dustingreen): When buffers are large enough and aren't reallocated,
+ // resolution switching should be seamless. See also TODO above re. keeping
+ // larger buffers if the needed buffer size goes down.
+ //
+ // The reason for having a higher bar for degree of seamless-ness when
+ // buffers are not reallocated (vs. lower-than-"perfect" bar when they are
+ // re-allocated) is partly because of the need for phsyically contiguous
+ // VMOs and the associated potential for physical memory fragmentation
+ // caused by piecemeal buffer allocation and deallocation given an arbitrary
+ // VP9 stream that has arbitrary resolution switching and
+ // show_existing_frame. The ability to seamlessly switch/adjust resolution
+ // within a buffer set that is large enough to support the max resolution of
+ // the stream should offer sufficient functionality to avoid causing
+ // practical problems for clients, and this bar being set where it is should
+ // avoid creating physical fragmentation / excessive physical reservation
+ // problems for the overall system. It also reduces complexity (vs.
+ // "perfect") for clients and for codecs without sacrificing resolution
+ // switching entirely. It also avoids assuming that buffers can be
+ // dynamically added/removed from a buffer set without creating timing
+ // problems (and/or requiring more buffers to compensate for timing effects
+ // of dynamic add/remove).
for (uint32_t i = 0; i < frames_.size(); i++) {
+ // In normal operation (outside decoder self-tests) this reset() is relied
+ // upon to essentially signal to the CodecBuffer::frame weak_ptr<> that
+ // ReturnFrame() should no longer be called on this frame. This implies
+ // (for now) that the VideoFrame must not be shared outside transients
+ // under video_decoder_lock_. See comment on Vp9Decoder::Frame::frame for
+ // more.
frames_[i]->frame.reset();
}
@@ -1051,14 +1113,14 @@
.vmo_usable_size = frame_vmo_bytes,
});
frames.emplace_back(CodecFrame{
- .codec_buffer =
+ .codec_buffer_spec =
fuchsia::mediacodec::CodecBuffer{
.buffer_lifetime_ordinal =
next_non_codec_buffer_lifetime_ordinal_,
.buffer_index = 0,
.data = std::move(codec_buffer_data),
},
- .codec_packet = nullptr,
+ .codec_buffer_ptr = nullptr,
});
}
next_non_codec_buffer_lifetime_ordinal_++;
diff --git a/drivers/video/amlogic-decoder/vp9_decoder.h b/drivers/video/amlogic-decoder/vp9_decoder.h
index e940871..55e0131 100644
--- a/drivers/video/amlogic-decoder/vp9_decoder.h
+++ b/drivers/video/amlogic-decoder/vp9_decoder.h
@@ -72,10 +72,14 @@
__WARN_UNUSED_RESULT zx_status_t Initialize() override;
__WARN_UNUSED_RESULT zx_status_t InitializeHardware() override;
void HandleInterrupt() override;
+ // In actual operation, the FrameReadyNotifier must not keep a reference on
+ // the frame shared_ptr<>, as that would interfere with muting calls to
+ // ReturnFrame(). See comment on Vp9Decoder::Frame::frame field.
void SetFrameReadyNotifier(FrameReadyNotifier notifier) override;
void ReturnFrame(std::shared_ptr<VideoFrame> frame) override;
void SetInitializeFramesHandler(InitializeFramesHandler handler) override;
void SetErrorHandler(fit::closure error_handler) override;
+ void SetCheckOutputReady(CheckOutputReady check_output_ready) override;
void InitializedFrames(std::vector<CodecFrame> frames, uint32_t width,
uint32_t height, uint32_t stride) override;
__WARN_UNUSED_RESULT bool CanBeSwappedIn() override;
@@ -177,7 +181,23 @@
// This is the count of references from reference_frame_map_, last_frame_,
// current_frame_, and any buffers the ultimate consumers have outstanding.
int32_t refcount = 0;
- // Allocated on demand.
+ // Each VideoFrame is managed via shared_ptr<> here and via weak_ptr<> in
+ // CodecBuffer. There is a frame.reset() performed under
+ // video_decoder_lock_ that essentially signals to the weak_ptr<> in
+ // CodecBuffer not to call ReturnFrame() any more for this frame. For this
+ // reason, under normal operation (not self-test), it's important that
+ // FrameReadyNotifier and weak_ptr<>::lock() not result in keeping any
+ // shared_ptr<> reference on VideoFrame that lasts beyond the current
+ // video_decoder_lock_ interval, since that could allow calling
+ // ReturnFrame() on a frame that the Vp9Decoder doesn't want to hear about
+ // any more.
+ //
+ // TODO(dustingreen): Mute ReturnFrame() a different way; maybe just
+ // explicitly. Ideally, we'd use a way that's more similar between decoder
+ // self-test and "normal operation".
+ //
+ // This shared_ptr<> must not actually be shared outside of while
+ // video_decoder_lock_ is held. See previous paragraphs.
std::shared_ptr<VideoFrame> frame;
// With the MMU enabled the compressed frame header is stored separately
// from the data itself, allowing the data to be allocated in noncontiguous
@@ -235,6 +255,7 @@
WorkingBuffers working_buffers_;
FrameReadyNotifier notifier_;
InitializeFramesHandler initialize_frames_handler_;
+ CheckOutputReady check_output_ready_;
fit::closure error_handler_;
DecoderState state_ = DecoderState::kSwappedOut;
@@ -244,7 +265,16 @@
std::unique_ptr<loop_filter_info_n> loop_filter_info_;
std::unique_ptr<loopfilter> loop_filter_;
std::unique_ptr<segmentation> segmentation_ = {};
+ // Waiting for an available frame buffer (with reference count 0).
bool waiting_for_empty_frames_ = false;
+ // Waiting for an available output packet, to avoid show_existing_frame
+ // potentially allowing too much queued output, as a show_existing_frame
+ // output frame doesn't use up a frame buffer - but it does use up an output
+ // packet. We don't directly track the output packets in the h264_decoder,
+ // but this bool corresponds to being out of output packets in
+ // codec_adapter_vp9. We re-try PrepareNewFrame() during ReturnFrame() even
+ // if no refcount on any Frame has reached 0
+ bool waiting_for_output_ready_ = false;
// This is the count of frames decoded since this object was created.
uint32_t decoded_frame_count_ = 0;
diff --git a/examples/media/use_media_decoder/codec_client.cc b/examples/media/use_media_decoder/codec_client.cc
index 1849002..859c2a5 100644
--- a/examples/media/use_media_decoder/codec_client.cc
+++ b/examples/media/use_media_decoder/codec_client.cc
@@ -282,6 +282,7 @@
fuchsia::mediacodec::CodecPacket::New();
packet->header.buffer_lifetime_ordinal = kInputBufferLifetimeOrdinal;
packet->header.packet_index = free_index;
+ packet->buffer_index = free_index;
return packet;
}
diff --git a/examples/media/use_media_decoder/use_video_decoder.cc b/examples/media/use_media_decoder/use_video_decoder.cc
index 8960778..b8a2706 100644
--- a/examples/media/use_media_decoder/use_video_decoder.cc
+++ b/examples/media/use_media_decoder/use_video_decoder.cc
@@ -122,6 +122,7 @@
while (bytes_so_far != byte_count) {
std::unique_ptr<fuchsia::mediacodec::CodecPacket> packet =
codec_client->BlockingGetFreeInputPacket();
+ // For input we do buffer_index == packet_index.
const CodecBuffer& buffer =
codec_client->GetInputBufferByIndex(packet->header.packet_index);
size_t bytes_to_copy =
@@ -201,6 +202,7 @@
uint32_t frame_pts) {
std::unique_ptr<fuchsia::mediacodec::CodecPacket> packet =
codec_client->BlockingGetFreeInputPacket();
+ // For input we do buffer_index == packet_index.
const CodecBuffer& buffer =
codec_client->GetInputBufferByIndex(packet->header.packet_index);
// VP9 decoder doesn't yet support splitting access units into multiple
@@ -404,7 +406,7 @@
// This will remain live long enough because this thread is the only
// thread that re-allocates output buffers.
const CodecBuffer& buffer =
- codec_client.GetOutputBufferByIndex(packet.header.packet_index);
+ codec_client.GetOutputBufferByIndex(packet.buffer_index);
if (stream_config &&
(config->format_details.format_details_version_ordinal !=
diff --git a/lib/media/codec_impl/codec_adapter.cc b/lib/media/codec_impl/codec_adapter.cc
index 945ba0b..d706feb 100644
--- a/lib/media/codec_impl/codec_adapter.cc
+++ b/lib/media/codec_impl/codec_adapter.cc
@@ -8,7 +8,10 @@
CodecAdapter::CodecAdapter(std::mutex& lock,
CodecAdapterEvents* codec_adapter_events)
- : lock_(lock), events_(codec_adapter_events) {
+ : lock_(lock),
+ events_(codec_adapter_events),
+ random_device_(),
+ not_for_security_prng_(random_device_()) {
ZX_DEBUG_ASSERT(events_);
// nothing else to do here
}
diff --git a/lib/media/codec_impl/codec_buffer.cc b/lib/media/codec_impl/codec_buffer.cc
index 908c621..5cf881a 100644
--- a/lib/media/codec_impl/codec_buffer.cc
+++ b/lib/media/codec_impl/codec_buffer.cc
@@ -65,3 +65,11 @@
const fuchsia::mediacodec::CodecBuffer& CodecBuffer::codec_buffer() const {
return buffer_;
}
+
+void CodecBuffer::SetVideoFrame(std::weak_ptr<VideoFrame> video_frame) const {
+ video_frame_ = video_frame;
+}
+
+std::weak_ptr<VideoFrame> CodecBuffer::video_frame() const {
+ return video_frame_;
+}
diff --git a/lib/media/codec_impl/codec_impl.cc b/lib/media/codec_impl/codec_impl.cc
index 120d071..e0a7eca 100644
--- a/lib/media/codec_impl/codec_impl.cc
+++ b/lib/media/codec_impl/codec_impl.cc
@@ -774,8 +774,13 @@
FailLocked("client QueueInputPacket() with packet_index out of range");
return;
}
+ if (packet.buffer_index >= all_buffers_[kInputPort].size()) {
+ FailLocked("client QueueInputPacket() with buffer_index out of range");
+ return;
+ }
- // Protocol check re. free/busy coherency.
+ // Protocol check re. free/busy coherency. This applies to packets only,
+ // not buffers.
if (!all_packets_[kInputPort][packet.header.packet_index]->is_free()) {
FailLocked("client QueueInputPacket() with packet_index !free");
return;
@@ -819,6 +824,8 @@
CodecPacket* core_codec_packet =
all_packets_[kInputPort][packet.header.packet_index].get();
+ core_codec_packet->SetBuffer(
+ all_buffers_[kInputPort][packet.buffer_index].get());
core_codec_packet->SetStartOffset(packet.start_offset);
core_codec_packet->SetValidLengthBytes(packet.valid_length_bytes);
if (packet.has_timestamp_ish) {
@@ -892,7 +899,7 @@
return;
}
std::unique_lock<std::mutex> lock(lock_);
- StartIgnoringClientOldOutputConfigLocked();
+ StartIgnoringClientOldOutputConfig(lock);
GenerateAndSendNewOutputConfig(lock, true);
}
@@ -1297,6 +1304,8 @@
CoreCodecAddBuffer(port, local_buffer.get());
all_buffers_[port].push_back(std::move(local_buffer));
if (all_buffers_[port].size() == required_buffer_count) {
+ ZX_DEBUG_ASSERT(buffer_lifetime_ordinal_[port] ==
+ port_settings_[port]->buffer_lifetime_ordinal);
// Stash this while we can, before the client de-configures.
last_provided_buffer_constraints_version_ordinal_[port] =
port_settings_[port]->buffer_constraints_version_ordinal;
@@ -1305,16 +1314,11 @@
uint32_t packet_count =
PacketCountFromPortSettings(*port_settings_[port]);
for (uint32_t i = 0; i < packet_count; i++) {
- uint32_t buffer_index = required_buffer_count == 1 ? 0 : i;
- CodecBuffer* buffer = all_buffers_[port][buffer_index].get();
- ZX_DEBUG_ASSERT(buffer_lifetime_ordinal_[port] ==
- port_settings_[port]->buffer_lifetime_ordinal);
// Private constructor to prevent core codec maybe creating its own
// Packet instances (which isn't the intent) seems worth the hassle of
// not using make_unique<>() here.
- all_packets_[port].push_back(
- std::unique_ptr<CodecPacket>(new CodecPacket(
- port_settings_[port]->buffer_lifetime_ordinal, i, buffer)));
+ all_packets_[port].push_back(std::unique_ptr<CodecPacket>(
+ new CodecPacket(port_settings_[port]->buffer_lifetime_ordinal, i)));
}
{ // scope unlock
@@ -1510,7 +1514,7 @@
}
if (is_new_config_needed) {
- StartIgnoringClientOldOutputConfigLocked();
+ StartIgnoringClientOldOutputConfig(lock);
EnsureBuffersNotConfigured(lock, kOutputPort);
// This does count as a mid-stream output config change, even when this is
// at the start of a stream - it's still while a stream is active, and still
@@ -1713,7 +1717,8 @@
// the client's messages re. the old buffer_lifetime_ordinal and old
// buffer_constraints_version_ordinal until the client catches up to the new
// last_required_buffer_constraints_version_ordinal_[kOutputPort].
-void CodecImpl::StartIgnoringClientOldOutputConfigLocked() {
+void CodecImpl::StartIgnoringClientOldOutputConfig(
+ std::unique_lock<std::mutex>& lock) {
ZX_DEBUG_ASSERT(thrd_current() == stream_control_thread_);
// The buffer_lifetime_ordinal_[kOutputPort] can be even on entry due to at
@@ -1734,6 +1739,32 @@
// next_output_buffer_constraints_version_ordinal_ in that method.
last_required_buffer_constraints_version_ordinal_[kOutputPort] =
next_output_buffer_constraints_version_ordinal_;
+
+ // Now that we've stopped any new calls to CoreCodecRecycleOutputPacket(),
+ // fence through any previously-started call to CoreCodecRecycleOutputPacket()
+ // that maybe have been started previously, before returning from this method.
+ //
+ // We can't just be holding lock_ during the call to
+ // CoreCodecRecycleOutputPacket() because it acquires the video_decoder_lock_
+ // and in other paths the video_decoder_lock_ is held while acquiring lock_.
+ //
+ // It's ok for the StreamControl domain to wait on the Output domain (but not
+ // the other way around).
+ bool is_output_ordering_domain_done_with_recycle_output_packet = false;
+ std::condition_variable condition_changed;
+ PostToSharedFidl([this,
+ &is_output_ordering_domain_done_with_recycle_output_packet,
+ &condition_changed] {
+ { // scope lock
+ std::lock_guard<std::mutex> lock(lock_);
+ is_output_ordering_domain_done_with_recycle_output_packet = true;
+ }
+ condition_changed.notify_all();
+ });
+ while (!is_output_ordering_domain_done_with_recycle_output_packet) {
+ condition_changed.wait(lock);
+ }
+ ZX_DEBUG_ASSERT(is_output_ordering_domain_done_with_recycle_output_packet);
}
void CodecImpl::GenerateAndSendNewOutputConfig(
@@ -1755,7 +1786,7 @@
// If buffer_constraints_action_required true, the caller bumped the
// last_required_buffer_constraints_version_ordinal_[kOutputPort] before
- // calling this method (using StartIgnoringClientOldOutputConfigLocked()), to
+ // calling this method (using StartIgnoringClientOldOutputConfig()), to
// ensure any output config messages from the client are ignored until the
// client catches up to at least
// last_required_buffer_constraints_version_ordinal_.
@@ -1847,7 +1878,7 @@
// OmxTryRecycleOutputPacketLocked() won't call OMX, and the interval during
// which we'll ingore any in-progress client output config until the client
// catches up.
- StartIgnoringClientOldOutputConfigLocked();
+ StartIgnoringClientOldOutputConfig(lock);
{ // scope unlock
ScopedUnlock unlock(lock);
@@ -2197,11 +2228,15 @@
});
}
-void CodecImpl::onCoreCodecInputPacketDone(const CodecPacket* packet) {
+void CodecImpl::onCoreCodecInputPacketDone(CodecPacket* packet) {
// Free/busy coherency from Codec interface to OMX doesn't involve trusting
// the client, so assert we're doing it right server-side.
{ // scope lock
std::unique_lock<std::mutex> lock(lock_);
+ // The core codec says the buffer-referening in-flight lifetime of this
+ // packet is over. We'll set the buffer again when this packet get's used
+ // by the client again to deliver more input data.
+ packet->SetBuffer(nullptr);
// Unfortunately we have to insist that the core codec not call
// onCoreCodecInputPacketDone() arbitrarily late because we need to know
// when it's safe to deallocate binding_, and the core codec, etc. So the
@@ -2223,6 +2258,13 @@
bool error_detected_during) {
{ // scope lock
std::unique_lock<std::mutex> lock(lock_);
+ // This helps verify that packet lifetimes are coherent, but we can't do the
+ // same for buffer_index because VP9 has show_existing_frame which is
+ // allowed to output the same buffer repeatedly.
+ //
+ // TODO(dustingreen): We could _optionally_ verify that buffer lifetimes are
+ // coherent for codecs that don't output the same buffer repeatedly and
+ // concurrently.
all_packets_[kOutputPort][packet->packet_index()]->SetFree(false);
ZX_DEBUG_ASSERT(packet->has_start_offset());
ZX_DEBUG_ASSERT(packet->has_valid_length_bytes());
@@ -2241,6 +2283,7 @@
.header.buffer_lifetime_ordinal =
packet->buffer_lifetime_ordinal(),
.header.packet_index = packet->packet_index(),
+ .buffer_index = packet->buffer()->buffer_index(),
.stream_lifetime_ordinal = stream_lifetime_ordinal_,
.start_offset = packet->start_offset(),
.valid_length_bytes = packet->valid_length_bytes(),
@@ -2401,7 +2444,7 @@
per_stream_override_format_details);
}
-void CodecImpl::CoreCodecQueueInputPacket(const CodecPacket* packet) {
+void CodecImpl::CoreCodecQueueInputPacket(CodecPacket* packet) {
ZX_DEBUG_ASSERT(thrd_current() == stream_control_thread_);
codec_adapter_->CoreCodecQueueInputPacket(packet);
}
diff --git a/lib/media/codec_impl/codec_input_item.cc b/lib/media/codec_impl/codec_input_item.cc
index a361411..771ba22 100644
--- a/lib/media/codec_impl/codec_input_item.cc
+++ b/lib/media/codec_impl/codec_input_item.cc
@@ -19,7 +19,7 @@
// nothing else to do here
}
-CodecInputItem::CodecInputItem(const CodecPacket* packet) : packet_(packet) {
+CodecInputItem::CodecInputItem(CodecPacket* packet) : packet_(packet) {
// nothing else do to here
}
@@ -31,7 +31,7 @@
return CodecInputItem(format_details);
}
-CodecInputItem CodecInputItem::Packet(const CodecPacket* packet) {
+CodecInputItem CodecInputItem::Packet(CodecPacket* packet) {
ZX_DEBUG_ASSERT(packet);
return CodecInputItem(packet);
}
@@ -56,7 +56,7 @@
return *format_details_;
}
-const CodecPacket* CodecInputItem::packet() const {
+CodecPacket* CodecInputItem::packet() const {
ZX_DEBUG_ASSERT(is_packet());
return packet_;
}
diff --git a/lib/media/codec_impl/codec_packet.cc b/lib/media/codec_impl/codec_packet.cc
index 48fed88..b68aa27 100644
--- a/lib/media/codec_impl/codec_packet.cc
+++ b/lib/media/codec_impl/codec_packet.cc
@@ -9,10 +9,9 @@
#include <stdint.h>
CodecPacket::CodecPacket(uint64_t buffer_lifetime_ordinal,
- uint32_t packet_index, CodecBuffer* buffer)
+ uint32_t packet_index)
: buffer_lifetime_ordinal_(buffer_lifetime_ordinal),
- packet_index_(packet_index),
- buffer_(buffer) {
+ packet_index_(packet_index) {
// nothing else to do here
}
@@ -26,7 +25,9 @@
uint32_t CodecPacket::packet_index() const { return packet_index_; }
-const CodecBuffer& CodecPacket::buffer() const { return *buffer_; }
+void CodecPacket::SetBuffer(const CodecBuffer* buffer) { buffer_ = buffer; }
+
+const CodecBuffer* CodecPacket::buffer() const { return buffer_; }
void CodecPacket::SetStartOffset(uint32_t start_offset) {
start_offset_ = start_offset;
@@ -81,14 +82,6 @@
bool CodecPacket::is_new() const { return is_new_; }
-void CodecPacket::SetVideoFrame(std::weak_ptr<VideoFrame> video_frame) {
- video_frame_ = video_frame;
-}
-
-std::weak_ptr<VideoFrame> CodecPacket::video_frame() const {
- return video_frame_;
-}
-
void CodecPacket::ClearStartOffset() { start_offset_ = kStartOffsetNotSet; }
void CodecPacket::ClearValidLengthBytes() {
diff --git a/lib/media/codec_impl/include/lib/media/codec_impl/codec_adapter.h b/lib/media/codec_impl/include/lib/media/codec_impl/codec_adapter.h
index 82bc1f9..1ba46a8 100644
--- a/lib/media/codec_impl/include/lib/media/codec_impl/codec_adapter.h
+++ b/lib/media/codec_impl/include/lib/media/codec_impl/codec_adapter.h
@@ -13,6 +13,7 @@
#include <fuchsia/mediacodec/cpp/fidl.h>
#include <list>
+#include <random>
class CodecBuffer;
@@ -136,7 +137,7 @@
per_stream_override_format_details) = 0;
// Only permitted between CoreCodecStartStream() and CoreCodecStopStream().
- virtual void CoreCodecQueueInputPacket(const CodecPacket* packet) = 0;
+ virtual void CoreCodecQueueInputPacket(CodecPacket* packet) = 0;
// Only permitted between CoreCodecStartStream() and CoreCodecStopStream().
virtual void CoreCodecQueueInputEndOfStream() = 0;
@@ -276,6 +277,13 @@
// A core codec will also want to track free output packets, but how best to
// do that is sub-class-specific.
+ // It's useful to have a source of random numbers that's compatible with std::
+ // for purposes such as scrambling the order of free packets. These are
+ // instance-specific only because of thread-safety considerations, not because
+ // of generated sequence considerations.
+ std::random_device random_device_;
+ std::mt19937 not_for_security_prng_;
+
private:
CodecAdapter() = delete;
DISALLOW_COPY_ASSIGN_AND_MOVE(CodecAdapter);
diff --git a/lib/media/codec_impl/include/lib/media/codec_impl/codec_adapter_events.h b/lib/media/codec_impl/include/lib/media/codec_impl/codec_adapter_events.h
index 4af7fbd..3b916ca 100644
--- a/lib/media/codec_impl/include/lib/media/codec_impl/codec_adapter_events.h
+++ b/lib/media/codec_impl/include/lib/media/codec_impl/codec_adapter_events.h
@@ -39,7 +39,7 @@
virtual void onCoreCodecMidStreamOutputConfigChange(
bool output_re_config_required) = 0;
- virtual void onCoreCodecInputPacketDone(const CodecPacket* packet) = 0;
+ virtual void onCoreCodecInputPacketDone(CodecPacket* packet) = 0;
virtual void onCoreCodecOutputPacket(CodecPacket* packet,
bool error_detected_before,
diff --git a/lib/media/codec_impl/include/lib/media/codec_impl/codec_buffer.h b/lib/media/codec_impl/include/lib/media/codec_impl/codec_buffer.h
index bcf8850..5ef861a 100644
--- a/lib/media/codec_impl/include/lib/media/codec_impl/codec_buffer.h
+++ b/lib/media/codec_impl/include/lib/media/codec_impl/codec_buffer.h
@@ -14,6 +14,17 @@
class CodecImpl;
+// Core codec representation of a video frame. Different core codecs may have
+// very different implementations of this.
+//
+// TODO(dustingreen): Have this be a base class that's defined by the
+// CodecImpl source_set, and have amlogic-video VideoFrame derive from that base
+// class.
+//
+// Regardless of codec, these will be managed by shared_ptr<>, because for
+// decoder reference frames, shared_ptr<> makes sense.
+struct VideoFrame;
+
// TODO(dustingreen): Support BufferCollection buffers.
//
// These are 1:1 with Codec buffers, but not necessarily 1:1 with core codec
@@ -33,6 +44,17 @@
const fuchsia::mediacodec::CodecBuffer& codec_buffer() const;
+ // The use of weak_ptr<> here is to emphasize that we don't need shared_ptr<>
+ // to keep the VideoFrame(s) alive. We'd use a raw pointer here if it weren't
+ // for needing to convert to a shared_ptr<> to call certain methods that
+ // expect shared_ptr<>.
+ //
+ // This is marked const because it only mutates a mutable field, which is
+ // considered mutable because it's about establishing an association between
+ // video_frame and CodecBuffer after CodecBuffer has been constructed.
+ void SetVideoFrame(std::weak_ptr<VideoFrame> video_frame) const;
+ std::weak_ptr<VideoFrame> video_frame() const;
+
private:
friend class CodecImpl;
friend class std::unique_ptr<CodecBuffer>;
@@ -52,6 +74,10 @@
// This msg still has the live vmo_handle.
fuchsia::mediacodec::CodecBuffer buffer_;
+ // Mutable only in the sense that it's set later than the constructor. The
+ // association does not switch to a different VideoFrame once set.
+ mutable std::weak_ptr<VideoFrame> video_frame_;
+
// This accounts for vmo_offset_begin. The content bytes are not part of
// a Buffer instance from a const-ness point of view.
uint8_t* buffer_base_ = nullptr;
diff --git a/lib/media/codec_impl/include/lib/media/codec_impl/codec_frame.h b/lib/media/codec_impl/include/lib/media/codec_impl/codec_frame.h
index 5805fa8..761c5d7 100644
--- a/lib/media/codec_impl/include/lib/media/codec_impl/codec_frame.h
+++ b/lib/media/codec_impl/include/lib/media/codec_impl/codec_frame.h
@@ -7,14 +7,14 @@
#include <fuchsia/mediacodec/cpp/fidl.h>
-class CodecPacket;
+class CodecBuffer;
// Move-only struct
struct CodecFrame {
CodecFrame(CodecFrame&& from) = default;
CodecFrame& operator=(CodecFrame&& from) = default;
- fuchsia::mediacodec::CodecBuffer codec_buffer;
- CodecPacket* codec_packet;
+ fuchsia::mediacodec::CodecBuffer codec_buffer_spec;
+ const CodecBuffer* codec_buffer_ptr;
};
#endif // GARNET_LIB_MEDIA_CODEC_IMPL_INCLUDE_LIB_MEDIA_CODEC_IMPL_CODEC_FRAME_H_
diff --git a/lib/media/codec_impl/include/lib/media/codec_impl/codec_impl.h b/lib/media/codec_impl/include/lib/media/codec_impl/codec_impl.h
index b9352f5..cdd6805 100644
--- a/lib/media/codec_impl/include/lib/media/codec_impl/codec_impl.h
+++ b/lib/media/codec_impl/include/lib/media/codec_impl/codec_impl.h
@@ -558,7 +558,7 @@
__WARN_UNUSED_RESULT bool EnsureFutureStreamFlushSeenLocked(
uint64_t stream_lifetime_ordinal);
- void StartIgnoringClientOldOutputConfigLocked();
+ void StartIgnoringClientOldOutputConfig(std::unique_lock<std::mutex>& lock);
void GenerateAndSendNewOutputConfig(std::unique_lock<std::mutex>& lock,
bool buffer_constraints_action_required);
@@ -680,7 +680,7 @@
void onCoreCodecMidStreamOutputConfigChange(
bool output_re_config_required) override;
- void onCoreCodecInputPacketDone(const CodecPacket* packet) override;
+ void onCoreCodecInputPacketDone(CodecPacket* packet) override;
void onCoreCodecOutputPacket(CodecPacket* packet, bool error_detected_before,
bool error_detected_during) override;
@@ -706,7 +706,7 @@
const fuchsia::mediacodec::CodecFormatDetails&
per_stream_override_format_details) override;
- void CoreCodecQueueInputPacket(const CodecPacket* packet) override;
+ void CoreCodecQueueInputPacket(CodecPacket* packet) override;
void CoreCodecQueueInputEndOfStream() override;
diff --git a/lib/media/codec_impl/include/lib/media/codec_impl/codec_input_item.h b/lib/media/codec_impl/include/lib/media/codec_impl/codec_input_item.h
index 2dbf077..14c6d82 100644
--- a/lib/media/codec_impl/include/lib/media/codec_impl/codec_input_item.h
+++ b/lib/media/codec_impl/include/lib/media/codec_impl/codec_input_item.h
@@ -23,7 +23,7 @@
static CodecInputItem Invalid();
static CodecInputItem FormatDetails(
const fuchsia::mediacodec::CodecFormatDetails& format_details);
- static CodecInputItem Packet(const CodecPacket* packet);
+ static CodecInputItem Packet(CodecPacket* packet);
static CodecInputItem EndOfStream();
bool is_valid() const;
@@ -32,21 +32,21 @@
bool is_end_of_stream() const;
const fuchsia::mediacodec::CodecFormatDetails& format_details();
- const CodecPacket* packet() const;
+ CodecPacket* packet() const;
private:
// !is_valid()
CodecInputItem();
explicit CodecInputItem(
const fuchsia::mediacodec::CodecFormatDetails& format_details);
- explicit CodecInputItem(const CodecPacket* packet);
+ explicit CodecInputItem(CodecPacket* packet);
// The fields of this class are relied upon to be move-able.
bool is_valid_ = true;
std::unique_ptr<fuchsia::mediacodec::CodecFormatDetails> format_details_;
// If nullptr, is_end_of_stream() is true.
- const CodecPacket* packet_ = nullptr;
+ CodecPacket* packet_ = nullptr;
// Lack of format_details_ and lack of packet_ means is_end_of_stream().
};
diff --git a/lib/media/codec_impl/include/lib/media/codec_impl/codec_packet.h b/lib/media/codec_impl/include/lib/media/codec_impl/codec_packet.h
index 1b00b37..0123191 100644
--- a/lib/media/codec_impl/include/lib/media/codec_impl/codec_packet.h
+++ b/lib/media/codec_impl/include/lib/media/codec_impl/codec_packet.h
@@ -15,14 +15,6 @@
class CodecBuffer;
-// Core codec representation of a video frame. Different core codecs may have
-// very different implementations of this.
-//
-// TODO(dustingreen): Have this be a base class that's defined by the
-// CodecImpl source_set, and have amlogic-video VideoFrame derive from that base
-// class.
-struct VideoFrame;
-
// Instances of this class are 1:1 with fuchsia::mediacodec::CodecPacket.
class CodecPacket {
public:
@@ -32,7 +24,8 @@
uint32_t packet_index() const;
- const CodecBuffer& buffer() const;
+ void SetBuffer(const CodecBuffer* buffer);
+ const CodecBuffer* buffer() const;
void SetStartOffset(uint32_t start_offset);
bool has_start_offset() const;
@@ -55,13 +48,6 @@
void SetIsNew(bool is_new);
bool is_new() const;
- // The use of weak_ptr<> here is to emphasize that we don't need shared_ptr<>
- // to keep the VideoFrame(s) alive. We'd use a raw pointer here if it weren't
- // for needing to convert to a shared_ptr<> to call certain methods that
- // expect shared_ptr<>.
- void SetVideoFrame(std::weak_ptr<VideoFrame> video_frame);
- std::weak_ptr<VideoFrame> video_frame() const;
-
private:
// The public section is for the core codec to call - the private section is
// only for CodecImpl to call.
@@ -74,8 +60,7 @@
// The buffer ptr is not owned. The buffer lifetime is slightly longer than
// the Packet lifetime.
- CodecPacket(uint64_t buffer_lifetime_ordinal, uint32_t packet_index,
- CodecBuffer* buffer);
+ CodecPacket(uint64_t buffer_lifetime_ordinal, uint32_t packet_index);
void ClearStartOffset();
void ClearValidLengthBytes();
@@ -83,8 +68,9 @@
uint64_t buffer_lifetime_ordinal_ = 0;
uint32_t packet_index_ = 0;
- // not owned
- CodecBuffer* buffer_ = nullptr;
+ // The buffer_ is meaningful only while a packet_index is in-flight, not
+ // while the packet_index is free.
+ const CodecBuffer* buffer_ = nullptr;
uint32_t start_offset_ = kStartOffsetNotSet;
uint32_t valid_length_bytes_ = kValidLengthBytesNotSet;
@@ -117,8 +103,6 @@
// call or equivalent for new packets (OMX), and some don't (amlogic-video).
bool is_new_ = true;
- std::weak_ptr<VideoFrame> video_frame_;
-
CodecPacket() = delete;
DISALLOW_COPY_ASSIGN_AND_MOVE(CodecPacket);
};
diff --git a/public/fidl/fuchsia.mediacodec/codec.fidl b/public/fidl/fuchsia.mediacodec/codec.fidl
index 8365992..e5f2bcc 100644
--- a/public/fidl/fuchsia.mediacodec/codec.fidl
+++ b/public/fidl/fuchsia.mediacodec/codec.fidl
@@ -51,188 +51,188 @@
//
// There are separate instances of this struct for codec input and codec output.
struct CodecBufferConstraints {
- // This is a version number the server sets on the constraints to allow the
- // server to determine when the client has caught up with the latest
- // constraints sent by the server. The server won't emit output data until
- // the client has configured output settings and buffers with a
- // buffer_constraints_version_ordinal >= the latest
- // buffer_constraints_version_ordinal that had
- // buffer_constraints_action_required true. See
- // buffer_constraints_action_required comments for more.
- //
- // A buffer_constraints_version_ordinal of 0 is not permitted, to simplify
- // initial state handling. Other than 0, both odd and even version ordinals
- // are allowed (in constrast to the stream_lifetime_ordinal, neither the
- // client nor server ever has a reason to consider the latest version to be
- // stale, so there would be no benefit to disallowing even values).
- uint64 buffer_constraints_version_ordinal;
+ // This is a version number the server sets on the constraints to allow the
+ // server to determine when the client has caught up with the latest
+ // constraints sent by the server. The server won't emit output data until
+ // the client has configured output settings and buffers with a
+ // buffer_constraints_version_ordinal >= the latest
+ // buffer_constraints_version_ordinal that had
+ // buffer_constraints_action_required true. See
+ // buffer_constraints_action_required comments for more.
+ //
+ // A buffer_constraints_version_ordinal of 0 is not permitted, to simplify
+ // initial state handling. Other than 0, both odd and even version ordinals
+ // are allowed (in constrast to the stream_lifetime_ordinal, neither the
+ // client nor server ever has a reason to consider the latest version to be
+ // stale, so there would be no benefit to disallowing even values).
+ uint64 buffer_constraints_version_ordinal;
- // default_settings
- //
- // These settings are "default" settings, not "recommended" settings.
- //
- // These "default" settings can be passed to SetInputBufferSettings() /
- // SetOutputBufferSettings() as-is without modification, but a client doing
- // that must still obey the semantics of packet_count_for_client, despite the
- // codec server not having any way to really know the proper setting for
- // that field.
- //
- // For CodecBufferConstraints fields whose names end in "recommended", the
- // default_settings will have the corresponding setting field set to that
- // recommended value.
- //
- // The codec promises that these default settings as-is (except for
- // buffer_lifetime_ordinal) are guaranteed to
- // satisfy the constraints indicated by the other fields of
- // CodecBufferConstraints. While client-side checking that these
- // settings are within the constraints is likely unnecessary in the client,
- // the client should still check that these values are within client-side
- // reasonable-ness bounds before using these values, to avoid letting a codec
- // server cause problems for the client.
- //
- // This structure will always have single_buffer_mode false. See
- // single_buffer_mode_allowed for whether single_buffer_mode true is allowed.
- //
- // The client must set the buffer_lifetime_ordinal field to a proper value
- // before sending back to the server. The 0 initially in this field will be
- // rejected by the server if sent back as-is. See comments on
- // CodecPortBufferSettings.buffer_lifetime_ordinal.
- CodecPortBufferSettings default_settings;
+ // default_settings
+ //
+ // These settings are "default" settings, not "recommended" settings.
+ //
+ // These "default" settings can be passed to SetInputBufferSettings() /
+ // SetOutputBufferSettings() as-is without modification, but a client doing
+ // that must still obey the semantics of packet_count_for_client, despite the
+ // codec server not having any way to really know the proper setting for
+ // that field.
+ //
+ // For CodecBufferConstraints fields whose names end in "recommended", the
+ // default_settings will have the corresponding setting field set to that
+ // recommended value.
+ //
+ // The codec promises that these default settings as-is (except for
+ // buffer_lifetime_ordinal) are guaranteed to
+ // satisfy the constraints indicated by the other fields of
+ // CodecBufferConstraints. While client-side checking that these
+ // settings are within the constraints is likely unnecessary in the client,
+ // the client should still check that these values are within client-side
+ // reasonable-ness bounds before using these values, to avoid letting a codec
+ // server cause problems for the client.
+ //
+ // This structure will always have single_buffer_mode false. See
+ // single_buffer_mode_allowed for whether single_buffer_mode true is allowed.
+ //
+ // The client must set the buffer_lifetime_ordinal field to a proper value
+ // before sending back to the server. The 0 initially in this field will be
+ // rejected by the server if sent back as-is. See comments on
+ // CodecPortBufferSettings.buffer_lifetime_ordinal.
+ CodecPortBufferSettings default_settings;
- // For uncompresesd video, separate and complete frames, each in its own
- // separate buffer (buffer-per-packet mode), is always a requirement.
+ // For uncompresesd video, separate and complete frames, each in its own
+ // separate buffer (buffer-per-packet mode), is always a requirement.
- // per_packet_buffer_bytes.*:
- //
- // These per-packet buffer bytes constraints apply to both buffer-per-packet
- // mode and single-buffer mode (see single_buffer_mode). If
- // buffer-per-packet mode, the constraints apply to each buffer separately.
- // If single-buffer mode, the constraints need to be multiplied by the number
- // of packets to determine the constraints on the single buffer.
+ // per_packet_buffer_bytes.*:
+ //
+ // These per-packet buffer bytes constraints apply to both buffer-per-packet
+ // mode and single-buffer mode (see single_buffer_mode). If
+ // buffer-per-packet mode, the constraints apply to each buffer separately.
+ // If single-buffer mode, the constraints need to be multiplied by the number
+ // of packets to determine the constraints on the single buffer.
- // per_packet_buffer_bytes_min:
- //
- // If a client is using buffer per packet mode, each buffer must be at least
- // this large. If a client is using single-buffer mode, the one buffer must
- // be at least per_packet_buffer_bytes_min * packet_count_for_codec_min in
- // size.
- uint32 per_packet_buffer_bytes_min;
- // Must be >= per_packet_buffer_bytes_min. Delivering more than
- // this per input packet might not perform any better, and in fact might
- // perform worse.
- uint32 per_packet_buffer_bytes_recommended;
- // Must be >= per_packet_buffer_bytes_recommended. Can be 0xFFFFFFFF if there
- // is no explicitly-enforced limit.
- uint32 per_packet_buffer_bytes_max;
+ // per_packet_buffer_bytes_min:
+ //
+ // If a client is using buffer per packet mode, each buffer must be at least
+ // this large. If a client is using single-buffer mode, the one buffer must
+ // be at least per_packet_buffer_bytes_min * packet_count_for_codec_min in
+ // size.
+ uint32 per_packet_buffer_bytes_min;
+ // Must be >= per_packet_buffer_bytes_min. Delivering more than
+ // this per input packet might not perform any better, and in fact might
+ // perform worse.
+ uint32 per_packet_buffer_bytes_recommended;
+ // Must be >= per_packet_buffer_bytes_recommended. Can be 0xFFFFFFFF if there
+ // is no explicitly-enforced limit.
+ uint32 per_packet_buffer_bytes_max;
- // Minimum number of packet_count_for_codec.
- //
- // Re. input and output:
- //
- // This is a strict min for packet_count_for_codec, but a client can use more
- // packets overall if the client wants to, by using a larger value for
- // packet_count_for_codec and/or using a non-zero packets_for_client. A good
- // reason to do the former would be if the client might tend to deliver a few
- // not-very-full buffers occasionally - or to have a few extra packets within
- // which to satisfy codec_input_bytes_min. A good reason to do the latter
- // would be if a client needs to hold onto some packets for any "extra"
- // duration.
- //
- // If a client specifies a larger packet_count_for_codec value than
- // packet_count_for_codec_min, a server is permitted (but not encouraged) to
- // not make progress until packet_count_for_codec are with the server,
- // not merely packet_count_for_codec_min.
- //
- // For decoder input and audio encoder input: The packet_count_for_codec_min
- // may or may not contain enough data to allow the codec to make progress
- // without copying into an internal side buffer. If there isn't enough data
- // delivered in packet_count_for_codec_min packets to permit progress, the
- // codec must copy into its own side buffer internally to make progress.
- //
- // If a client intends to use extra packets for client-side purposes, the
- // client should specify the extra packets in packets_for_client instead of
- // packet_count_for_codec, but packet_count_for_codec must still be >=
- // packet_count_for_codec_min.
- //
- // See codec.md for more on packet_count_for_codec_min.
- uint32 packet_count_for_codec_min;
+ // Minimum number of packet_count_for_codec.
+ //
+ // Re. input and output:
+ //
+ // This is a strict min for packet_count_for_codec, but a client can use more
+ // packets overall if the client wants to, by using a larger value for
+ // packet_count_for_codec and/or using a non-zero packets_for_client. A good
+ // reason to do the former would be if the client might tend to deliver a few
+ // not-very-full buffers occasionally - or to have a few extra packets within
+ // which to satisfy codec_input_bytes_min. A good reason to do the latter
+ // would be if a client needs to hold onto some packets for any "extra"
+ // duration.
+ //
+ // If a client specifies a larger packet_count_for_codec value than
+ // packet_count_for_codec_min, a server is permitted (but not encouraged) to
+ // not make progress until packet_count_for_codec are with the server,
+ // not merely packet_count_for_codec_min.
+ //
+ // For decoder input and audio encoder input: The packet_count_for_codec_min
+ // may or may not contain enough data to allow the codec to make progress
+ // without copying into an internal side buffer. If there isn't enough data
+ // delivered in packet_count_for_codec_min packets to permit progress, the
+ // codec must copy into its own side buffer internally to make progress.
+ //
+ // If a client intends to use extra packets for client-side purposes, the
+ // client should specify the extra packets in packets_for_client instead of
+ // packet_count_for_codec, but packet_count_for_codec must still be >=
+ // packet_count_for_codec_min.
+ //
+ // See codec.md for more on packet_count_for_codec_min.
+ uint32 packet_count_for_codec_min;
- // This must be at least packet_count_for_codec_min and at most
- // packet_count_for_codec_recommended_max.
- //
- // This value is likely to be used as-is by most clients, so if having one
- // additional packet is a big performance win in a large percentage of
- // scenarios, it can be good for the server to include that additional packet
- // in this value.
- uint32 packet_count_for_codec_recommended;
+ // This must be at least packet_count_for_codec_min and at most
+ // packet_count_for_codec_recommended_max.
+ //
+ // This value is likely to be used as-is by most clients, so if having one
+ // additional packet is a big performance win in a large percentage of
+ // scenarios, it can be good for the server to include that additional packet
+ // in this value.
+ uint32 packet_count_for_codec_recommended;
- // This can be the same as packet_count_for_codec_max or can be lower.
- // Values above this value and <= packet_count_for_codec_max are not
- // recommended by the codec, but should still work given sufficient resoures
- // available to both the client and the codec.
- uint32 packet_count_for_codec_recommended_max;
+ // This can be the same as packet_count_for_codec_max or can be lower.
+ // Values above this value and <= packet_count_for_codec_max are not
+ // recommended by the codec, but should still work given sufficient resoures
+ // available to both the client and the codec.
+ uint32 packet_count_for_codec_recommended_max;
- // This can be 0xFFFFFFFF if there's no codec-enforced max, but codecs are
- // encouraged to set a large but still plausibly-workable max, and clients
- // are encouraged to request a number of packets that isn't excessively large
- // for the client's scenario.
- uint32 packet_count_for_codec_max;
+ // This can be 0xFFFFFFFF if there's no codec-enforced max, but codecs are
+ // encouraged to set a large but still plausibly-workable max, and clients
+ // are encouraged to request a number of packets that isn't excessively large
+ // for the client's scenario.
+ uint32 packet_count_for_codec_max;
- // Normally this would be an implicit 0, but for now we have a min so we can
- // force the total number of packets to be a specific number that we know
- // works for the moment.
- uint32 packet_count_for_client_min;
+ // Normally this would be an implicit 0, but for now we have a min so we can
+ // force the total number of packets to be a specific number that we know
+ // works for the moment.
+ uint32 packet_count_for_client_min;
- // packet_count_for_client_max
- //
- // The client must set packet_count_for_client to be <=
- // packet_count_for_client_max.
- //
- // This value must be at least 1. This can be 0xFFFFFFFF if there's no
- // codec-enforced max. Clients are encouraged to request a number of
- // packets that isn't excessively large for the client's scenario.
- uint32 packet_count_for_client_max;
+ // packet_count_for_client_max
+ //
+ // The client must set packet_count_for_client to be <=
+ // packet_count_for_client_max.
+ //
+ // This value must be at least 1. This can be 0xFFFFFFFF if there's no
+ // codec-enforced max. Clients are encouraged to request a number of
+ // packets that isn't excessively large for the client's scenario.
+ uint32 packet_count_for_client_max;
- // single_buffer_mode_allowed false allows a codec that's not required to
- // support single-buffer mode for a given input or output the ability to
- // decline to support single-buffer mode on that input/output.
- //
- // All encoder output, regardless of audio or video: server support for
- // single-buffer mode is optional. Please see codec.md for all the rules
- // regarding single-buffer mode on output before using single-buffer mode on
- // an output.
- //
- // Audio decoder output: server support for single-buffer mode is required.
- //
- // Video decoder output: There is little reason for a video decoder to
- // support single-buffer mode on output. Nearly all video decoders will set
- // this to false for their output.
- //
- // All decoder inputs: Servers must support single-buffer mode on input. The
- // client is responsible for managing the input buffer space such that
- // filling an input packet doesn't over-write any portion of an input packet
- // already in flight to the codec.
- //
- // Encoder inputs: Server support for single-buffer mode on encoder input is
- // optional. This is more often useful for audio than for video.
- //
- // Support for buffer-per-packet mode is always required on both input and
- // output, regardless of codec type.
- bool single_buffer_mode_allowed;
+ // single_buffer_mode_allowed false allows a codec that's not required to
+ // support single-buffer mode for a given input or output the ability to
+ // decline to support single-buffer mode on that input/output.
+ //
+ // All encoder output, regardless of audio or video: server support for
+ // single-buffer mode is optional. Please see codec.md for all the rules
+ // regarding single-buffer mode on output before using single-buffer mode on
+ // an output.
+ //
+ // Audio decoder output: server support for single-buffer mode is required.
+ //
+ // Video decoder output: There is little reason for a video decoder to
+ // support single-buffer mode on output. Nearly all video decoders will set
+ // this to false for their output.
+ //
+ // All decoder inputs: Servers must support single-buffer mode on input. The
+ // client is responsible for managing the input buffer space such that
+ // filling an input packet doesn't over-write any portion of an input packet
+ // already in flight to the codec.
+ //
+ // Encoder inputs: Server support for single-buffer mode on encoder input is
+ // optional. This is more often useful for audio than for video.
+ //
+ // Support for buffer-per-packet mode is always required on both input and
+ // output, regardless of codec type.
+ bool single_buffer_mode_allowed;
- // If true, the buffers need to be physically contiguous pages, such as can be
- // allocated using zx_vmo_create_contiguous().
- bool is_physically_contiguous_required;
- // VERY TEMPORARY HACK / KLUDGE - we want the BufferAllocator (or one of the
- // participant drivers that needs physically contiguous buffers) to call
- // zx_vmo_create_contiguous(), definitely not the Codec client, but until the
- // BufferAllocator can be reached from a driver, this is to grant the client
- // special powers it really shouldn't have, very temporarily until
- // BufferAllocator is hooked up properly at which point this can be removed.
- // Strictly speaking we could reverse which end allocates buffers in the Codec
- // interface to avoid this hack even before BufferAllocator, but the overall
- // path seems shorter if we jump directly from this to using BufferAllocator.
- handle? very_temp_kludge_bti_handle;
+ // If true, the buffers need to be physically contiguous pages, such as can be
+ // allocated using zx_vmo_create_contiguous().
+ bool is_physically_contiguous_required;
+ // VERY TEMPORARY HACK / KLUDGE - we want the BufferAllocator (or one of the
+ // participant drivers that needs physically contiguous buffers) to call
+ // zx_vmo_create_contiguous(), definitely not the Codec client, but until the
+ // BufferAllocator can be reached from a driver, this is to grant the client
+ // special powers it really shouldn't have, very temporarily until
+ // BufferAllocator is hooked up properly at which point this can be removed.
+ // Strictly speaking we could reverse which end allocates buffers in the Codec
+ // interface to avoid this hack even before BufferAllocator, but the overall
+ // path seems shorter if we jump directly from this to using BufferAllocator.
+ handle? very_temp_kludge_bti_handle;
};
// CodecOutputConfig
@@ -246,112 +246,112 @@
// to some extent. It's different than any configurable output settings the
// client might specifiy for output of an encoder.
struct CodecOutputConfig {
- // A client which always immediately re-configures output buffers on reciept
- // of OnOutputConfig() with buffer_constraints_action_required true can safely
- // ignore this field.
- //
- // A client is permitted to ignore an OnOutputConfig() message even with
- // buffer_constraints_action_required true if the client knows the server has
- // already been told to discard the remainder of the stream with the same
- // stream_lifetime_ordinal or if this stream_lifetime_ordinal field is set to
- // 0. The server is required to re-send needed output config via
- // OnOutputConfig() with new stream_lifetime_ordinal and
- // buffer_constraints_action_required true, if the the most recent completed
- // server-side output config isn't what the server wants/needs yet for the
- // new stream.
- uint64 stream_lifetime_ordinal;
+ // A client which always immediately re-configures output buffers on reciept
+ // of OnOutputConfig() with buffer_constraints_action_required true can safely
+ // ignore this field.
+ //
+ // A client is permitted to ignore an OnOutputConfig() message even with
+ // buffer_constraints_action_required true if the client knows the server has
+ // already been told to discard the remainder of the stream with the same
+ // stream_lifetime_ordinal or if this stream_lifetime_ordinal field is set to
+ // 0. The server is required to re-send needed output config via
+ // OnOutputConfig() with new stream_lifetime_ordinal and
+ // buffer_constraints_action_required true, if the the most recent completed
+ // server-side output config isn't what the server wants/needs yet for the
+ // new stream.
+ uint64 stream_lifetime_ordinal;
- // buffer_constraints_action_required
- //
- // When the buffer constraints are delivered, they indicate whether action is
- // required. A false value here permits delivery of constraints which are
- // fresher without forcing a buffer reconfiguration. If this is false, a
- // client cannot assume that it's safe to immediately re-configure output
- // buffers. If this is true, the client can assume it's safe to immediately
- // configure output buffers once.
- //
- // A client is permitted to ignore buffer constraint versions which have
- // buffer_constraints_action_required false. The server is not permitted to
- // change buffer_constraints_action_required from false to true for the same
- // buffer_constraints_version_ordinal.
- //
- // For each configuration, a client must use new buffers, never buffers that
- // were prevoiusly used for anything else, and never buffers previously used
- // for any other Codec purposes. This rule exists for multiple good reasons,
- // relevant to both mid-stream changes, and changes on stream boundaries.
- // A client should just use new buffers each time.
- //
- // When this is true, the server has already de-refed as many low-level
- // output buffers as the server can while still performing efficient
- // transition to the new buffers and will de-ref the rest asap. A Sync() is
- // not necessary to achieve non-overlap of resource usage to the extent
- // efficiently permitted by the formats involved.
- //
- // If buffer_constraints_action_required is true, the server _must_ not
- // deliver more output data until after output buffers have been configured
- // (or re-configured) by the client.
- //
- // See codec.md for more on buffer_constraints_action_required.
- bool buffer_constraints_action_required;
- CodecBufferConstraints buffer_constraints;
+ // buffer_constraints_action_required
+ //
+ // When the buffer constraints are delivered, they indicate whether action is
+ // required. A false value here permits delivery of constraints which are
+ // fresher without forcing a buffer reconfiguration. If this is false, a
+ // client cannot assume that it's safe to immediately re-configure output
+ // buffers. If this is true, the client can assume it's safe to immediately
+ // configure output buffers once.
+ //
+ // A client is permitted to ignore buffer constraint versions which have
+ // buffer_constraints_action_required false. The server is not permitted to
+ // change buffer_constraints_action_required from false to true for the same
+ // buffer_constraints_version_ordinal.
+ //
+ // For each configuration, a client must use new buffers, never buffers that
+ // were prevoiusly used for anything else, and never buffers previously used
+ // for any other Codec purposes. This rule exists for multiple good reasons,
+ // relevant to both mid-stream changes, and changes on stream boundaries.
+ // A client should just use new buffers each time.
+ //
+ // When this is true, the server has already de-refed as many low-level
+ // output buffers as the server can while still performing efficient
+ // transition to the new buffers and will de-ref the rest asap. A Sync() is
+ // not necessary to achieve non-overlap of resource usage to the extent
+ // efficiently permitted by the formats involved.
+ //
+ // If buffer_constraints_action_required is true, the server _must_ not
+ // deliver more output data until after output buffers have been configured
+ // (or re-configured) by the client.
+ //
+ // See codec.md for more on buffer_constraints_action_required.
+ bool buffer_constraints_action_required;
+ CodecBufferConstraints buffer_constraints;
- // format_details
- //
- // It's up to the client to determine if a change in
- // format_details.format_details_version_ordinal implies any client action is
- // required, based on particular fields in format_details vs. any old value.
- // The server guarantees that if the format has changed, then
- // format_details.format_details_version_ordinal will change, but a change to
- // format_details.format_details_version_ordinal does not guarantee that the
- // format details actually changed. Servers are strongly encouraged to not
- // change format_details.format_details_version_ordinal other than before the
- // first output data of a stream unless there is a real mid-stream format
- // change in the stream. Unnecessary mid-stream format changes can cause
- // simpler clients that have no need to handle mid-stream format changes to
- // just close the channel. Format changes before the first output data of a
- // stream are not "mid-stream" in this context - those can be useful for
- // stream format detection / setup reasons.
- //
- // Note that in case output buffers don't really need to be re-configured
- // despite a format change, a server is encouraged, but not required, to
- // set buffer_constraints_action_required false on the message that conveys
- // the new format details. Simpler servers may just treat the whole output
- // situation as one big thing and demand output buffer reconfiguration on any
- // change in the output situation.
- //
- // A client may or may not actually handle a new buffer_constraints with
- // buffer_constraints_action_required false, but the client should always
- // track the latest format_details.
- //
- // An updated format_details is ordered with respect to emitted output
- // packets, and applies to all subsequent packets until the next
- // format_details with larger version_ordinal. A simple client that does not
- // intend to handle mid-stream format changes should still keep track of the
- // most recently received format_details until the first output packet
- // arrives, then lock down the format details, handle those format details,
- // and verify that any format_details.format_details_version_ordinal received
- // from the server is the same as the locked-down format_details, until the
- // client is done with the stream. Even such a simple client must tolerate
- // format_details.format_details_version_ordinal changing multiple times
- // before the start of data output from a stream (any stream - the first
- // stream or a subsequent stream). This allows a codec to request that
- // output buffers and output format be configured speculatively, and for the
- // output config to be optionally adjusted by the server before the first
- // data output from a stream after the server knows everything it needs to
- // know to fully establish the initial output format details. This
- // simplifies codec server implementation, and allows a clever codec server
- // to guess it's output config for lower latency before any input data, while
- // still being able to fix the output config (including format details) if
- // the guess turns out to be wrong.
- //
- // Whether the format_details.format_details_version_ordinal will actually
- // change mid-stream is a per-codec and per-stream detail that is not
- // specified in comments here, and in most cases also depends on whether the
- // format changes on the input to the codec. Probably it'll be fairly common
- // for a client to use a format which technically supports mid-stream format
- // change, but the client happens to know that none of the streams the client
- // intends to process will ever have a mid-stream format change.
- CodecFormatDetails format_details;
+ // format_details
+ //
+ // It's up to the client to determine if a change in
+ // format_details.format_details_version_ordinal implies any client action is
+ // required, based on particular fields in format_details vs. any old value.
+ // The server guarantees that if the format has changed, then
+ // format_details.format_details_version_ordinal will change, but a change to
+ // format_details.format_details_version_ordinal does not guarantee that the
+ // format details actually changed. Servers are strongly encouraged to not
+ // change format_details.format_details_version_ordinal other than before the
+ // first output data of a stream unless there is a real mid-stream format
+ // change in the stream. Unnecessary mid-stream format changes can cause
+ // simpler clients that have no need to handle mid-stream format changes to
+ // just close the channel. Format changes before the first output data of a
+ // stream are not "mid-stream" in this context - those can be useful for
+ // stream format detection / setup reasons.
+ //
+ // Note that in case output buffers don't really need to be re-configured
+ // despite a format change, a server is encouraged, but not required, to
+ // set buffer_constraints_action_required false on the message that conveys
+ // the new format details. Simpler servers may just treat the whole output
+ // situation as one big thing and demand output buffer reconfiguration on any
+ // change in the output situation.
+ //
+ // A client may or may not actually handle a new buffer_constraints with
+ // buffer_constraints_action_required false, but the client should always
+ // track the latest format_details.
+ //
+ // An updated format_details is ordered with respect to emitted output
+ // packets, and applies to all subsequent packets until the next
+ // format_details with larger version_ordinal. A simple client that does not
+ // intend to handle mid-stream format changes should still keep track of the
+ // most recently received format_details until the first output packet
+ // arrives, then lock down the format details, handle those format details,
+ // and verify that any format_details.format_details_version_ordinal received
+ // from the server is the same as the locked-down format_details, until the
+ // client is done with the stream. Even such a simple client must tolerate
+ // format_details.format_details_version_ordinal changing multiple times
+ // before the start of data output from a stream (any stream - the first
+ // stream or a subsequent stream). This allows a codec to request that
+ // output buffers and output format be configured speculatively, and for the
+ // output config to be optionally adjusted by the server before the first
+ // data output from a stream after the server knows everything it needs to
+ // know to fully establish the initial output format details. This
+ // simplifies codec server implementation, and allows a clever codec server
+ // to guess it's output config for lower latency before any input data, while
+ // still being able to fix the output config (including format details) if
+ // the guess turns out to be wrong.
+ //
+ // Whether the format_details.format_details_version_ordinal will actually
+ // change mid-stream is a per-codec and per-stream detail that is not
+ // specified in comments here, and in most cases also depends on whether the
+ // format changes on the input to the codec. Probably it'll be fairly common
+ // for a client to use a format which technically supports mid-stream format
+ // change, but the client happens to know that none of the streams the client
+ // intends to process will ever have a mid-stream format change.
+ CodecFormatDetails format_details;
};
// Default values for input and output
@@ -405,120 +405,120 @@
// TODO(dustingreen): Rename from CodecPortBufferSettings to CodecBufferSettings
// in a separate CL (after some other stuff is out of the way first).
struct CodecPortBufferSettings {
- // buffer_lifetime_ordinal
- //
- // The containing message starts a new buffer_lifetime_ordinal.
- //
- // There is a separate buffer_lifetime_ordinal for input vs. output.
- //
- // Re-use of the same value is not allowed. Values must be odd. Values must
- // only increase (increasing by more than 2 is permitted).
- //
- // A buffer_lifetime_ordinal lifetime starts at SetInputBufferSettings() or
- // SetOutputBufferSettings(), and ends at the the earlier of
- // CloseCurrentStream() with release_input_buffers/release_output_buffers set
- // or SetOutputBufferSettings() with new buffer_lifetime_ordinal in the case
- // of mid-stream output config change.
- //
- // See codec.md for more on buffer_lifetime_ordinal.
- uint64 buffer_lifetime_ordinal;
+ // buffer_lifetime_ordinal
+ //
+ // The containing message starts a new buffer_lifetime_ordinal.
+ //
+ // There is a separate buffer_lifetime_ordinal for input vs. output.
+ //
+ // Re-use of the same value is not allowed. Values must be odd. Values must
+ // only increase (increasing by more than 2 is permitted).
+ //
+ // A buffer_lifetime_ordinal lifetime starts at SetInputBufferSettings() or
+ // SetOutputBufferSettings(), and ends at the the earlier of
+ // CloseCurrentStream() with release_input_buffers/release_output_buffers set
+ // or SetOutputBufferSettings() with new buffer_lifetime_ordinal in the case
+ // of mid-stream output config change.
+ //
+ // See codec.md for more on buffer_lifetime_ordinal.
+ uint64 buffer_lifetime_ordinal;
- // buffer_constraints_version_ordinal
- //
- // This value indicates which version of constraints the client is/was aware
- // of so far.
- //
- // For input, this must always be 0 because constraints don't change for
- // input (settings can change, but there's no settings vs current constraints
- // synchronization issue on input).
- //
- // For output, this allows the server to know when the client is sufficiently
- // caught up before the server will generate any more output.
- //
- // When there is no active stream, a client is permitted to re-configure
- // buffers again using the same buffer_constraints_version_ordinal.
- //
- // See codec.md for more on buffer_constraints_version_ordinal.
- uint64 buffer_constraints_version_ordinal;
+ // buffer_constraints_version_ordinal
+ //
+ // This value indicates which version of constraints the client is/was aware
+ // of so far.
+ //
+ // For input, this must always be 0 because constraints don't change for
+ // input (settings can change, but there's no settings vs current constraints
+ // synchronization issue on input).
+ //
+ // For output, this allows the server to know when the client is sufficiently
+ // caught up before the server will generate any more output.
+ //
+ // When there is no active stream, a client is permitted to re-configure
+ // buffers again using the same buffer_constraints_version_ordinal.
+ //
+ // See codec.md for more on buffer_constraints_version_ordinal.
+ uint64 buffer_constraints_version_ordinal;
- // The total packet count is split into two pieces to disambiguate how many
- // packets are allocated for the client to hold onto for whatever reason, vs.
- // how many packets are allocated for the server to hold onto for whatever
- // reason.
- //
- // Extra packets to provide slack for peformance reasons can be in either
- // category, but typically packet_count_for_codec_recommended will already
- // include any performance-relevant slack for the server's benefit.
+ // The total packet count is split into two pieces to disambiguate how many
+ // packets are allocated for the client to hold onto for whatever reason, vs.
+ // how many packets are allocated for the server to hold onto for whatever
+ // reason.
+ //
+ // Extra packets to provide slack for peformance reasons can be in either
+ // category, but typically packet_count_for_codec_recommended will already
+ // include any performance-relevant slack for the server's benefit.
- // packet_count_for_codec
- //
- // How many packets the client is allocating for the codec server's use.
- // This must be >= CodecBufferConstraints.packet_count_for_codec_min. If
- // constraints change such that this would no longer be true, the server will
- // send an OnOutputConfig() event.
- //
- // The codec server is allowed to demand that all of packet_count_for_codec
- // become free before making further progress, even if
- // packet_count_for_codec is > packet_count_for_codec_min.
- //
- // A reasonable value for this is
- // CodecBufferConstraints.packet_count_for_codec_recommended.
- //
- // See codec.md for more on packet_count_for_codec.
- uint32 packet_count_for_codec;
+ // packet_count_for_codec
+ //
+ // How many packets the client is allocating for the codec server's use.
+ // This must be >= CodecBufferConstraints.packet_count_for_codec_min. If
+ // constraints change such that this would no longer be true, the server will
+ // send an OnOutputConfig() event.
+ //
+ // The codec server is allowed to demand that all of packet_count_for_codec
+ // become free before making further progress, even if
+ // packet_count_for_codec is > packet_count_for_codec_min.
+ //
+ // A reasonable value for this is
+ // CodecBufferConstraints.packet_count_for_codec_recommended.
+ //
+ // See codec.md for more on packet_count_for_codec.
+ uint32 packet_count_for_codec;
- // packet_count_for_client
- //
- // This must be at least 1. The server will close the channel if this is 0.
- //
- // How many packets the client is allocating for the client's use. The
- // client may hold onto this many packets for arbitrarily-long duration
- // without handing these packets to the codec, and despite doing so, the
- // codec will continue to make progress and function normally without getting
- // stuck. The client holding onto additional packets transiently is ok, but
- // the client needs to hand those additional packets back to the codec
- // eventually if the client wants the codec to make further progress.
- //
- // In addition to this value needing to include at least as many packets as
- // the client ever intends to concurrently camp on indefinitely, any extra
- // slack to benefit client-side performance should also be included here.
- //
- // A typical value for this could be at least 2, but it depends strongly on
- // client implementation and overall client buffering goals. It is up to the
- // client to determine how many packets are needed in this category by any
- // parts of the overall system that will be holding onto packets for any
- // reason. Those parts of the system should have a documented and possibly
- // queryable defined value to help determine this number. Setting this value
- // lower than it actaully needs to be can result in the codec not making
- // progress as it sits waiting for packets, with the client unable to recycle
- // any more packets to the codec. That situation can be difficult to
- // diagnose, while excessively-large values here are wasteful, so care is
- // warranted to set this value properly.
- uint32 packet_count_for_client;
+ // packet_count_for_client
+ //
+ // This must be at least 1. The server will close the channel if this is 0.
+ //
+ // How many packets the client is allocating for the client's use. The
+ // client may hold onto this many packets for arbitrarily-long duration
+ // without handing these packets to the codec, and despite doing so, the
+ // codec will continue to make progress and function normally without getting
+ // stuck. The client holding onto additional packets transiently is ok, but
+ // the client needs to hand those additional packets back to the codec
+ // eventually if the client wants the codec to make further progress.
+ //
+ // In addition to this value needing to include at least as many packets as
+ // the client ever intends to concurrently camp on indefinitely, any extra
+ // slack to benefit client-side performance should also be included here.
+ //
+ // A typical value for this could be at least 2, but it depends strongly on
+ // client implementation and overall client buffering goals. It is up to the
+ // client to determine how many packets are needed in this category by any
+ // parts of the overall system that will be holding onto packets for any
+ // reason. Those parts of the system should have a documented and possibly
+ // queryable defined value to help determine this number. Setting this value
+ // lower than it actaully needs to be can result in the codec not making
+ // progress as it sits waiting for packets, with the client unable to recycle
+ // any more packets to the codec. That situation can be difficult to
+ // diagnose, while excessively-large values here are wasteful, so care is
+ // warranted to set this value properly.
+ uint32 packet_count_for_client;
- // per_packet_buffer_bytes
- //
- // In buffer-per-packet mode, we require that each buffer have usable bytes
- // equal to per_packet_buffer_bytes. Use of differently-sized low-level
- // buffers is possible, but the size of the portion used via the Codec
- // interface per CodecBuffer must be the same for all the buffers.
- //
- // In single-buffer mode, we require the portion of the low-level buffer used
- // via the Codec interface to be size (packet_count_for_codec +
- // packet_count_for_client) * per_packet_buffer_bytes.
- //
- // TODO(dustingreen): determine if we need to relax these restrictions a bit
- // for convenience when using gralloc video buffers.
- uint32 per_packet_buffer_bytes;
+ // per_packet_buffer_bytes
+ //
+ // In buffer-per-packet mode, we require that each buffer have usable bytes
+ // equal to per_packet_buffer_bytes. Use of differently-sized low-level
+ // buffers is possible, but the size of the portion used via the Codec
+ // interface per CodecBuffer must be the same for all the buffers.
+ //
+ // In single-buffer mode, we require the portion of the low-level buffer used
+ // via the Codec interface to be size (packet_count_for_codec +
+ // packet_count_for_client) * per_packet_buffer_bytes.
+ //
+ // TODO(dustingreen): determine if we need to relax these restrictions a bit
+ // for convenience when using gralloc video buffers.
+ uint32 per_packet_buffer_bytes;
- // If true, there is only one buffer with index 0 which all packets
- // implicitly refer to. If false, the packet_index and buffer_index are 1:1,
- // and each packet refers to its corresponding buffer.
- //
- // While it's possible to set up single_buffer_mode false with each buffer
- // referring to the same underlying VMO, single_buffer_mode true is more
- // efficient for that case since only one mapping is created.
- bool single_buffer_mode;
+ // If true, there is only one buffer with index 0 which all packets
+ // implicitly refer to. If false, the packet_index and buffer_index are 1:1,
+ // and each packet refers to its corresponding buffer.
+ //
+ // While it's possible to set up single_buffer_mode false with each buffer
+ // referring to the same underlying VMO, single_buffer_mode true is more
+ // efficient for that case since only one mapping is created.
+ bool single_buffer_mode;
};
// CodecBuffer
@@ -538,36 +538,36 @@
// CodecPacket(s) of the relevant input or output. When single_buffer_mode is
// false, the buffer_index equals the packet_index.
struct CodecBuffer {
- // When using AddOutputBuffer()/AddInputBuffer(), this must match the
- // buffer_lifetime_ordinal of the most recent
- // SetOutputBufferSettings()/SetInputBufferSettings().
- uint64 buffer_lifetime_ordinal;
+ // When using AddOutputBuffer()/AddInputBuffer(), this must match the
+ // buffer_lifetime_ordinal of the most recent
+ // SetOutputBufferSettings()/SetInputBufferSettings().
+ uint64 buffer_lifetime_ordinal;
- // Buffers must be added via AddOutputBuffer() / AddInputBuffer() in order by
- // buffer_index, and the buffer_index is always equal to 0 or equal to the
- // packet_index (depending on single_buffer_mode true or false), but it's
- // still nice to have CodecBuffer include the buffer_index if only for easier
- // debugging.
- uint32 buffer_index;
+ // Buffers must be added via AddOutputBuffer() / AddInputBuffer() in order by
+ // buffer_index, and the buffer_index is always equal to 0 or equal to the
+ // packet_index (depending on single_buffer_mode true or false), but it's
+ // still nice to have CodecBuffer include the buffer_index if only for easier
+ // debugging.
+ uint32 buffer_index;
- // For each new buffer_lifetime_ordinal, a client must use new low-level
- // buffers. This rule exists for multiple very good reasons, and is relevant
- // to mid-stream changes, changes on stream boundaries, and both input and
- // output buffers. A new buffer_lifetime_ordinal needs new low-level
- // buffers, not just new CodecBuffer(s). If you find yourself copying
- // compressed input data into new low-level input buffers solely due to this
- // rule, consider asking the source of the data for the ability to directly
- // fill new VMOs. The rule exists for good reasons, even for input buffers.
- //
- // The previous paragraph does not prohibit carving up VMOs into sub-pieces
- // and using different sub-pieces as different CodecBuffer(s), with some VMOs
- // used for more than one CodecBuffer and possibly others used for only one
- // CodecBuffer. While this is permitted and enables some optimizations, it's
- // not expected to be particularly common.
- //
- // See codec.md for more on CodecBufferData, and more on why we never re-use
- // the same low-level buffers for different buffer_lifetime_ordinal(s).
- CodecBufferData data;
+ // For each new buffer_lifetime_ordinal, a client must use new low-level
+ // buffers. This rule exists for multiple very good reasons, and is relevant
+ // to mid-stream changes, changes on stream boundaries, and both input and
+ // output buffers. A new buffer_lifetime_ordinal needs new low-level
+ // buffers, not just new CodecBuffer(s). If you find yourself copying
+ // compressed input data into new low-level input buffers solely due to this
+ // rule, consider asking the source of the data for the ability to directly
+ // fill new VMOs. The rule exists for good reasons, even for input buffers.
+ //
+ // The previous paragraph does not prohibit carving up VMOs into sub-pieces
+ // and using different sub-pieces as different CodecBuffer(s), with some VMOs
+ // used for more than one CodecBuffer and possibly others used for only one
+ // CodecBuffer. While this is permitted and enables some optimizations, it's
+ // not expected to be particularly common.
+ //
+ // See codec.md for more on CodecBufferData, and more on why we never re-use
+ // the same low-level buffers for different buffer_lifetime_ordinal(s).
+ CodecBufferData data;
};
// CodecBufferData
@@ -579,29 +579,29 @@
// the data. Any multi-domain storage within a gralloc buffer will likely be
// only indirectly represented here.
union CodecBufferData {
- CodecBufferDataVmo vmo;
+ CodecBufferDataVmo vmo;
- // TODO(dustingreen): add the gralloc way
+ // TODO(dustingreen): add the gralloc way
};
// CodecBufferDataVmo
//
// Details for a buffer backed by a VMO.
struct CodecBufferDataVmo {
- // The same VMO can be used by more than one CodecBuffer (only of the same
- // buffer_lifetime_ordinal), but each vmo_handle must be a separate handle.
- handle<vmo> vmo_handle;
+ // The same VMO can be used by more than one CodecBuffer (only of the same
+ // buffer_lifetime_ordinal), but each vmo_handle must be a separate handle.
+ handle<vmo> vmo_handle;
- // Offset within the VMO of the first usable byte. Must be < the VMO's size
- // in bytes.
- uint64 vmo_usable_start;
+ // Offset within the VMO of the first usable byte. Must be < the VMO's size
+ // in bytes.
+ uint64 vmo_usable_start;
- // VMO-relative offset that's one past the last usable byte. This can point
- // one byte beyond the end of the VMO if desired. In other words, this can
- // be equal to the VMO's size, to indicate that the last byte of the VMO is
- // usable (and possibly many byte before that, depending on
- // vmo_usable_start).
- uint64 vmo_usable_size;
+ // VMO-relative offset that's one past the last usable byte. This can point
+ // one byte beyond the end of the VMO if desired. In other words, this can
+ // be equal to the VMO's size, to indicate that the last byte of the VMO is
+ // usable (and possibly many byte before that, depending on
+ // vmo_usable_start).
+ uint64 vmo_usable_size;
};
// CodecPacketHeader
@@ -612,40 +612,40 @@
//
// A populated CodecPacket also has a CodecPacketHeader.
struct CodecPacketHeader {
- // This is which buffer configuration lifetime this header is referring to.
- //
- // A packet_index is only really meaningful with respect to a particular
- // buffer_lifetime_ordinal.
- //
- // See CodecPortBufferSettings.buffer_lifetime_ordinal.
- //
- // For QueueInputPacket(), a server receiving a buffer_lifetime_ordinal that
- // isn't the current input buffer_lifetime_ordinal will close the channel.
- //
- // For OnFreeInputPacket() and RecycleOutputPacket(), the reciever (client or
- // server) must ignore a message with stale buffer_lifetime_ordinal.
- uint64 buffer_lifetime_ordinal;
+ // This is which buffer configuration lifetime this header is referring to.
+ //
+ // A packet_index is only really meaningful with respect to a particular
+ // buffer_lifetime_ordinal.
+ //
+ // See CodecPortBufferSettings.buffer_lifetime_ordinal.
+ //
+ // For QueueInputPacket(), a server receiving a buffer_lifetime_ordinal that
+ // isn't the current input buffer_lifetime_ordinal will close the channel.
+ //
+ // For OnFreeInputPacket() and RecycleOutputPacket(), the reciever (client or
+ // server) must ignore a message with stale buffer_lifetime_ordinal.
+ uint64 buffer_lifetime_ordinal;
- // The overall set of packet_index values is densely packed from 0..count-1
- // for input and output separately. They can be queued in any order.
- //
- // Both the client and server should validate the packet_index against the
- // known bound and disconnect if it's out of bounds.
- //
- // When running in buffer-per-packet mode, the packet_index is also the
- // buffer index. When running in single-buffer mode, the buffer index is
- // always 0 referring to the single buffer.
- //
- // The packet_index values don't imply anything about order of use of
- // packets. The client should not expect the ordering to remain the same over
- // time - the codec is free to hold on to an input or output packet for a
- // while during which other packet_index values may be used multiple times.
- //
- // For a given properly-functioning Codec instance, packet_index values will
- // be unique among concurrently-outstanding packets. Servers should validate
- // that a client isn't double-using a packet and clients should validate as
- // necessary to avoid undefined or unexpected client behavior.
- uint32 packet_index;
+ // The overall set of packet_index values is densely packed from 0..count-1
+ // for input and output separately. They can be queued in any order.
+ //
+ // Both the client and server should validate the packet_index against the
+ // known bound and disconnect if it's out of bounds.
+ //
+ // When running in buffer-per-packet mode, the packet_index is also the
+ // buffer index. When running in single-buffer mode, the buffer index is
+ // always 0 referring to the single buffer.
+ //
+ // The packet_index values don't imply anything about order of use of
+ // packets. The client should not expect the ordering to remain the same over
+ // time - the codec is free to hold on to an input or output packet for a
+ // while during which other packet_index values may be used multiple times.
+ //
+ // For a given properly-functioning Codec instance, packet_index values will
+ // be unique among concurrently-outstanding packets. Servers should validate
+ // that a client isn't double-using a packet and clients should validate as
+ // necessary to avoid undefined or unexpected client behavior.
+ uint32 packet_index;
};
// CodecPacket
@@ -664,128 +664,139 @@
// The client must not modify the referenced data in between QueueInputPacket()
// and OnFreeInputPacket().
struct CodecPacket {
- CodecPacketHeader header;
+ CodecPacketHeader header;
- // stream_lifetime_ordinal
- //
- // The value 1 is the lowest permitted value after codec creation. Values
- // sent by the client must be odd. Values must only increase.
- //
- // A stream_lifetime_ordinal represents the lifetime of a stream. All
- // messages that are specific to a stream have the stream_lifetime_ordinal
- // value and the value is the same for all messages relating to a given
- // stream.
- //
- // See codec.md for more on stream_lifetime_ordinal.
- uint64 stream_lifetime_ordinal;
+ // Which buffer this packet refers to. For single-buffer mode this will
+ // always be 0, but for multi-buffer mode, a given in-flight interval of a
+ // packet can refer to any buffer. The packet has an associated buffer only
+ // while the packet is in-flight, not while the packet is free.
+ //
+ // The default value makes accidental inappropriate use of index 0 less
+ // likely (will tend to complain in an obvious way if not filled out instead
+ // of a non-obvious data corruption when decoding buffer 0 repeatedly instead
+ // of the correct buffers).
+ uint32 buffer_index = 0x80000000;
- // start_offset and valid_length_bytes
- //
- // Which part of the relevant buffer is this packet using. These are valid
- // for input data that's in-flight to the codec, and are valid for output data
- // from the codec.
- //
- // For compressed formats and uncompressed audio, the data in
- // [start_offset, start_offset + valid_length_bytes) is the contiguously valid
- // data refered to by this packet.
- //
- // For uncompressed video frames, CodecFormatDetails is the primary means of
- // determining which bytes are relevant. The offsets in CodecFormatDetails
- // are relative to the start_offset here. The valid_length_bytes must be
- // large enough to include the full last line of pixel data, including the
- // full line stride of the last line (not just the width in pixels of the
- // last line).
- //
- // Despite these being filled out, some uncompressed video buffers are of
- // types that are not readable by the CPU. These fields being here don't
- // imply there's any way for the CPU to read an uncompressed frame.
- //
- // TODO(dustingreen): Do we have any reason to require that these be filled
- // out for opaque uncompressed video frames that the CPU can't read? In that
- // case do we want to require them just so they can be potentially passed on
- // to a HW renderer in case the HW renderer has any use for them? Or more
- // likely, it may just be that these tend to refer to the whole size of an
- // uncompressed buffer, with format_details taking care of specifying which
- // bytes are actually relevant.
- uint32 start_offset;
+ // stream_lifetime_ordinal
+ //
+ // The value 1 is the lowest permitted value after codec creation. Values
+ // sent by the client must be odd. Values must only increase.
+ //
+ // A stream_lifetime_ordinal represents the lifetime of a stream. All
+ // messages that are specific to a stream have the stream_lifetime_ordinal
+ // value and the value is the same for all messages relating to a given
+ // stream.
+ //
+ // See codec.md for more on stream_lifetime_ordinal.
+ uint64 stream_lifetime_ordinal;
- // valid_length_bytes
- //
- // This must be > 0.
- //
- // The semantics for valid data per packet vary depending on data type as
- // follows.
- //
- // uncompressed video - A video frame can't be split across packets. Each
- // packet is one video frame.
- //
- // uncompressed audio - Regardless of float or int, linear or uLaw, or number
- // of channels, a packet must contain an non-negative number of complete
- // audio frames, where a single audio frame consists of data for all the
- // channels for the same single point in time. Any codec-specific internal
- // details re. lower rate sampling for LFE channel or the like should be
- // hidden by the Codec server implementation.
- //
- // compressed data input - A packet must contain at least one byte of data.
- // See also codec_input_bytes_min. Splitting AUs at arbitrary byte
- // boundaries is permitted, including at boundaries that are in AU headers.
- //
- // compressed data output - The codec is not required to fully fill each
- // output packet's buffer.
- uint32 valid_length_bytes;
+ // start_offset and valid_length_bytes
+ //
+ // Which part of the relevant buffer is this packet using. These are valid
+ // for input data that's in-flight to the codec, and are valid for output data
+ // from the codec.
+ //
+ // For compressed formats and uncompressed audio, the data in
+ // [start_offset, start_offset + valid_length_bytes) is the contiguously valid
+ // data refered to by this packet.
+ //
+ // For uncompressed video frames, CodecFormatDetails is the primary means of
+ // determining which bytes are relevant. The offsets in CodecFormatDetails
+ // are relative to the start_offset here. The valid_length_bytes must be
+ // large enough to include the full last line of pixel data, including the
+ // full line stride of the last line (not just the width in pixels of the
+ // last line).
+ //
+ // Despite these being filled out, some uncompressed video buffers are of
+ // types that are not readable by the CPU. These fields being here don't
+ // imply there's any way for the CPU to read an uncompressed frame.
+ //
+ // TODO(dustingreen): Do we have any reason to require that these be filled
+ // out for opaque uncompressed video frames that the CPU can't read? In that
+ // case do we want to require them just so they can be potentially passed on
+ // to a HW renderer in case the HW renderer has any use for them? Or more
+ // likely, it may just be that these tend to refer to the whole size of an
+ // uncompressed buffer, with format_details taking care of specifying which
+ // bytes are actually relevant.
+ uint32 start_offset;
- // This value is not strictly speaking a timestamp. It is an arbitrary
- // unsigned 64-bit number that, under some circumstances, will be passed by a
- // codec unmodified from an input packet to the exactly-corresponding output
- // packet.
- //
- // For timestamp_ish values to be propagated from input to output the
- // following conditions must be true:
- // * promise_separate_access_units_on_input must be true
- // * has_timestamp_ish must be true for a given input packet, to have that
- // timestamp_ish value (potentially) propagate through to an output
- // * the Codec instance itself decides (async) that the input packet
- // generates an output packet - if a given input never generates an output
- // packet then the timestamp_ish value on the input will never show up on
- // any output packet - depending on the characteristics of the input and
- // output formats, and whether a decoder is willing to join mid-stream, etc
- // this can be more or less likely to occur, but clients should be written
- // to accomodate timestamp_ish values that are fed on input but never show
- // up on output, at least to a reasonable degree (not crashing, not
- // treating as an error).
- //
- // See codec.md for more on timestamp_ish.
- bool has_timestamp_ish;
- uint64 timestamp_ish;
+ // valid_length_bytes
+ //
+ // This must be > 0.
+ //
+ // The semantics for valid data per packet vary depending on data type as
+ // follows.
+ //
+ // uncompressed video - A video frame can't be split across packets. Each
+ // packet is one video frame.
+ //
+ // uncompressed audio - Regardless of float or int, linear or uLaw, or number
+ // of channels, a packet must contain an non-negative number of complete
+ // audio frames, where a single audio frame consists of data for all the
+ // channels for the same single point in time. Any codec-specific internal
+ // details re. lower rate sampling for LFE channel or the like should be
+ // hidden by the Codec server implementation.
+ //
+ // compressed data input - A packet must contain at least one byte of data.
+ // See also codec_input_bytes_min. Splitting AUs at arbitrary byte
+ // boundaries is permitted, including at boundaries that are in AU headers.
+ //
+ // compressed data output - The codec is not required to fully fill each
+ // output packet's buffer.
+ uint32 valid_length_bytes;
- // start_access_unit
- //
- // If promise_separate_access_units_on_input (TODO(dustingreen): or any
- // similar mode for output) is true, this bool must be set appropriately
- // depending on whether byte 0 _is_ or _is not_ the start of an access unit.
- // The client is required to know, and required to set this boolean properly.
- // The server is allowed to infer that when this boolean is false, byte 0 is
- // the first byte of a continuation of a previously-started AU. (The byte at
- // start_offset is "byte 0".)
- //
- // If promise_separate_access_units_on_input is false, this boolean is
- // ignored.
- bool start_access_unit;
+ // This value is not strictly speaking a timestamp. It is an arbitrary
+ // unsigned 64-bit number that, under some circumstances, will be passed by a
+ // codec unmodified from an input packet to the exactly-corresponding output
+ // packet.
+ //
+ // For timestamp_ish values to be propagated from input to output the
+ // following conditions must be true:
+ // * promise_separate_access_units_on_input must be true
+ // * has_timestamp_ish must be true for a given input packet, to have that
+ // timestamp_ish value (potentially) propagate through to an output
+ // * the Codec instance itself decides (async) that the input packet
+ // generates an output packet - if a given input never generates an output
+ // packet then the timestamp_ish value on the input will never show up on
+ // any output packet - depending on the characteristics of the input and
+ // output formats, and whether a decoder is willing to join mid-stream, etc
+ // this can be more or less likely to occur, but clients should be written
+ // to accomodate timestamp_ish values that are fed on input but never show
+ // up on output, at least to a reasonable degree (not crashing, not
+ // treating as an error).
+ //
+ // See codec.md for more on timestamp_ish.
+ bool has_timestamp_ish;
+ uint64 timestamp_ish;
- // known_end_access_unit
- //
- // A client is never required to set this boolean to true.
- //
- // If promise_separate_access_units_on_input is true, for input data, this
- // boolean must be false if the last byte of this packet is not the last byte
- // of an AU, and this boolean _may_ be true if the last byte of this packet
- // is the last byte of an AU. A client delivering one AU at a time that's
- // interested in the lowest possible latency via the decoder should set this
- // boolean to true when it can be set to true.
- //
- // If promise_separate_access_units_on_input is false, this boolean is
- // ignored.
- bool known_end_access_unit;
+ // start_access_unit
+ //
+ // If promise_separate_access_units_on_input (TODO(dustingreen): or any
+ // similar mode for output) is true, this bool must be set appropriately
+ // depending on whether byte 0 _is_ or _is not_ the start of an access unit.
+ // The client is required to know, and required to set this boolean properly.
+ // The server is allowed to infer that when this boolean is false, byte 0 is
+ // the first byte of a continuation of a previously-started AU. (The byte at
+ // start_offset is "byte 0".)
+ //
+ // If promise_separate_access_units_on_input is false, this boolean is
+ // ignored.
+ bool start_access_unit;
+
+ // known_end_access_unit
+ //
+ // A client is never required to set this boolean to true.
+ //
+ // If promise_separate_access_units_on_input is true, for input data, this
+ // boolean must be false if the last byte of this packet is not the last byte
+ // of an AU, and this boolean _may_ be true if the last byte of this packet
+ // is the last byte of an AU. A client delivering one AU at a time that's
+ // interested in the lowest possible latency via the decoder should set this
+ // boolean to true when it can be set to true.
+ //
+ // If promise_separate_access_units_on_input is false, this boolean is
+ // ignored.
+ bool known_end_access_unit;
};
// Codec
@@ -832,473 +843,473 @@
// and is out of scope of this paragraph, the client should always use separate
// Codec instances for security-separate client-side contexts.
interface Codec {
- // EnableOnStreamFailed()
- //
- // Permit the server to use OnStreamFailed() instead of the server just
- // closing the whole Codec channel on stream failure.
- //
- // If the server hasn't seen this message by the time a stream fails, the
- // server will close the Codec channel instead of sending OnStreamFailed().
- 1: EnableOnStreamFailed();
+ // EnableOnStreamFailed()
+ //
+ // Permit the server to use OnStreamFailed() instead of the server just
+ // closing the whole Codec channel on stream failure.
+ //
+ // If the server hasn't seen this message by the time a stream fails, the
+ // server will close the Codec channel instead of sending OnStreamFailed().
+ 1: EnableOnStreamFailed();
- // OnStreamFailed()
- //
- // The stream has failed, but the Codec instance is still usable for a new
- // stream.
- //
- // This message is only ever sent by the server if the client previously sent
- // EnableOnStreamFailed(). If the client didn't send EnableOnStreamFailed()
- // then the server closes the Codec channel instead.
- //
- // Codec server implementations are encouraged to handle stream errors (and
- // ideally to also report them via error_ bools of OnOutputPacket() and
- // OnOutputEndOfStream()) without failing the whole stream, but if a codec
- // server is unable to do that, but still can cleanly contain the failure to
- // the stream, the codec server can (assuming EnableOnStreamFailed() was
- // called) use OnStreamFailed() to indicate the stream failure to the client
- // without closing the Codec channel.
- //
- // An ideal Codec server handles problems with input data without sending
- // this message, but sending this message is preferred vs. closing the server
- // end of the Codec channel if the Codec server can 100% reliably contain the
- // stream failure to the stream, without any adverse impact to any later
- // stream.
- //
- // No further messages will arrive from the server regarding the failed
- // stream. This includes any OnOutputEndOfStream() that the client would
- // have otherwise expected.
- //
- // TODO(dustingreen): Add at least an error_message string and _maybe_ a
- // zx_status_t, though that might tend to encourage mis-use of zx_status_t so
- // maybe just error_message for quicker debugging on the client side. Also
- // plumb from CodecAdapterH264 and similar.
- 2: -> OnStreamFailed(uint64 stream_lifetime_ordinal);
+ // OnStreamFailed()
+ //
+ // The stream has failed, but the Codec instance is still usable for a new
+ // stream.
+ //
+ // This message is only ever sent by the server if the client previously sent
+ // EnableOnStreamFailed(). If the client didn't send EnableOnStreamFailed()
+ // then the server closes the Codec channel instead.
+ //
+ // Codec server implementations are encouraged to handle stream errors (and
+ // ideally to also report them via error_ bools of OnOutputPacket() and
+ // OnOutputEndOfStream()) without failing the whole stream, but if a codec
+ // server is unable to do that, but still can cleanly contain the failure to
+ // the stream, the codec server can (assuming EnableOnStreamFailed() was
+ // called) use OnStreamFailed() to indicate the stream failure to the client
+ // without closing the Codec channel.
+ //
+ // An ideal Codec server handles problems with input data without sending
+ // this message, but sending this message is preferred vs. closing the server
+ // end of the Codec channel if the Codec server can 100% reliably contain the
+ // stream failure to the stream, without any adverse impact to any later
+ // stream.
+ //
+ // No further messages will arrive from the server regarding the failed
+ // stream. This includes any OnOutputEndOfStream() that the client would
+ // have otherwise expected.
+ //
+ // TODO(dustingreen): Add at least an error_message string and _maybe_ a
+ // zx_status_t, though that might tend to encourage mis-use of zx_status_t so
+ // maybe just error_message for quicker debugging on the client side. Also
+ // plumb from CodecAdapterH264 and similar.
+ 2: -> OnStreamFailed(uint64 stream_lifetime_ordinal);
- // OnInputConstraints()
- //
- // The server sends this shortly after Codec creation to indicate input
- // buffer constraints. The "min" and "max" input constraints don't change
- // for the life of the Codec.
- //
- // The "max" values for buffer size and count are large enough to support the
- // most demanding format the server supports on input. The "recommended"
- // values should be workable for use with the input CodecFormatDetails
- // conveyed during Codec creation. The "recommended" values are not
- // necessarily suitable if the client uses QueueInputFormatDetails() to
- // change the input format. In that case it's up to the client to determine
- // suitable values, either by creating a new Codec instance instead, or
- // knowing suitable values outside the scope of this protocol.
- //
- // See comments on CodecBufferConstraints.
- //
- // This message is guaranteed to be sent unsolicited to the Codec client
- // during or shortly after Codec creation. Clients should not depend on this
- // being the very first message to arrive at the client. OnOutputConfig()
- // may be sent first by some codecs that already know their initial output
- // config without any input data, to encourage (but not stictly require) the
- // client to configure output buffers before feeding the first input, to
- // avoid a wasteful OnOutputConfig() being generated for that first stream if
- // the client has started configuring output but isn't done configuring
- // output before the client sends the first input data for the first stream.
- // A client is free to ignore OnOutputConfig() with a stale
- // stream_lifetime_ordinal, but handling OnOutputConfig() with
- // stream_lifetime_ordinal 0 (if any are sent) can help reduce latency to
- // first output. See OnOutputConfig() for more details.
- //
- // The "min" and "max" input constraints are guaranteed not to change for a
- // given Codec instance. The "recommended" values may effectively change
- // when the server processes QueueInputFormatDetails(). There is not any way
- // in the protocol short of creating a new Codec instance for the client to
- // get those new "recommended" values.
- //
- // TODO(dustingreen): Maybe provide a way for the client to get updated
- // "recommended" values for input, maybe only on request rather than via this
- // event, to keep things simpler for simpler clients. Maybe separate the
- // recommendations from the constraints.
- 3: -> OnInputConstraints(CodecBufferConstraints input_constraints);
+ // OnInputConstraints()
+ //
+ // The server sends this shortly after Codec creation to indicate input
+ // buffer constraints. The "min" and "max" input constraints don't change
+ // for the life of the Codec.
+ //
+ // The "max" values for buffer size and count are large enough to support the
+ // most demanding format the server supports on input. The "recommended"
+ // values should be workable for use with the input CodecFormatDetails
+ // conveyed during Codec creation. The "recommended" values are not
+ // necessarily suitable if the client uses QueueInputFormatDetails() to
+ // change the input format. In that case it's up to the client to determine
+ // suitable values, either by creating a new Codec instance instead, or
+ // knowing suitable values outside the scope of this protocol.
+ //
+ // See comments on CodecBufferConstraints.
+ //
+ // This message is guaranteed to be sent unsolicited to the Codec client
+ // during or shortly after Codec creation. Clients should not depend on this
+ // being the very first message to arrive at the client. OnOutputConfig()
+ // may be sent first by some codecs that already know their initial output
+ // config without any input data, to encourage (but not stictly require) the
+ // client to configure output buffers before feeding the first input, to
+ // avoid a wasteful OnOutputConfig() being generated for that first stream if
+ // the client has started configuring output but isn't done configuring
+ // output before the client sends the first input data for the first stream.
+ // A client is free to ignore OnOutputConfig() with a stale
+ // stream_lifetime_ordinal, but handling OnOutputConfig() with
+ // stream_lifetime_ordinal 0 (if any are sent) can help reduce latency to
+ // first output. See OnOutputConfig() for more details.
+ //
+ // The "min" and "max" input constraints are guaranteed not to change for a
+ // given Codec instance. The "recommended" values may effectively change
+ // when the server processes QueueInputFormatDetails(). There is not any way
+ // in the protocol short of creating a new Codec instance for the client to
+ // get those new "recommended" values.
+ //
+ // TODO(dustingreen): Maybe provide a way for the client to get updated
+ // "recommended" values for input, maybe only on request rather than via this
+ // event, to keep things simpler for simpler clients. Maybe separate the
+ // recommendations from the constraints.
+ 3: -> OnInputConstraints(CodecBufferConstraints input_constraints);
- // SetInputBufferSettings() and AddInputBuffer()
- //
- // Configuring input buffers consists of calling SetInputBufferSettings()
- // followed by a number of calls to AddInputBuffer() equal to the number of
- // buffers set via SetInputBufferSettings(). In buffer-per-packet mode, this
- // is the same as the number of packets. In single-buffer mode, this is 1.
- //
- // After OnInputConstraints(), the client uses these two methods to set up
- // input buffers and packets.
- //
- // Configuring input buffers is required before QueueInputPacket().
- //
- // The client can also re-set-up input buffers any time there is no current
- // stream. The client need not wait until all previously-set-up input
- // buffers are with the client via OnFreeInputPacket(). The old
- // buffer_lifetime_ordinal just ends. See codec.md for more info on "buffer
- // lifetime".
- //
- // The recommended way to de-overlap resource usage (when/if the client wants
- // to) is to send CloseCurrentStream() with release_input_buffers true then
- // send Sync() and wait for its response before allocating any new buffers.
- // How to cause other parts of the system to release their references on
- // low-level buffers is outside the scope of this interface.
- //
- // This call ends any previous buffer_lifetime_ordinal, and starts a new one.
- 4: SetInputBufferSettings(CodecPortBufferSettings input_settings);
+ // SetInputBufferSettings() and AddInputBuffer()
+ //
+ // Configuring input buffers consists of calling SetInputBufferSettings()
+ // followed by a number of calls to AddInputBuffer() equal to the number of
+ // buffers set via SetInputBufferSettings(). In buffer-per-packet mode, this
+ // is the same as the number of packets. In single-buffer mode, this is 1.
+ //
+ // After OnInputConstraints(), the client uses these two methods to set up
+ // input buffers and packets.
+ //
+ // Configuring input buffers is required before QueueInputPacket().
+ //
+ // The client can also re-set-up input buffers any time there is no current
+ // stream. The client need not wait until all previously-set-up input
+ // buffers are with the client via OnFreeInputPacket(). The old
+ // buffer_lifetime_ordinal just ends. See codec.md for more info on "buffer
+ // lifetime".
+ //
+ // The recommended way to de-overlap resource usage (when/if the client wants
+ // to) is to send CloseCurrentStream() with release_input_buffers true then
+ // send Sync() and wait for its response before allocating any new buffers.
+ // How to cause other parts of the system to release their references on
+ // low-level buffers is outside the scope of this interface.
+ //
+ // This call ends any previous buffer_lifetime_ordinal, and starts a new one.
+ 4: SetInputBufferSettings(CodecPortBufferSettings input_settings);
- // The client is required to add all the input buffers before sending any
- // message that starts a new stream else the codec will close the Codec
- // channel.
- //
- // When the last buffer is added with this message, all the input packets
- // effectively jump from non-existent to free with the client. The Codec
- // will not generate an OnFreeInputPacket() for each new input packet. The
- // client can immediately start sending QueueInputPacket() after sending the
- // last AddInputBuffer().
- 5: AddInputBuffer(CodecBuffer buffer);
+ // The client is required to add all the input buffers before sending any
+ // message that starts a new stream else the codec will close the Codec
+ // channel.
+ //
+ // When the last buffer is added with this message, all the input packets
+ // effectively jump from non-existent to free with the client. The Codec
+ // will not generate an OnFreeInputPacket() for each new input packet. The
+ // client can immediately start sending QueueInputPacket() after sending the
+ // last AddInputBuffer().
+ 5: AddInputBuffer(CodecBuffer buffer);
- // OnOutputConfig()
- //
- // This event informs the client of new output config. The server will send
- // at least one of these messages before the first output packet of a stream,
- // but that message might not have buffer_constraints_action_required true.
- //
- // If buffer_constraints_action_required is true and the
- // stream_lifetime_ordinal matches the current stream, the client must react
- // by configuring or re-configuring output buffers.
- //
- // Some clients may prefer not to support mid-stream output config changes,
- // but even those clients are required to process OnOutputConfig() messages
- // up to the first output packet of each stream, as OnOutputConfig() is used
- // for stream format detection as well as for potential mid-stream output
- // config changes.
- //
- // For more on OnOutputConfig(), see cocec.md.
- 6: -> OnOutputConfig(CodecOutputConfig output_config);
+ // OnOutputConfig()
+ //
+ // This event informs the client of new output config. The server will send
+ // at least one of these messages before the first output packet of a stream,
+ // but that message might not have buffer_constraints_action_required true.
+ //
+ // If buffer_constraints_action_required is true and the
+ // stream_lifetime_ordinal matches the current stream, the client must react
+ // by configuring or re-configuring output buffers.
+ //
+ // Some clients may prefer not to support mid-stream output config changes,
+ // but even those clients are required to process OnOutputConfig() messages
+ // up to the first output packet of each stream, as OnOutputConfig() is used
+ // for stream format detection as well as for potential mid-stream output
+ // config changes.
+ //
+ // For more on OnOutputConfig(), see cocec.md.
+ 6: -> OnOutputConfig(CodecOutputConfig output_config);
- // SetOutputBufferSettings() and AddOutputBuffer()
- //
- // These are not permitted until after the first OnOutputConfig().
- //
- // Roughly speaking, these messages are sent in response to OnOutputConfig()
- // with buffer_constraints_action_required true.
- //
- // Configuring output buffers consists of calling SetOutputBufferSettings()
- // followed by a number of calls to AddOutputBuffer() equal to the number of
- // buffers set via SetOutputBufferSettings(). In buffer-per-packet mode, this
- // is the same as the number of packets. In single-buffer mode, this is 1.
- //
- // Configuring output buffers is _required_ after OnOutputConfig() is
- // received by the client with buffer_constraints_action_required true and
- // stream_lifetime_ordinal equal to the client's current
- // stream_lifetime_ordinal (even if there is an active stream), and is
- // _permitted_ any time there is no current stream.
- //
- // For more on SetOutputBufferSettings() and AddOutputBuffer(), see codec.md.
- 7: SetOutputBufferSettings(CodecPortBufferSettings output_settings);
- 8: AddOutputBuffer(CodecBuffer buffer);
+ // SetOutputBufferSettings() and AddOutputBuffer()
+ //
+ // These are not permitted until after the first OnOutputConfig().
+ //
+ // Roughly speaking, these messages are sent in response to OnOutputConfig()
+ // with buffer_constraints_action_required true.
+ //
+ // Configuring output buffers consists of calling SetOutputBufferSettings()
+ // followed by a number of calls to AddOutputBuffer() equal to the number of
+ // buffers set via SetOutputBufferSettings(). In buffer-per-packet mode, this
+ // is the same as the number of packets. In single-buffer mode, this is 1.
+ //
+ // Configuring output buffers is _required_ after OnOutputConfig() is
+ // received by the client with buffer_constraints_action_required true and
+ // stream_lifetime_ordinal equal to the client's current
+ // stream_lifetime_ordinal (even if there is an active stream), and is
+ // _permitted_ any time there is no current stream.
+ //
+ // For more on SetOutputBufferSettings() and AddOutputBuffer(), see codec.md.
+ 7: SetOutputBufferSettings(CodecPortBufferSettings output_settings);
+ 8: AddOutputBuffer(CodecBuffer buffer);
- // FlushEndOfStreamAndCloseStream()
- //
- // This message is optional.
- //
- // This message is only valid after QueueInputEndOfStream() for this stream.
- // The stream_lifetime_ordinal input parameter must match the
- // stream_lifetime_ordinal of the QueueInputEndOfStream(), else the server
- // will close the channel.
- //
- // A client can use this message to flush through (not discard) the last
- // input data of a stream so that the codec server generates corresponding
- // output data for all the input data before the server moves on to the next
- // stream, without forcing the client to wait for OnOutputEndOfStream()
- // before queueing data of another stream.
- //
- // The difference between QueueInputEndOfStream() and
- // FlushEndOfStreamAndCloseStream(): QueueInputEndOfStream() is a promise
- // from the client that there will not be any more input data for the stream
- // (and this info is needed by some codecs for the codec to ever emit the very
- // last output data). The QueueInputEndOfStream() having been sent doesn't
- // prevent the client from later completely discarding the rest of the
- // current stream by closing the current stream (with or without a stream
- // switch). In contrast, FlushEndOfStreamAndCloseStream() is a request from
- // the client that all the previously-queued input data be processed
- // including the logical "EndOfStream" showing up as OnOutputEndOfStream()
- // (in success case) before moving on to any newer stream - this essentially
- // changes the close-stream handling from discard to flush-through for this
- // stream only.
- //
- // A client using this message can start providing input data for a new
- // stream without that causing discard of old stream data. That's the purpose
- // of this message - to allow a client to flush through (not discard) the old
- // stream's last data (instead of the default when closing or switching
- // streams which is discard).
- //
- // Because the old stream is not done processing yet and the old stream's
- // data is not being discarded, the client must be prepared to continue to
- // process OnOutputConfig() messages until the stream_lifetime_ordinal is
- // done. The client will know the stream_lifetime_ordinal is done when
- // OnOutputEndOfStream(), OnStreamFailed(), or the Codec channel closes.
- //
- // For more on FlushEndOfStreamAndCloseStream(), see codec.md.
- 9: FlushEndOfStreamAndCloseStream(uint64 stream_lifetime_ordinal);
+ // FlushEndOfStreamAndCloseStream()
+ //
+ // This message is optional.
+ //
+ // This message is only valid after QueueInputEndOfStream() for this stream.
+ // The stream_lifetime_ordinal input parameter must match the
+ // stream_lifetime_ordinal of the QueueInputEndOfStream(), else the server
+ // will close the channel.
+ //
+ // A client can use this message to flush through (not discard) the last
+ // input data of a stream so that the codec server generates corresponding
+ // output data for all the input data before the server moves on to the next
+ // stream, without forcing the client to wait for OnOutputEndOfStream()
+ // before queueing data of another stream.
+ //
+ // The difference between QueueInputEndOfStream() and
+ // FlushEndOfStreamAndCloseStream(): QueueInputEndOfStream() is a promise
+ // from the client that there will not be any more input data for the stream
+ // (and this info is needed by some codecs for the codec to ever emit the very
+ // last output data). The QueueInputEndOfStream() having been sent doesn't
+ // prevent the client from later completely discarding the rest of the
+ // current stream by closing the current stream (with or without a stream
+ // switch). In contrast, FlushEndOfStreamAndCloseStream() is a request from
+ // the client that all the previously-queued input data be processed
+ // including the logical "EndOfStream" showing up as OnOutputEndOfStream()
+ // (in success case) before moving on to any newer stream - this essentially
+ // changes the close-stream handling from discard to flush-through for this
+ // stream only.
+ //
+ // A client using this message can start providing input data for a new
+ // stream without that causing discard of old stream data. That's the purpose
+ // of this message - to allow a client to flush through (not discard) the old
+ // stream's last data (instead of the default when closing or switching
+ // streams which is discard).
+ //
+ // Because the old stream is not done processing yet and the old stream's
+ // data is not being discarded, the client must be prepared to continue to
+ // process OnOutputConfig() messages until the stream_lifetime_ordinal is
+ // done. The client will know the stream_lifetime_ordinal is done when
+ // OnOutputEndOfStream(), OnStreamFailed(), or the Codec channel closes.
+ //
+ // For more on FlushEndOfStreamAndCloseStream(), see codec.md.
+ 9: FlushEndOfStreamAndCloseStream(uint64 stream_lifetime_ordinal);
- // CloseCurrentStream()
- //
- // This "closes" the current stream, leaving no current stream. In addition,
- // this message can optionally release input buffers or output buffers.
- //
- // If there has never been any active stream, the stream_lifetime_ordinal must
- // be zero or the server will close the channel. If there has been an active
- // stream, the stream_lifetime_ordinal must be the most recent active stream
- // whether that stream is still active or not. Else the server will close the
- // channel.
- //
- // Multiple of this message without any new active stream in between is not
- // to be considered an error, which allows a client to use this message to
- // close the current stream to stop wasting processing power on a stream the
- // user no longer cares about, then later decide that buffers should be
- // released and send this message again with release_input_buffers and/or
- // release_output_buffers true to get the buffers released, if the client is
- // interested in trying to avoid overlap in resource usage between old
- // buffers and new buffers (not all clients are).
- //
- // See also Sync().
- //
- // For more on CloseCurrentStream(), see codec.md.
- 10: CloseCurrentStream(
- uint64 stream_lifetime_ordinal,
- bool release_input_buffers,
- bool release_output_buffers);
+ // CloseCurrentStream()
+ //
+ // This "closes" the current stream, leaving no current stream. In addition,
+ // this message can optionally release input buffers or output buffers.
+ //
+ // If there has never been any active stream, the stream_lifetime_ordinal must
+ // be zero or the server will close the channel. If there has been an active
+ // stream, the stream_lifetime_ordinal must be the most recent active stream
+ // whether that stream is still active or not. Else the server will close the
+ // channel.
+ //
+ // Multiple of this message without any new active stream in between is not
+ // to be considered an error, which allows a client to use this message to
+ // close the current stream to stop wasting processing power on a stream the
+ // user no longer cares about, then later decide that buffers should be
+ // released and send this message again with release_input_buffers and/or
+ // release_output_buffers true to get the buffers released, if the client is
+ // interested in trying to avoid overlap in resource usage between old
+ // buffers and new buffers (not all clients are).
+ //
+ // See also Sync().
+ //
+ // For more on CloseCurrentStream(), see codec.md.
+ 10: CloseCurrentStream(
+ uint64 stream_lifetime_ordinal,
+ bool release_input_buffers,
+ bool release_output_buffers);
- // Sync() -> ()
- //
- // On completion, all previous Codec calls have done what they're going to do
- // server-side, _except_ for processing of data queued using
- // QueueInputPacket().
- //
- // The main purpose of this call is to enable the client to wait until
- // CloseCurrentStream() with release_input_buffers and/or
- // release_output_buffers set to true to take effect, before the client
- // allocates new buffers and re-sets-up input and/or output buffers. This
- // de-overlapping of resource usage can be worthwhile for media buffers which
- // can consume resource types whose overall pools aren't necessarily vast in
- // comparsion to resources consumed. Especially if a client is reconfiguring
- // buffers multiple times.
- //
- // Note that Sync() prior to allocating new media buffers is not alone
- // sufficient to achieve non-overlap of media buffer resource usage system
- // wide, but it can be a useful part of achieving that.
- //
- // The Sync() transits the Output ordering domain and the StreamControl
- // ordering domain, but not the InputData ordering domain. For more on
- // ordering domains see codec.md.
- //
- // This request can be used to avoid hitting kMaxInFlightStreams which is
- // presently 10. A client that stays <= 8 in-flight streams will comfortably
- // stay under the limit of 10. While the protocol permits repeated
- // SetInputBufferSettings() and the like, a client that spams the channel
- // can expect that the channel will just close if the server or the channel
- // itself gets too far behind.
- 11: Sync() -> ();
+ // Sync() -> ()
+ //
+ // On completion, all previous Codec calls have done what they're going to do
+ // server-side, _except_ for processing of data queued using
+ // QueueInputPacket().
+ //
+ // The main purpose of this call is to enable the client to wait until
+ // CloseCurrentStream() with release_input_buffers and/or
+ // release_output_buffers set to true to take effect, before the client
+ // allocates new buffers and re-sets-up input and/or output buffers. This
+ // de-overlapping of resource usage can be worthwhile for media buffers which
+ // can consume resource types whose overall pools aren't necessarily vast in
+ // comparsion to resources consumed. Especially if a client is reconfiguring
+ // buffers multiple times.
+ //
+ // Note that Sync() prior to allocating new media buffers is not alone
+ // sufficient to achieve non-overlap of media buffer resource usage system
+ // wide, but it can be a useful part of achieving that.
+ //
+ // The Sync() transits the Output ordering domain and the StreamControl
+ // ordering domain, but not the InputData ordering domain. For more on
+ // ordering domains see codec.md.
+ //
+ // This request can be used to avoid hitting kMaxInFlightStreams which is
+ // presently 10. A client that stays <= 8 in-flight streams will comfortably
+ // stay under the limit of 10. While the protocol permits repeated
+ // SetInputBufferSettings() and the like, a client that spams the channel
+ // can expect that the channel will just close if the server or the channel
+ // itself gets too far behind.
+ 11: Sync() -> ();
- // OnOutputPacket()
- //
- // This is how the codec emits an output packet to the codec client.
- //
- // Order is significant.
- //
- // The client should eventually call RecycleOutputPacket() (possibly after
- // switching streams multiple times), unless the buffer_lifetime_ordinal has
- // moved on. A stream change doesn't change which packets are busy with
- // the client vs. free with the server. See "packet lifetime" in codec.md for
- // more.
- //
- // The relevant buffer is buffer 0 if running in single-buffer mode, or the
- // buffer index is the same as packet_index if running in buffer-per-packet
- // mode.
- //
- // For low-level buffer types that support it, a Codec is free to emit an
- // output packet before the low-level buffer actually has any usable data in
- // the buffer, with the mechanism for signalling the presence of data
- // separate from the OnOutputPacket() message. For such low-level buffer
- // types, downstream consumers of data from the emitted packet must
- // participate in the low-level buffer signalling mechanism to know when it's
- // safe to consume the data. This is most likely to be relevant when using a
- // video decoder and gralloc-style buffers.
- //
- // The error_ bool(s) allow (but do not require) a Codec server to report
- // errors that happen during an AU or between AUs.
- //
- // The scope of error_detected_before starts at the end of the last delivered
- // output packet on this stream, or the start of stream if there were no
- // previous output packets on this stream. The scope ends at the start of
- // the output_packet.
- //
- // The error_detected_before bool is separate so that discontinuities can be
- // indicated separately from whether the current packet is damaged.
- //
- // The scope of error_detected_during is from the start to the end of this
- // output_packet.
- 12: -> OnOutputPacket(
- CodecPacket output_packet,
- bool error_detected_before,
- bool error_detected_during);
+ // OnOutputPacket()
+ //
+ // This is how the codec emits an output packet to the codec client.
+ //
+ // Order is significant.
+ //
+ // The client should eventually call RecycleOutputPacket() (possibly after
+ // switching streams multiple times), unless the buffer_lifetime_ordinal has
+ // moved on. A stream change doesn't change which packets are busy with
+ // the client vs. free with the server. See "packet lifetime" in codec.md for
+ // more.
+ //
+ // The relevant buffer is buffer 0 if running in single-buffer mode, or the
+ // buffer index is the same as packet_index if running in buffer-per-packet
+ // mode.
+ //
+ // For low-level buffer types that support it, a Codec is free to emit an
+ // output packet before the low-level buffer actually has any usable data in
+ // the buffer, with the mechanism for signalling the presence of data
+ // separate from the OnOutputPacket() message. For such low-level buffer
+ // types, downstream consumers of data from the emitted packet must
+ // participate in the low-level buffer signalling mechanism to know when it's
+ // safe to consume the data. This is most likely to be relevant when using a
+ // video decoder and gralloc-style buffers.
+ //
+ // The error_ bool(s) allow (but do not require) a Codec server to report
+ // errors that happen during an AU or between AUs.
+ //
+ // The scope of error_detected_before starts at the end of the last delivered
+ // output packet on this stream, or the start of stream if there were no
+ // previous output packets on this stream. The scope ends at the start of
+ // the output_packet.
+ //
+ // The error_detected_before bool is separate so that discontinuities can be
+ // indicated separately from whether the current packet is damaged.
+ //
+ // The scope of error_detected_during is from the start to the end of this
+ // output_packet.
+ 12: -> OnOutputPacket(
+ CodecPacket output_packet,
+ bool error_detected_before,
+ bool error_detected_during);
- // RecycleOutputPacket()
- //
- // After the client is done with an output packet, the client needs to tell
- // the codec that the output packet can be re-used for more output, via this
- // method.
- //
- // It's not permitted to recycle an output packet that's already free with the
- // codec server. It's permitted but discouraged for a client to recycle an
- // output packet that has been deallocated by an explicit or implicit output
- // buffer de-configuration(). See buffer_lifetime_ordinal for more on that.
- // A server must ignore any such stale RecycleOutputPacket() calls.
- //
- // For more on RecycleOutputPacket(), see codec.md.
- 13: RecycleOutputPacket(CodecPacketHeader available_output_packet);
+ // RecycleOutputPacket()
+ //
+ // After the client is done with an output packet, the client needs to tell
+ // the codec that the output packet can be re-used for more output, via this
+ // method.
+ //
+ // It's not permitted to recycle an output packet that's already free with the
+ // codec server. It's permitted but discouraged for a client to recycle an
+ // output packet that has been deallocated by an explicit or implicit output
+ // buffer de-configuration(). See buffer_lifetime_ordinal for more on that.
+ // A server must ignore any such stale RecycleOutputPacket() calls.
+ //
+ // For more on RecycleOutputPacket(), see codec.md.
+ 13: RecycleOutputPacket(CodecPacketHeader available_output_packet);
- // OnOutputEndOfStream()
- //
- // After QueueInputEndOfStream() is sent by the Codec client, within a
- // reasonable duration the corresponding OnOutputEndOfStream() will be sent by
- // the Codec server. Similar to QueueInputEndOfStream(),
- // OnOutputEndOfStream() is sent a maximum of once per stream.
- //
- // No more stream data for this stream will be sent after this message. All
- // input data for this stream was processed.
- //
- // While a Codec client is not required to QueueInputEndOfStream() (unless
- // the client wants to use FlushEndOfStreamAndCloseStream()), if a Codec
- // server receives QueueInputEndOfStream(), and the client hasn't closed the
- // stream, the Codec server must generate a corresponding
- // OnOutputEndOfStream() if nothing went wrong, or must send
- // OnStreamFailed(), or must close the server end of the Codec channel. An
- // ideal Codec server would handle and report stream errors via the error_
- // flags and complete stream processing without sending OnStreamFailed(), but
- // in any case, the above-listed options are the only ways that an
- // OnOutputEndOfStream() won't happen after QueueInputEndOfStream().
- //
- // There will be no more OnOutputPacket() or OnOutputConfig() messages for
- // this stream_lifetime_ordinal after this message - if a server doesn't
- // follow this rule, a client should close the Codec channel.
- //
- // The error_detected_before bool has the same semantics as the
- // error_detected_before bool in OnOutputPacket().
- 14: -> OnOutputEndOfStream(
- uint64 stream_lifetime_ordinal,
- bool error_detected_before);
+ // OnOutputEndOfStream()
+ //
+ // After QueueInputEndOfStream() is sent by the Codec client, within a
+ // reasonable duration the corresponding OnOutputEndOfStream() will be sent by
+ // the Codec server. Similar to QueueInputEndOfStream(),
+ // OnOutputEndOfStream() is sent a maximum of once per stream.
+ //
+ // No more stream data for this stream will be sent after this message. All
+ // input data for this stream was processed.
+ //
+ // While a Codec client is not required to QueueInputEndOfStream() (unless
+ // the client wants to use FlushEndOfStreamAndCloseStream()), if a Codec
+ // server receives QueueInputEndOfStream(), and the client hasn't closed the
+ // stream, the Codec server must generate a corresponding
+ // OnOutputEndOfStream() if nothing went wrong, or must send
+ // OnStreamFailed(), or must close the server end of the Codec channel. An
+ // ideal Codec server would handle and report stream errors via the error_
+ // flags and complete stream processing without sending OnStreamFailed(), but
+ // in any case, the above-listed options are the only ways that an
+ // OnOutputEndOfStream() won't happen after QueueInputEndOfStream().
+ //
+ // There will be no more OnOutputPacket() or OnOutputConfig() messages for
+ // this stream_lifetime_ordinal after this message - if a server doesn't
+ // follow this rule, a client should close the Codec channel.
+ //
+ // The error_detected_before bool has the same semantics as the
+ // error_detected_before bool in OnOutputPacket().
+ 14: -> OnOutputEndOfStream(
+ uint64 stream_lifetime_ordinal,
+ bool error_detected_before);
- //
- // Stream specific messages:
- //
+ //
+ // Stream specific messages:
+ //
- // QueueInputFormatDetails()
- //
- // If the input format details are still the same as specified during Codec
- // creation, this message is unnecessary and does not need to be sent.
- //
- // If the stream doesn't exist yet, this message creates the stream.
- //
- // All servers must permit QueueInputFormatDetails() at the start of a stream
- // without failing, as long as the new format is supported by the Codec
- // instance. Technically this allows for a server to only support the exact
- // input format set during Codec creation, and that is by design. A client
- // that tries to switch formats and gets a Codec channel failure should try
- // again one more time with a fresh Codec instance created with CodecFactory
- // using the new input format during creation, before giving up.
- //
- // These format details override the format details
- // specified during codec creation for this stream only. The next stream will
- // default back to the format details set during codec creation.
- //
- // For now, QueueInputFormatDetails() sent mid-stream will fail the Codec
- // channel. Clients shouldn't do this for now.
- //
- // This message is permitted at the start of the first stream (just like at
- // the start of any stream). The format specified need not match what was
- // specified during codec creation, but if it doesn't match, the Codec channel
- // might close as described above.
- 15: QueueInputFormatDetails(
- uint64 stream_lifetime_ordinal, CodecFormatDetails format_details);
+ // QueueInputFormatDetails()
+ //
+ // If the input format details are still the same as specified during Codec
+ // creation, this message is unnecessary and does not need to be sent.
+ //
+ // If the stream doesn't exist yet, this message creates the stream.
+ //
+ // All servers must permit QueueInputFormatDetails() at the start of a stream
+ // without failing, as long as the new format is supported by the Codec
+ // instance. Technically this allows for a server to only support the exact
+ // input format set during Codec creation, and that is by design. A client
+ // that tries to switch formats and gets a Codec channel failure should try
+ // again one more time with a fresh Codec instance created with CodecFactory
+ // using the new input format during creation, before giving up.
+ //
+ // These format details override the format details
+ // specified during codec creation for this stream only. The next stream will
+ // default back to the format details set during codec creation.
+ //
+ // For now, QueueInputFormatDetails() sent mid-stream will fail the Codec
+ // channel. Clients shouldn't do this for now.
+ //
+ // This message is permitted at the start of the first stream (just like at
+ // the start of any stream). The format specified need not match what was
+ // specified during codec creation, but if it doesn't match, the Codec channel
+ // might close as described above.
+ 15: QueueInputFormatDetails(
+ uint64 stream_lifetime_ordinal, CodecFormatDetails format_details);
- // QueueInputPacket()
- //
- // This message queues input data to the codec for processing.
- //
- // If the stream doesn't exist yet, this message creates the new stream.
- //
- // The client is required to be willing to send QueueInputPacket() prior to
- // the server's first OnOutputConfig(), and is permitted to start a new stream
- // without output buffers configured yet.
- //
- // The client must continue to deliver input data via this message even if the
- // codec has not yet generated the first OnOutputConfig(), and even if the
- // Codec is generating OnFreeInputPacket() for previously-queued input
- // packets. The input data must continue as long as there are free packets
- // to be assured that the server will ever generate the first
- // OnOutputConfig().
- //
- // For more on QueueInputPacket(), see codec.md.
- 16: QueueInputPacket(CodecPacket packet);
+ // QueueInputPacket()
+ //
+ // This message queues input data to the codec for processing.
+ //
+ // If the stream doesn't exist yet, this message creates the new stream.
+ //
+ // The client is required to be willing to send QueueInputPacket() prior to
+ // the server's first OnOutputConfig(), and is permitted to start a new stream
+ // without output buffers configured yet.
+ //
+ // The client must continue to deliver input data via this message even if the
+ // codec has not yet generated the first OnOutputConfig(), and even if the
+ // Codec is generating OnFreeInputPacket() for previously-queued input
+ // packets. The input data must continue as long as there are free packets
+ // to be assured that the server will ever generate the first
+ // OnOutputConfig().
+ //
+ // For more on QueueInputPacket(), see codec.md.
+ 16: QueueInputPacket(CodecPacket packet);
- // OnFreeInputPacket()
- //
- // The server sends this message when the codec is done consuming this packet
- // and the packet can be re-filled by the client.
- //
- // This is not sent for all packets when a new buffer_lifetime_ordinal starts
- // as in that case all the packets are initially free with the client.
- //
- // See comments on QueueInputBuffer() and "packet lifetime" in codec.md for
- // for description of buffer lifetime and packet lifetime.
- //
- // After receiving the available input buffer via this event, the codec
- // client can call later call QueueInputBuffer with appropriate offset and
- // length set.
- //
- // TODO(dustingreen): At the moment, there is no guarantee re. the order of
- // these messages with respect to the order of QueueInputPacket(), but at
- // least for decoders, it might be worthwhile to require that servers preserve
- // the order vs. QueueInputPacket(), to make it easier to feed input from a
- // ring buffer or similar. For audio encoders it might still make sense. For
- // video encoders probably not.
- 17: -> OnFreeInputPacket(CodecPacketHeader free_input_packet);
+ // OnFreeInputPacket()
+ //
+ // The server sends this message when the codec is done consuming this packet
+ // and the packet can be re-filled by the client.
+ //
+ // This is not sent for all packets when a new buffer_lifetime_ordinal starts
+ // as in that case all the packets are initially free with the client.
+ //
+ // See comments on QueueInputBuffer() and "packet lifetime" in codec.md for
+ // for description of buffer lifetime and packet lifetime.
+ //
+ // After receiving the available input buffer via this event, the codec
+ // client can call later call QueueInputBuffer with appropriate offset and
+ // length set.
+ //
+ // TODO(dustingreen): At the moment, there is no guarantee re. the order of
+ // these messages with respect to the order of QueueInputPacket(), but at
+ // least for decoders, it might be worthwhile to require that servers preserve
+ // the order vs. QueueInputPacket(), to make it easier to feed input from a
+ // ring buffer or similar. For audio encoders it might still make sense. For
+ // video encoders probably not.
+ 17: -> OnFreeInputPacket(CodecPacketHeader free_input_packet);
- // Inform the server that all QueueInputPacket() messages for this stream
- // have been sent.
- //
- // If the stream isn't closed first (by the client, or by OnStreamFailed(), or
- // Codec channel closing), there will later be a corresponding
- // OnOutputEndOfStream().
- //
- // The corresponding OnOutputEndOfStream() message will be generated only if
- // the server finishes processing the stream before the server sees the
- // client close the stream (such as by starting a new stream). A way to
- // force the server to finish the stream before closing is to use
- // FlushEndOfStreamAndCloseStream() after QueueInputEndOfStream() before any
- // new stream. Another way to force the server to finish the stream before
- // closing is to wait for the OnOutputEndOfStream() before taking any action
- // that closes the stream.
- //
- // In addition to serving as an "EndOfStream" marker to make it obvious
- // client-side when all input data has been processed, if a client never
- // sends QueueInputEndOfStream(), no amount of waiting will necessarily
- // result in all input data getting processed through to the output. Some
- // codecs have some internally-delayed data which only gets pushed through by
- // additional input data _or_ by this EndOfStream marker. In that sense,
- // this message can be viewed as a flush-through at InputData domain level,
- // but the flush-through only takes effect if the codec even gets that far
- // before the stream is just closed at StreamControl domain level. This
- // message is not alone sufficient to act as an overall flush-through at
- // StreamControl level. For that, send this message first and then send
- // FlushEndOfStreamAndCloseStream() (at which point it becomes possible to
- // queue input data for a new stream without causing discard of this older
- // stream's data), or wait for the OnOutputEndOfStream() before closing the
- // current stream.
- //
- // If a client sends QueueInputPacket(), QueueInputFormatDetails(),
- // QueueInputEndOfStream() for this stream after the first
- // QueueInputEndOfStream() for this stream, a server should close the Codec
- // channel.
- 18: QueueInputEndOfStream(uint64 stream_lifetime_ordinal);
+ // Inform the server that all QueueInputPacket() messages for this stream
+ // have been sent.
+ //
+ // If the stream isn't closed first (by the client, or by OnStreamFailed(), or
+ // Codec channel closing), there will later be a corresponding
+ // OnOutputEndOfStream().
+ //
+ // The corresponding OnOutputEndOfStream() message will be generated only if
+ // the server finishes processing the stream before the server sees the
+ // client close the stream (such as by starting a new stream). A way to
+ // force the server to finish the stream before closing is to use
+ // FlushEndOfStreamAndCloseStream() after QueueInputEndOfStream() before any
+ // new stream. Another way to force the server to finish the stream before
+ // closing is to wait for the OnOutputEndOfStream() before taking any action
+ // that closes the stream.
+ //
+ // In addition to serving as an "EndOfStream" marker to make it obvious
+ // client-side when all input data has been processed, if a client never
+ // sends QueueInputEndOfStream(), no amount of waiting will necessarily
+ // result in all input data getting processed through to the output. Some
+ // codecs have some internally-delayed data which only gets pushed through by
+ // additional input data _or_ by this EndOfStream marker. In that sense,
+ // this message can be viewed as a flush-through at InputData domain level,
+ // but the flush-through only takes effect if the codec even gets that far
+ // before the stream is just closed at StreamControl domain level. This
+ // message is not alone sufficient to act as an overall flush-through at
+ // StreamControl level. For that, send this message first and then send
+ // FlushEndOfStreamAndCloseStream() (at which point it becomes possible to
+ // queue input data for a new stream without causing discard of this older
+ // stream's data), or wait for the OnOutputEndOfStream() before closing the
+ // current stream.
+ //
+ // If a client sends QueueInputPacket(), QueueInputFormatDetails(),
+ // QueueInputEndOfStream() for this stream after the first
+ // QueueInputEndOfStream() for this stream, a server should close the Codec
+ // channel.
+ 18: QueueInputEndOfStream(uint64 stream_lifetime_ordinal);
};
