| # Updating the VMO file format |
| |
| This document describes how to update or extend the [Component Inspection File |
| Format][inspect-file-format]. |
| |
| While extending the format, it is imperative that you don't break any existing functionality, in |
| particular, the Inspect readers and validation tests. However, packing all changes into a single |
| change can be overwhelming to review. In general, there should be 5-9 changes or steps in a chain |
| to alter the VMO format: |
| |
| 1. [(Possibly not applicable) Choose a type number by updating the VMO Format Docs](#choose-type). |
| 1. [Update the Rust reader.](#update-rust-reader) |
| 1. [Update the C++ reader.](#update-cpp-reader) |
| 1. [Update the Rust writer.](#update-rust-writer) |
| 1. [Update the C++ writer.](#update-cpp-writer) |
| 1. [(Potentially) Update the validator.](#update-validator-tests) |
| 1. Update the documentation. |
| 1. (Not a change)(Optional) Send a feature announcement. |
| |
| ## Choosing a type number {#choose-type} |
| |
| View the type table in the [Inspect file format][inspect-file-format] and choose an |
| available type number. There are a total of 256 possible types in the current specification. |
| |
| In order to reserve the new type, update the [Inspect file format][inspect-file-format]. |
| |
| ## Implementation |
| |
| It is cumbersome to test a reader or writer without the other and difficult to get the proper |
| block-level API without both a reader and writer. |
| |
| To test a reader and writer: |
| |
| 1. Pick a language and design and implement the feature entirely in that language, using unit |
| tests to model actual usage of the API. |
| |
| 1. Split the changes into separate reader and writer changes, stacking the writer on top of |
| the reader. |
| |
| At this point, tests in the reader that rely on the writer API are probably broken |
| in the reader change. |
| |
| 1. Rebase into that change and rewrite the tests (keeping around the original version of the tests) |
| using lower level functionality. |
| |
| Typically, you can put all changes to block-code in the reader change, making it possible but |
| messy to write tests for the reader API. |
| |
| 1. The test will be ugly. Rebase into the writer change and remove the modified tests, replacing |
| them with the original tests that are written in terms of the high level API. |
| |
| 1. Use the two changes for reference and duplicate them in the second language. |
| |
| 1. (Optional) Depending on the contents of your change, you may have to update the |
| [validator][validator-tests] tests as you go, since changes to the existing format of existing |
| blocks will likely break them. |
| |
| The following sections outline how to put together each change, but in practice, take these as hints |
| for designing the whole system cohesively before splitting the design as described above. |
| |
| ## Update the Rust implementation |
| |
| The examples in this section create a new type called `MyFoo`. |
| |
| ### Set up |
| |
| 1. Include tests: |
| |
| ``` |
| fx set core.x64 --with //src/lib/diagnostics:tests |
| ``` |
| |
| 1. Run tests: |
| |
| Note: The `inspect-format-tests` are the VMO block format tests, and `fuchsia-inspect-tests` are |
| the core library tests. |
| |
| ``` |
| fx test inspect-format-tests fuchsia-inspect-tests |
| ``` |
| |
| ### Reader change {#update-rust-reader} |
| |
| #### Bitfield updates |
| |
| 1. If defining a new Block Type, update [the `BlockType` enum][block-type-rs]. |
| |
| Note: This must be the same as the VMO format docs and the C++ implementation. |
| 1. Update the methods and functions defined for `BlockType`. |
| |
| 1. If changing the fields in an existing block or creating a new `BlockType`, update |
| [the bitfield layout][bitfield-rs]. |
| |
| 1. Run `inspect-format-tests` to verify changes compile: |
| |
| ``` |
| fx test inspect-format-tests |
| ``` |
| |
| 1. Update [the block definition][block-rs] to include methods for reading and writing the new |
| fields. It is appropriate to include write-functionality at this time in the block library; |
| it is next to impossible to write a reader test without it. |
| |
| Note: Use `#[cfg(test)]` if the function is currently test-only and only used within the |
| current crate. Otherwise, use `#[doc(hidden)]` to elide the documentation from public view |
| for now. This will keep it out of the crate public API. |
| |
| 1. Write block tests that exercise the new functionality. |
| |
| 1. Write tests that make assertions on the first 8-16 bytes of the block. |
| |
| Note: The slice should be formatted as little endian. |
| |
| Typically this means writing the expected contents as a `&[u8]` that |
| contains hex values and asserting its equivalence to the buffer the block is using as a |
| container. |
| |
| #### Update the reader |
| |
| You can find the reader code in [mod.rs][reader-mod-rs]. |
| The tests in this change will probably be tricky, because the high-level API writer doesn't exist |
| yet. |
| |
| ### Writer change {#update-rust-writer} |
| |
| #### State |
| |
| The primary changes here will be in [the `State` functionality][state-rs]. |
| This is where blocks can be allocated and converted into the new type. |
| If all you're doing is modifying an existing block, this is likely the only place you need |
| to make changes. |
| |
| #### Creating a new value type |
| |
| There is a [types directory][types-dir] where you can add a new file for your type. |
| |
| 1. Create the new type in the file created. Use an existing type as an example. Types always |
| have access to internal `State`. Use this to create the necessary |
| methods on your new type, calling into the methods created in [`State`][state-rs]. |
| |
| 1. Add a method to [`Node`][node-rs] for creating the new type. |
| |
| 1. Ensure that your type has RAII semantics in the VMO. If your type is |
| a value, this is probably done automatically by the boilerplate copied from an existing |
| type in step 1. |
| |
| Finally, go back and update the tests from the Reader change to use the new API! |
| |
| ## Update the C++ implementation |
| |
| The examples in this section create a new type called `MyFoo`. |
| |
| As noted above, this section should be two changes in Gerrit. |
| |
| ### Set up |
| |
| 1. Include tests: |
| |
| Note: You can also use Fuchsia products other than `core.x64`. |
| |
| ``` |
| fx set core.x64 --with //zircon/system/ulib/inspect:tests |
| ``` |
| |
| 1. Run tests. |
| |
| ``` |
| fx test inspect-cpp-unittest |
| ``` |
| |
| ### Reader change {#update-cpp-reader} |
| |
| #### Bitfield updates |
| |
| This section describes how to define the bitfields for your new type. |
| |
| Update [the block definition][block-header]. |
| |
| 1. Change `BlockType` to include your new type. For example: `kMyFoo = #;` |
| |
| Note: This must be the same as the VMO format docs and the Rust implementation. |
| |
| 1. If your type needs a new header (typically if it is not a `VALUE`), |
| define the header bitfields for your type with a struct. For example: |
| `struct MyFooBlockFields final : public BlockFields`. |
| |
| 1. If your type needs a new payload (it requires using the second 8 bytes of the block), |
| define the payload bitfields for your type with a struct. For example: |
| `struct MyFooBlockPayload final`. |
| |
| 1. If your type contains enums (such as format), |
| define a new enum at the top of block.h. For example: `enum class MyFooBlockFormat : uint8_t`. |
| |
| #### Implement the type reader |
| |
| This section describes how to make your new type readable. |
| |
| Update [the Inspect hierarchy][hierarchy-header] based on your type: |
| |
| * {A value (child of `Node`)} |
| 1. Update `PropertyFormat` enum with a new number for your type. This must be sequential in |
| this specific enum and does not need to match the format type ordinal you chose. |
| |
| 1. Create a new value type. For example, |
| `using MyFooValue = internal::Value<T, static_cast<size_t>(PropertyFormat::kMyFoo)>;` |
| |
| 1. Update `PropertyValue` variant with the new value. |
| Note: The index in `fit::internal::variant` must match the value of `PropertyFormat`. |
| |
| * {Not a value} |
| 1. You need to make your own in-memory representation objects in the |
| [hierarchy][hierarchy-header] file. |
| |
| 1. Update the [actual reader][reader-cc]. |
| |
| 1. Update `InnerScanBlocks` to dispatch your type. If you are creating a new `Property`, you may |
| only have to add your `BlockType`. |
| |
| 1. If you need a custom parser, implement `InnerParseMyFoo`, |
| which takes a parent (if needed) and the pointer to the scanned block. |
| |
| ### Writer change {#update-cpp-writer} |
| |
| #### Type wrapper declaration |
| |
| This section describes how to declare a C++ RAII-style wrapper for your new type. |
| |
| Type wrappers contain indices of blocks that are owned by the type. You are responsible for |
| implementing operations on those blocks, including creation and deletion, in |
| [State action updates](#state-action). |
| |
| Update [the writer types definition][types-header]. |
| |
| Determine if you can reuse an existing wrapper or if you need a bespoke type: |
| |
| * {Reuse} |
| 1. If you need to support Add, Subtract, and Set: `using MyFoo = internal::NumericProperty<T>`, |
| where `T` is the argument type to those operations. |
| |
| 1. If you need to support Set: `using MyFoo = internal::Property<T>`, where `T` is the argument |
| type to Set. |
| |
| 1. If you need to support numeric operations on an array: |
| `using MyFood = internal::ArrayProperty<T>`, where `T` is the argument type for slots in the |
| array. |
| |
| 1. If you need to support inserting to a histogram: |
| `using MyFoo = internal::{Linear,Exponential}Histogram<T>`, where `T` is the argument to |
| Insert. |
| |
| * {Bespoke} |
| 1. Create a new type wrapper. For example `class MyFoo final`. |
| |
| 1. Ensure your class has `internal::State` as a friend class. Note: See `class Link` for a |
| copyable starting point. |
| |
| #### State action updates {#state-action} |
| |
| The `State` class is the actual implementation for all operations on all types. This section |
| describes how to implement the operations you will need to complete your wrapper implementation. |
| |
| 1. Update [`State` header][state-header]: |
| 1. Add Create and Free methods. For example: |
| `MyFoo CreateMyFoo(<args>); void FreeMyFoo(MyFoo* property);` where `args` typically |
| includes name, parent, and some initial value. |
| |
| 1. Add methods for each operation you need to support on your type. For example, if your type |
| can be Set, `void SetMyFoo(MyFoo* property, T)`, where `T` is the same type from your |
| update to types.h. |
| |
| 1. Update [`State`][state-cc]: |
| |
| Note: Always lock the state before accessing any internal data, using |
| `std::lock_guard<std::mutex> lock(mutex_);`. |
| |
| Note: Always lock the buffer before making any modifications to blocks, using |
| `AutoGenerationIncrement gen(header_, heap_.get());`. |
| |
| 1. Implement your new type's methods. The implementation varies between the different types. |
| This section provides a high-level overview of what each method must do: |
| |
| * `MyFoo CreateMyFoo(Args...)` is responsible for allocating a number of blocks, setting |
| their values, and returning them wrapped in a `MyFoo`. You may use a private constructor |
| to create `MyFoo` from the `BlockIndex` objects it wraps. Various internal helpers exist |
| to simplify this operation. See `CreateIntProperty` for an example. |
| |
| * `void FreeMyFoo(MyFoo* property)` is responsible for freeing all blocks wrapped by the |
| `MyFoo`. There are sometimes particular ordering requirements or updates necessary for |
| freeing blocks. See `InnerFreeValue` for an example of how values are freed. |
| |
| * Operations, such as `void SetMyFoo(MyFoo* property, T value)` change the value of blocks |
| allocated to `MyFoo` to implement the operation. See `SetIntProperty` for an example. |
| |
| |
| #### Implement the type wrapper |
| |
| This section describes how to implement the wrapper methods declared previously. |
| |
| 1. Update [the writer type definitions][types-cc]: |
| * If you used an existing templated type, you need to override each method for your new base |
| type `T`. For example, if you typed `using MyFoo = internal::Property<T>`, you will write: |
| `template<> void internal::Property<T>::OPERATION(...) { ... }` |
| |
| * If you created your own type, simply create definitions for the methods you declared. You need |
| to do the following: |
| * Make your constructor call `state_->CreateMyFoo(...);` |
| |
| * Make your destructor call `state_->FreeMyFoo(...);` |
| |
| * Make your other methods call the corresponding implementation on State. |
| |
| * Have all your constructors and methods check that `state_` is not null before calling. |
| |
| #### Implement tests |
| |
| 1. Update [state unit tests][state-unittest-cc] with tests for your low-level operations. |
| |
| 1. Update [reader unit tests][reader-unittest-cc] with tests for your high-level reader |
| implementation. |
| |
| |
| ## Example change chain |
| |
| 1. [C++ Reader](https://fuchsia-review.googlesource.com/c/fuchsia/+/603056/6) |
| 1. [Rust Reader](https://fuchsia-review.googlesource.com/c/fuchsia/+/599946/14) |
| 1. [Rust Writer](https://fuchsia-review.googlesource.com/c/fuchsia/+/599947) |
| 1. [Rust Validator Changes][rust-puppet-validator] |
| 1. [C++ Validator Changes][cpp-puppet-validator] |
| 1. [Documentation Updates][document-updates] |
| |
| <!-- xrefs --> |
| [update-this-doc-bug]: https://bugs.fuchsia.dev/p/fuchsia/issues/detail?id=43131 |
| [inspect-file-format]: /docs/reference/platform-spec/diagnostics/inspect-vmo-format.md |
| [bitfield-rs]: /src/lib/diagnostics/inspect/format/rust/src/bitfields.rs |
| [block-type-rs]: /src/lib/diagnostics/inspect/format/rust/src/block_type.rs |
| [block-rs]: /src/lib/diagnostics/inspect/format/rust/src/block.rs |
| [reader-mod-rs]: /src/lib/diagnostics/inspect/rust/src/reader/mod.rs |
| [state-rs]: /src/lib/diagnostics/inspect/rust/src/writer/state.rs?q=state.rs |
| [types-dir]: /src/lib/diagnostics/inspect/rust/src/writer/types/ |
| [node-rs]: /src/lib/diagnostics/inspect/rust/src/writer/types/node.rs |
| [block-header]: /zircon/system/ulib/inspect/include/lib/inspect/cpp/vmo/block.h |
| [hierarchy-header]: /zircon/system/ulib/inspect/include/lib/inspect/cpp/hierarchy.h |
| [reader-cc]: /zircon/system/ulib/inspect/reader.cc |
| [reader-unittest-cc]: /zircon/system/ulib/inspect/tests/reader_unittest.cc |
| [state-cc]: /zircon/system/ulib/inspect/vmo/state.cc |
| [state-header]: /zircon/system/ulib/inspect/include/lib/inspect/cpp/vmo/state.h |
| [state-unittest-cc]: /zircon/system/ulib/inspect/tests/state_unittest.cc |
| [types-cc]: /zircon/system/ulib/inspect/vmo/types.cc |
| [types-header]: /zircon/system/ulib/inspect/include/lib/inspect/cpp/vmo/types.h |
| [inspect-vmo]: /docs/reference/platform-spec/diagnostics/inspect-vmo-format.md |
| [validator-tests]: /src/diagnostics/validator/inspect/src |
| [rust-puppet-validator]: https://fuchsia-review.googlesource.com/c/fuchsia/+/643054/ |
| [cpp-puppet-validator]: https://fuchsia-review.googlesource.com/c/fuchsia/+/642322/ |
| [document-updates]: https://fuchsia-review.googlesource.com/c/fuchsia/+/642401/ |
| |