| // Copyright 2018-2021 The Khronos Group, Inc. |
| // |
| // SPDX-License-Identifier: CC-BY-4.0 |
| |
| include::{generated}/meta/{refprefix}VK_EXT_image_drm_format_modifier.txt[] |
| |
| === Other Extension Metadata |
| |
| *Last Modified Date*:: |
| 2021-09-30 |
| *IP Status*:: |
| No known IP claims. |
| *Contributors*:: |
| - Antoine Labour, Google |
| - Bas Nieuwenhuizen, Google |
| - Chad Versace, Google |
| - James Jones, NVIDIA |
| - Jason Ekstrand, Intel |
| - Jőrg Wagner, ARM |
| - Kristian Høgsberg Kristensen, Google |
| - Ray Smith, ARM |
| |
| === Description |
| |
| This extension provides the ability to use _DRM format modifiers_ with |
| images, enabling Vulkan to better integrate with the Linux ecosystem of |
| graphics, video, and display APIs. |
| |
| Its functionality closely overlaps with |
| `EGL_EXT_image_dma_buf_import_modifiers`^<<VK_EXT_image_drm_format_modifier-fn2,2>>^ |
| and |
| `EGL_MESA_image_dma_buf_export`^<<VK_EXT_image_drm_format_modifier-fn3,3>>^. |
| Unlike the EGL extensions, this extension does not require the use of a |
| specific handle type (such as a dma_buf) for external memory and provides |
| more explicit control of image creation. |
| |
| === Introduction to DRM Format Modifiers |
| |
| A _DRM format modifier_ is a 64-bit, vendor-prefixed, semi-opaque unsigned |
| integer. |
| Most _modifiers_ represent a concrete, vendor-specific tiling format for |
| images. |
| Some exceptions are etext:DRM_FORMAT_MOD_LINEAR (which is not |
| vendor-specific); etext:DRM_FORMAT_MOD_NONE (which is an alias of |
| etext:DRM_FORMAT_MOD_LINEAR due to historical accident); and |
| etext:DRM_FORMAT_MOD_INVALID (which does not represent a tiling format). |
| The _modifier's_ vendor prefix consists of the 8 most significant bits. |
| The canonical list of _modifiers_ and vendor prefixes is found in |
| https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h[`drm_fourcc.h`] |
| in the Linux kernel source. |
| The other dominant source of _modifiers_ are vendor kernel trees. |
| |
| One goal of _modifiers_ in the Linux ecosystem is to enumerate for each |
| vendor a reasonably sized set of tiling formats that are appropriate for |
| images shared across processes, APIs, and/or devices, where each |
| participating component may possibly be from different vendors. |
| A non-goal is to enumerate all tiling formats supported by all vendors. |
| Some tiling formats used internally by vendors are inappropriate for |
| sharing; no _modifiers_ should be assigned to such tiling formats. |
| |
| Modifier values typically do not _describe_ memory layouts. |
| More precisely, a _modifier_'s lower 56 bits usually have no structure. |
| Instead, modifiers _name_ memory layouts; they name a small set of |
| vendor-preferred layouts for image sharing. |
| As a consequence, in each vendor namespace the modifier values are often |
| sequentially allocated starting at 1. |
| |
| Each _modifier_ is usually supported by a single vendor and its name matches |
| the pattern `{VENDOR}_FORMAT_MOD_*` or `DRM_FORMAT_MOD_{VENDOR}_*`. |
| Examples are etext:I915_FORMAT_MOD_X_TILED and |
| etext:DRM_FORMAT_MOD_BROADCOM_VC4_T_TILED. |
| An exception is etext:DRM_FORMAT_MOD_LINEAR, which is supported by most |
| vendors. |
| |
| Many APIs in Linux use _modifiers_ to negotiate and specify the memory |
| layout of shared images. |
| For example, a Wayland compositor and Wayland client may, by relaying |
| _modifiers_ over the Wayland protocol `zwp_linux_dmabuf_v1`, negotiate a |
| vendor-specific tiling format for a shared code:wl_buffer. |
| The client may allocate the underlying memory for the code:wl_buffer with |
| GBM, providing the chosen _modifier_ to code:gbm_bo_create_with_modifiers. |
| The client may then import the code:wl_buffer into Vulkan for producing |
| image content, providing the resource's dma_buf to |
| slink:VkImportMemoryFdInfoKHR and its _modifier_ to |
| slink:VkImageDrmFormatModifierExplicitCreateInfoEXT. |
| The compositor may then import the code:wl_buffer into OpenGL for sampling, |
| providing the resource's dma_buf and _modifier_ to code:eglCreateImage. |
| The compositor may also bypass OpenGL and submit the code:wl_buffer directly |
| to the kernel's display API, providing the dma_buf and _modifier_ through |
| code:drm_mode_fb_cmd2. |
| |
| === Format Translation |
| |
| _Modifier_-capable APIs often pair _modifiers_ with DRM formats, which are |
| defined in |
| https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h[`drm_fourcc.h`]. |
| However, `VK_EXT_image_drm_format_modifier` uses elink:VkFormat instead of |
| DRM formats. |
| The application must convert between elink:VkFormat and DRM format when it |
| sends or receives a DRM format to or from an external API. |
| |
| The mapping from elink:VkFormat to DRM format is lossy. |
| Therefore, when receiving a DRM format from an external API, often the |
| application must use information from the external API to accurately map the |
| DRM format to a elink:VkFormat. |
| For example, DRM formats do not distinguish between RGB and sRGB (as of |
| 2018-03-28); external information is required to identify the image's |
| colorspace. |
| |
| The mapping between elink:VkFormat and DRM format is also incomplete. |
| For some DRM formats there exist no corresponding Vulkan format, and for |
| some Vulkan formats there exist no corresponding DRM format. |
| |
| === Usage Patterns |
| |
| Three primary usage patterns are intended for this extension: |
| |
| * *Negotiation.* The application negotiates with _modifier_-aware, |
| external components to determine sets of image creation parameters |
| supported among all components. |
| + |
| -- |
| In the Linux ecosystem, the negotiation usually assumes the image is a 2D, |
| single-sampled, non-mipmapped, non-array image; this extension permits that |
| assumption but does not require it. |
| The result of the negotiation usually resembles a set of tuples such as |
| _(drmFormat, drmFormatModifier)_, where each participating component |
| supports all tuples in the set. |
| |
| Many details of this negotiation—such as the protocol used during |
| negotiation, the set of image creation parameters expressable in the |
| protocol, and how the protocol chooses which process and which API will |
| create the image—are outside the scope of this specification. |
| |
| In this extension, flink:vkGetPhysicalDeviceFormatProperties2 with |
| slink:VkDrmFormatModifierPropertiesListEXT serves a primary role during the |
| negotiation, and flink:vkGetPhysicalDeviceImageFormatProperties2 with |
| slink:VkPhysicalDeviceImageDrmFormatModifierInfoEXT serves a secondary role. |
| -- |
| |
| * *Import.* The application imports an image with a _modifier_. |
| + |
| -- |
| In this pattern, the application receives from an external source the |
| image's memory and its creation parameters, which are often the result of |
| the negotiation described above. |
| Some image creation parameters are implicitly defined by the external |
| source; for example, ename:VK_IMAGE_TYPE_2D is often assumed. |
| Some image creation parameters are usually explicit, such as the image's |
| pname:format, pname:drmFormatModifier, and pname:extent; and each plane's |
| pname:offset and pname:rowPitch. |
| |
| Before creating the image, the application first verifies that the physical |
| device supports the received creation parameters by querying |
| flink:vkGetPhysicalDeviceFormatProperties2 with |
| slink:VkDrmFormatModifierPropertiesListEXT and |
| flink:vkGetPhysicalDeviceImageFormatProperties2 with |
| slink:VkPhysicalDeviceImageDrmFormatModifierInfoEXT. |
| Then the application creates the image by chaining |
| slink:VkImageDrmFormatModifierExplicitCreateInfoEXT and |
| slink:VkExternalMemoryImageCreateInfo onto slink:VkImageCreateInfo. |
| -- |
| |
| * *Export.* The application creates an image and allocates its memory. |
| Then the application exports to _modifier_-aware consumers the image's |
| memory handles; its creation parameters; its _modifier_; and the |
| <<VkSubresourceLayout,pname:offset>>, |
| <<VkSubresourceLayout,pname:size>>, and |
| <<VkSubresourceLayout,pname:rowPitch>> of each _memory plane_. |
| + |
| -- |
| In this pattern, the Vulkan device is the authority for the image; it is the |
| allocator of the image's memory and the decider of the image's creation |
| parameters. |
| When choosing the image's creation parameters, the application usually |
| chooses a tuple _(format, drmFormatModifier)_ from the result of the |
| negotiation described above. |
| The negotiation's result often contains multiple tuples that share the same |
| format but differ in their _modifier_. |
| In this case, the application should defer the choice of the image's |
| _modifier_ to the Vulkan implementation by providing all such _modifiers_ to |
| slink:VkImageDrmFormatModifierListCreateInfoEXT::pname:pDrmFormatModifiers; |
| and the implementation should choose from pname:pDrmFormatModifiers the |
| optimal _modifier_ in consideration with the other image parameters. |
| |
| The application creates the image by chaining |
| slink:VkImageDrmFormatModifierListCreateInfoEXT and |
| slink:VkExternalMemoryImageCreateInfo onto slink:VkImageCreateInfo. |
| The protocol and APIs by which the application will share the image with |
| external consumers will likely determine the value of |
| slink:VkExternalMemoryImageCreateInfo::pname:handleTypes. |
| The implementation chooses for the image an optimal _modifier_ from |
| slink:VkImageDrmFormatModifierListCreateInfoEXT::pname:pDrmFormatModifiers. |
| The application then queries the implementation-chosen _modifier_ with |
| flink:vkGetImageDrmFormatModifierPropertiesEXT, and queries the memory |
| layout of each plane with flink:vkGetImageSubresourceLayout. |
| |
| The application then allocates the image's memory with |
| slink:VkMemoryAllocateInfo, adding chained extending structures for external |
| memory; binds it to the image; and exports the memory, for example, with |
| flink:vkGetMemoryFdKHR. |
| |
| Finally, the application sends the image's creation parameters, its |
| _modifier_, its per-plane memory layout, and the exported memory handle to |
| the external consumers. |
| The details of how the application transmits this information to external |
| consumers is outside the scope of this specification. |
| -- |
| |
| === Prior Art |
| |
| Extension |
| `EGL_EXT_image_dma_buf_import`^<<VK_EXT_image_drm_format_modifier-fn1,1>>^ |
| introduced the ability to create an code:EGLImage by importing for each |
| plane a dma_buf, offset, and row pitch. |
| |
| Later, extension |
| `EGL_EXT_image_dma_buf_import_modifiers`^<<VK_EXT_image_drm_format_modifier-fn2,2>>^ |
| introduced the ability to query which combination of formats and _modifiers_ |
| the implementation supports and to specify _modifiers_ during creation of |
| the code:EGLImage. |
| |
| Extension |
| `EGL_MESA_image_dma_buf_export`^<<VK_EXT_image_drm_format_modifier-fn3,3>>^ |
| is the inverse of `EGL_EXT_image_dma_buf_import_modifiers`. |
| |
| The Linux kernel modesetting API (KMS), when configuring the display's |
| framebuffer with `struct |
| drm_mode_fb_cmd2`^<<VK_EXT_image_drm_format_modifier-fn4,4>>^, allows one to |
| specify the frambuffer's _modifier_ as well as a per-plane memory handle, |
| offset, and row pitch. |
| |
| GBM, a graphics buffer manager for Linux, allows creation of a `gbm_bo` |
| (that is, a graphics _buffer object_) by importing data similar to that in |
| `EGL_EXT_image_dma_buf_import_modifiers`^<<VK_EXT_image_drm_format_modifier-fn1,1>>^; |
| and symmetrically allows exporting the same data from the `gbm_bo`. |
| See the references to _modifier_ and _plane_ in |
| `gbm.h`^<<VK_EXT_image_drm_format_modifier-fn5,5>>^. |
| |
| include::{generated}/interfaces/VK_EXT_image_drm_format_modifier.txt[] |
| |
| === Issues |
| |
| 1) Should this extension define a single DRM format modifier per |
| sname:VkImage? Or define one per plane? |
| + |
| -- |
| *RESOLVED*: There exists a single DRM format modifier per sname:VkImage. |
| |
| *DISCUSSION*: Prior art, such as |
| `EGL_EXT_image_dma_buf_import_modifiers`^<<VK_EXT_image_drm_format_modifier-fn2,2>>^, |
| `struct drm_mode_fb_cmd2`^<<VK_EXT_image_drm_format_modifier-fn4,4>>^, and |
| `struct |
| gbm_import_fd_modifier_data`^<<VK_EXT_image_drm_format_modifier-fn5,5>>^, |
| allows defining one _modifier_ per plane. |
| However, developers of the GBM and kernel APIs concede it was a mistake. |
| Beginning in Linux 4.10, the kernel requires that the application provide |
| the same DRM format _modifier_ for each plane. |
| (See Linux commit |
| https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bae781b259269590109e8a4a8227331362b88212[bae781b259269590109e8a4a8227331362b88212]). |
| And GBM provides an entry point, code:gbm_bo_get_modifier, for querying the |
| _modifier_ of the image but does not provide one to query the modifier of |
| individual planes. |
| -- |
| |
| 2) When creating an image with |
| slink:VkImageDrmFormatModifierExplicitCreateInfoEXT, which is typically used |
| when _importing_ an image, should the application explicitly provide the |
| size of each plane? |
| + |
| -- |
| *RESOLVED*: No. |
| The application must: not provide the size. |
| To enforce this, the API requires that |
| slink:VkImageDrmFormatModifierExplicitCreateInfoEXT::pname:pPlaneLayouts->size |
| must: be 0. |
| |
| *DISCUSSION*: Prior art, such as |
| `EGL_EXT_image_dma_buf_import_modifiers`^<<VK_EXT_image_drm_format_modifier-fn2,2>>^, |
| `struct drm_mode_fb_cmd2`^<<VK_EXT_image_drm_format_modifier-fn4,4>>^, and |
| `struct |
| gbm_import_fd_modifier_data`^<<VK_EXT_image_drm_format_modifier-fn5,5>>^, |
| omits from the API the size of each plane. |
| Instead, the APIs infer each plane's size from the import parameters, which |
| include the image's pixel format and a dma_buf, offset, and row pitch for |
| each plane. |
| |
| However, Vulkan differs from EGL and GBM with regards to image creation in |
| the following ways: |
| |
| .Differences in Image Creation |
| |
| - *Undedicated allocation by default.* When importing or exporting a set |
| of dma_bufs as an code:EGLImage or code:gbm_bo, common practice mandates |
| that each dma_buf's memory be dedicated (in the sense of |
| `VK_KHR_dedicated_allocation`) to the image (though not necessarily |
| dedicated to a single plane). |
| In particular, neither the GBM documentation nor the EGL extension |
| specifications explicitly state this requirement, but in light of common |
| practice this is likely due to under-specification rather than |
| intentional omission. |
| In contrast, `VK_EXT_image_drm_format_modifier` permits, but does not |
| require, the implementation to require dedicated allocations for images |
| created with ename:VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. |
| |
| - *Separation of image creation and memory allocation.* When importing a |
| set of dma_bufs as an code:EGLImage or code:gbm_bo, EGL and GBM create |
| the image resource and bind it to memory (the dma_bufs) simultaneously. |
| This allows EGL and GBM to query each dma_buf's size during image |
| creation. |
| In Vulkan, image creation and memory allocation are independent unless a |
| dedicated allocation is used (as in `VK_KHR_dedicated_allocation`). |
| Therefore, without requiring dedicated allocation, Vulkan cannot query |
| the size of each dma_buf (or other external handle) when calculating the |
| image's memory layout. |
| Even if dedication allocation were required, Vulkan cannot calculate the |
| image's memory layout until after the image is bound to its dma_ufs. |
| |
| The above differences complicate the potential inference of plane size in |
| Vulkan. |
| Consider the following problematic cases: |
| |
| .Problematic Plane Size Calculations |
| |
| - *Padding.* Some plane of the image may require implementation-dependent |
| padding. |
| |
| - *Metadata.* For some _modifiers_, the image may have a metadata plane |
| which requires a non-trivial calculation to determine its size. |
| |
| - *Mipmapped, array, and 3D images.* The implementation may support |
| ename:VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT for images whose |
| pname:mipLevels, pname:arrayLayers, or pname:depth is greater than 1. |
| For such images with certain _modifiers_, the calculation of each |
| plane's size may be non-trivial. |
| |
| However, an application-provided plane size solves none of the above |
| problems. |
| |
| For simplicity, consider an external image with a single memory plane. |
| The implementation is obviously capable calculating the image's size when |
| its tiling is ename:VK_IMAGE_TILING_OPTIMAL. |
| Likewise, any reasonable implementation is capable of calculating the |
| image's size when its tiling uses a supported _modifier_. |
| |
| Suppose that the external image's size is smaller than the |
| implementation-calculated size. |
| If the application provided the external image's size to |
| flink:vkCreateImage, the implementation would observe the mismatched size |
| and recognize its inability to comprehend the external image's layout |
| (unless the implementation used the application-provided size to select a |
| refinement of the tiling layout indicated by the _modifier_, which is |
| strongly discouraged). |
| The implementation would observe the conflict, and reject image creation |
| with ename:VK_ERROR_INVALID_DRM_FORMAT_MODIFIER_PLANE_LAYOUT_EXT. |
| On the other hand, if the application did not provide the external image's |
| size to flink:vkCreateImage, then the application would observe after |
| calling flink:vkGetImageMemoryRequirements that the external image's size is |
| less than the size required by the implementation. |
| The application would observe the conflict and refuse to bind the |
| sname:VkImage to the external memory. |
| In both cases, the result is explicit failure. |
| |
| Suppose that the external image's size is larger than the |
| implementation-calculated size. |
| If the application provided the external image's size to |
| flink:vkCreateImage, for reasons similar to above the implementation would |
| observe the mismatched size and recognize its inability to comprehend the |
| image data residing in the extra size. |
| The implementation, however, must assume that image data resides in the |
| entire size provided by the application. |
| The implementation would observe the conflict and reject image creation with |
| ename:VK_ERROR_INVALID_DRM_FORMAT_MODIFIER_PLANE_LAYOUT_EXT. |
| On the other hand, if the application did not provide the external image's |
| size to flink:vkCreateImage, then the application would observe after |
| calling flink:vkGetImageMemoryRequirements that the external image's size is |
| larger than the implementation-usable size. |
| The application would observe the conflict and refuse to bind the |
| sname:VkImage to the external memory. |
| In both cases, the result is explicit failure. |
| |
| Therefore, an application-provided size provides no benefit, and this |
| extension should not require it. |
| This decision renders slink:VkSubresourceLayout::pname:size an unused field |
| during image creation, and thus introduces a risk that implementations may |
| require applications to submit sideband creation parameters in the unused |
| field. |
| To prevent implementations from relying on sideband data, this extension |
| _requires_ the application to set pname:size to 0. |
| -- |
| |
| ==== References |
| |
| . [[VK_EXT_image_drm_format_modifier-fn1]] |
| https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_image_dma_buf_import.txt[`EGL_EXT_image_dma_buf_import`] |
| . [[VK_EXT_image_drm_format_modifier-fn2]] |
| https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_image_dma_buf_import_modifiers.txt[`EGL_EXT_image_dma_buf_import_modifiers`] |
| . [[VK_EXT_image_drm_format_modifier-fn3]] |
| https://www.khronos.org/registry/EGL/extensions/MESA/EGL_MESA_image_dma_buf_export.txt[`EGL_MESA_image_dma_buf_export`] |
| . [[VK_EXT_image_drm_format_modifier-fn4]] |
| https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_mode.h?id=refs/tags/v4.10#n392[`struct |
| drm_mode_fb_cmd2`] |
| . [[VK_EXT_image_drm_format_modifier-fn5]] |
| https://cgit.freedesktop.org/mesa/mesa/tree/src/gbm/main/gbm.h?id=refs/tags/mesa-18.0.0-rc1[`gbm.h`] |
| |
| ==== Version History |
| |
| * Revision 1, 2018-08-29 (Chad Versace) |
| - First stable revision |
| * Revision 2, 2021-09-30 (Jon Leech) |
| - Add interaction with `apiext:VK_KHR_format_feature_flags2` to `vk.xml` |