docs/system/devices/nvme.rst - third_party/qemu - Git at Google

 ==============
 NVMe Emulation
 ==============

 QEMU provides NVMe emulation through the ``nvme``, ``nvme-ns`` and
 ``nvme-subsys`` devices.

 See the following sections for specific information on

   * `Adding NVMe Devices`_, `additional namespaces`_ and `NVM subsystems`_.
   * Configuration of `Optional Features`_ such as `Controller Memory Buffer`_,
     `Simple Copy`_, `Zoned Namespaces`_, `metadata`_ and `End-to-End Data
     Protection`_,

 Adding NVMe Devices
 ===================

 Controller Emulation
 --------------------

 The QEMU emulated NVMe controller implements version 1.4 of the NVM Express
 specification. All mandatory features are implement with a couple of exceptions
 and limitations:

   * Accounting numbers in the SMART/Health log page are reset when the device
     is power cycled.
   * Interrupt Coalescing is not supported and is disabled by default.

 The simplest way to attach an NVMe controller on the QEMU PCI bus is to add the
 following parameters:

 .. code-block:: console

     -drive file=nvm.img,if=none,id=nvm
     -device nvme,serial=deadbeef,drive=nvm

 There are a number of optional general parameters for the ``nvme`` device. Some
 are mentioned here, but see ``-device nvme,help`` to list all possible
 parameters.

 ``max_ioqpairs=UINT32`` (default: ``64``)
   Set the maximum number of allowed I/O queue pairs. This replaces the
   deprecated ``num_queues`` parameter.

 ``msix_qsize=UINT16`` (default: ``65``)
   The number of MSI-X vectors that the device should support.

 ``mdts=UINT8`` (default: ``7``)
   Set the Maximum Data Transfer Size of the device.

 ``use-intel-id`` (default: ``off``)
   Since QEMU 5.2, the device uses a QEMU allocated "Red Hat" PCI Device and
   Vendor ID. Set this to ``on`` to revert to the unallocated Intel ID
   previously used.

 Additional Namespaces
 ---------------------

 In the simplest possible invocation sketched above, the device only support a
 single namespace with the namespace identifier ``1``. To support multiple
 namespaces and additional features, the ``nvme-ns`` device must be used.

 .. code-block:: console

    -device nvme,id=nvme-ctrl-0,serial=deadbeef
    -drive file=nvm-1.img,if=none,id=nvm-1
    -device nvme-ns,drive=nvm-1
    -drive file=nvm-2.img,if=none,id=nvm-2
    -device nvme-ns,drive=nvm-2

 The namespaces defined by the ``nvme-ns`` device will attach to the most
 recently defined ``nvme-bus`` that is created by the ``nvme`` device. Namespace
 identifiers are allocated automatically, starting from ``1``.

 There are a number of parameters available:

 ``nsid`` (default: ``0``)
   Explicitly set the namespace identifier.

 ``uuid`` (default: *autogenerated*)
   Set the UUID of the namespace. This will be reported as a "Namespace UUID"
   descriptor in the Namespace Identification Descriptor List.

 ``eui64``
   Set the EUI-64 of the namespace. This will be reported as a "IEEE Extended
   Unique Identifier" descriptor in the Namespace Identification Descriptor List.
   Since machine type 6.1 a non-zero default value is used if the parameter
   is not provided. For earlier machine types the field defaults to 0.

 ``bus``
   If there are more ``nvme`` devices defined, this parameter may be used to
   attach the namespace to a specific ``nvme`` device (identified by an ``id``
   parameter on the controller device).

 NVM Subsystems
 --------------

 Additional features becomes available if the controller device (``nvme``) is
 linked to an NVM Subsystem device (``nvme-subsys``).

 The NVM Subsystem emulation allows features such as shared namespaces and
 multipath I/O.

 .. code-block:: console

    -device nvme-subsys,id=nvme-subsys-0,nqn=subsys0
    -device nvme,serial=deadbeef,subsys=nvme-subsys-0
    -device nvme,serial=deadbeef,subsys=nvme-subsys-0

 This will create an NVM subsystem with two controllers. Having controllers
 linked to an ``nvme-subsys`` device allows additional ``nvme-ns`` parameters:

 ``shared`` (default: ``on`` since 6.2)
   Specifies that the namespace will be attached to all controllers in the
   subsystem. If set to ``off``, the namespace will remain a private namespace
   and may only be attached to a single controller at a time. Shared namespaces
   are always automatically attached to all controllers (also when controllers
   are hotplugged).

 ``detached`` (default: ``off``)
   If set to ``on``, the namespace will be be available in the subsystem, but
   not attached to any controllers initially. A shared namespace with this set
   to ``on`` will never be automatically attached to controllers.

 Thus, adding

 .. code-block:: console

    -drive file=nvm-1.img,if=none,id=nvm-1
    -device nvme-ns,drive=nvm-1,nsid=1
    -drive file=nvm-2.img,if=none,id=nvm-2
    -device nvme-ns,drive=nvm-2,nsid=3,shared=off,detached=on

 will cause NSID 1 will be a shared namespace that is initially attached to both
 controllers. NSID 3 will be a private namespace due to ``shared=off`` and only
 attachable to a single controller at a time. Additionally it will not be
 attached to any controller initially (due to ``detached=on``) or to hotplugged
 controllers.

 Optional Features
 =================

 Controller Memory Buffer
 ------------------------

 ``nvme`` device parameters related to the Controller Memory Buffer support:

 ``cmb_size_mb=UINT32`` (default: ``0``)
   This adds a Controller Memory Buffer of the given size at offset zero in BAR
   2.

 ``legacy-cmb`` (default: ``off``)
   By default, the device uses the "v1.4 scheme" for the Controller Memory
   Buffer support (i.e, the CMB is initially disabled and must be explicitly
   enabled by the host). Set this to ``on`` to behave as a v1.3 device wrt. the
   CMB.

 Simple Copy
 -----------

 The device includes support for TP 4065 ("Simple Copy Command"). A number of
 additional ``nvme-ns`` device parameters may be used to control the Copy
 command limits:

 ``mssrl=UINT16`` (default: ``128``)
   Set the Maximum Single Source Range Length (``MSSRL``). This is the maximum
   number of logical blocks that may be specified in each source range.

 ``mcl=UINT32`` (default: ``128``)
   Set the Maximum Copy Length (``MCL``). This is the maximum number of logical
   blocks that may be specified in a Copy command (the total for all source
   ranges).

 ``msrc=UINT8`` (default: ``127``)
   Set the Maximum Source Range Count (``MSRC``). This is the maximum number of
   source ranges that may be used in a Copy command. This is a 0's based value.

 Zoned Namespaces
 ----------------

 A namespaces may be "Zoned" as defined by TP 4053 ("Zoned Namespaces"). Set
 ``zoned=on`` on an ``nvme-ns`` device to configure it as a zoned namespace.

 The namespace may be configured with additional parameters

 ``zoned.zone_size=SIZE`` (default: ``128MiB``)
   Define the zone size (``ZSZE``).

 ``zoned.zone_capacity=SIZE`` (default: ``0``)
   Define the zone capacity (``ZCAP``). If left at the default (``0``), the zone
   capacity will equal the zone size.

 ``zoned.descr_ext_size=UINT32`` (default: ``0``)
   Set the Zone Descriptor Extension Size (``ZDES``). Must be a multiple of 64
   bytes.

 ``zoned.cross_read=BOOL`` (default: ``off``)
   Set to ``on`` to allow reads to cross zone boundaries.

 ``zoned.max_active=UINT32`` (default: ``0``)
   Set the maximum number of active resources (``MAR``). The default (``0``)
   allows all zones to be active.

 ``zoned.max_open=UINT32`` (default: ``0``)
   Set the maximum number of open resources (``MOR``). The default (``0``)
   allows all zones to be open. If ``zoned.max_active`` is specified, this value
   must be less than or equal to that.

 ``zoned.zasl=UINT8`` (default: ``0``)
   Set the maximum data transfer size for the Zone Append command. Like
   ``mdts``, the value is specified as a power of two (2^n) and is in units of
   the minimum memory page size (CAP.MPSMIN). The default value (``0``)
   has this property inherit the ``mdts`` value.

 Metadata
 --------

 The virtual namespace device supports LBA metadata in the form separate
 metadata (``MPTR``-based) and extended LBAs.

 ``ms=UINT16`` (default: ``0``)
   Defines the number of metadata bytes per LBA.

 ``mset=UINT8`` (default: ``0``)
   Set to ``1`` to enable extended LBAs.

 End-to-End Data Protection
 --------------------------

 The virtual namespace device supports DIF- and DIX-based protection information
 (depending on ``mset``).

 ``pi=UINT8`` (default: ``0``)
   Enable protection information of the specified type (type ``1``, ``2`` or
   ``3``).

 ``pil=UINT8`` (default: ``0``)
   Controls the location of the protection information within the metadata. Set
   to ``1`` to transfer protection information as the first eight bytes of
   metadata. Otherwise, the protection information is transferred as the last
   eight bytes.

 Virtualization Enhancements and SR-IOV (Experimental Support)
 -------------------------------------------------------------

 The ``nvme`` device supports Single Root I/O Virtualization and Sharing
 along with Virtualization Enhancements. The controller has to be linked to
 an NVM Subsystem device (``nvme-subsys``) for use with SR-IOV.

 A number of parameters are present (**please note, that they may be
 subject to change**):

 ``sriov_max_vfs`` (default: ``0``)
   Indicates the maximum number of PCIe virtual functions supported
   by the controller. Specifying a non-zero value enables reporting of both
   SR-IOV and ARI (Alternative Routing-ID Interpretation) capabilities
   by the NVMe device. Virtual function controllers will not report SR-IOV.

 ``sriov_vq_flexible``
   Indicates the total number of flexible queue resources assignable to all
   the secondary controllers. Implicitly sets the number of primary
   controller's private resources to ``(max_ioqpairs - sriov_vq_flexible)``.

 ``sriov_vi_flexible``
   Indicates the total number of flexible interrupt resources assignable to
   all the secondary controllers. Implicitly sets the number of primary
   controller's private resources to ``(msix_qsize - sriov_vi_flexible)``.

 ``sriov_max_vi_per_vf`` (default: ``0``)
   Indicates the maximum number of virtual interrupt resources assignable
   to a secondary controller. The default ``0`` resolves to
   ``(sriov_vi_flexible / sriov_max_vfs)``

 ``sriov_max_vq_per_vf`` (default: ``0``)
   Indicates the maximum number of virtual queue resources assignable to
   a secondary controller. The default ``0`` resolves to
   ``(sriov_vq_flexible / sriov_max_vfs)``

 The simplest possible invocation enables the capability to set up one VF
 controller and assign an admin queue, an IO queue, and a MSI-X interrupt.

 .. code-block:: console

    -device nvme-subsys,id=subsys0
    -device nvme,serial=deadbeef,subsys=subsys0,sriov_max_vfs=1,
     sriov_vq_flexible=2,sriov_vi_flexible=1

 The minimum steps required to configure a functional NVMe secondary
 controller are:

   * unbind flexible resources from the primary controller

 .. code-block:: console

    nvme virt-mgmt /dev/nvme0 -c 0 -r 1 -a 1 -n 0
    nvme virt-mgmt /dev/nvme0 -c 0 -r 0 -a 1 -n 0

   * perform a Function Level Reset on the primary controller to actually
     release the resources

 .. code-block:: console

    echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset

   * enable VF

 .. code-block:: console

    echo 1 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs

   * assign the flexible resources to the VF and set it ONLINE

 .. code-block:: console

    nvme virt-mgmt /dev/nvme0 -c 1 -r 1 -a 8 -n 1
    nvme virt-mgmt /dev/nvme0 -c 1 -r 0 -a 8 -n 2
    nvme virt-mgmt /dev/nvme0 -c 1 -r 0 -a 9 -n 0

   * bind the NVMe driver to the VF

 .. code-block:: console

    echo 0000:01:00.1 > /sys/bus/pci/drivers/nvme/bind
	==============
	NVMe Emulation
	==============

	QEMU provides NVMe emulation through the ``nvme``, ``nvme-ns`` and
	``nvme-subsys`` devices.

	See the following sections for specific information on

	* `Adding NVMe Devices`_, `additional namespaces`_ and `NVM subsystems`_.
	* Configuration of `Optional Features`_ such as `Controller Memory Buffer`_,
	`Simple Copy`_, `Zoned Namespaces`_, `metadata`_ and `End-to-End Data
	Protection`_,

	Adding NVMe Devices
	===================

	Controller Emulation
	--------------------

	The QEMU emulated NVMe controller implements version 1.4 of the NVM Express
	specification. All mandatory features are implement with a couple of exceptions
	and limitations:

	* Accounting numbers in the SMART/Health log page are reset when the device
	is power cycled.
	* Interrupt Coalescing is not supported and is disabled by default.

	The simplest way to attach an NVMe controller on the QEMU PCI bus is to add the
	following parameters:

	.. code-block:: console

	-drive file=nvm.img,if=none,id=nvm
	-device nvme,serial=deadbeef,drive=nvm

	There are a number of optional general parameters for the ``nvme`` device. Some
	are mentioned here, but see ``-device nvme,help`` to list all possible
	parameters.

	``max_ioqpairs=UINT32`` (default: ``64``)
	Set the maximum number of allowed I/O queue pairs. This replaces the
	deprecated ``num_queues`` parameter.

	``msix_qsize=UINT16`` (default: ``65``)
	The number of MSI-X vectors that the device should support.

	``mdts=UINT8`` (default: ``7``)
	Set the Maximum Data Transfer Size of the device.

	``use-intel-id`` (default: ``off``)
	Since QEMU 5.2, the device uses a QEMU allocated "Red Hat" PCI Device and
	Vendor ID. Set this to ``on`` to revert to the unallocated Intel ID
	previously used.

	Additional Namespaces
	---------------------

	In the simplest possible invocation sketched above, the device only support a
	single namespace with the namespace identifier ``1``. To support multiple
	namespaces and additional features, the ``nvme-ns`` device must be used.

	.. code-block:: console

	-device nvme,id=nvme-ctrl-0,serial=deadbeef
	-drive file=nvm-1.img,if=none,id=nvm-1
	-device nvme-ns,drive=nvm-1
	-drive file=nvm-2.img,if=none,id=nvm-2
	-device nvme-ns,drive=nvm-2

	The namespaces defined by the ``nvme-ns`` device will attach to the most
	recently defined ``nvme-bus`` that is created by the ``nvme`` device. Namespace
	identifiers are allocated automatically, starting from ``1``.

	There are a number of parameters available:

	``nsid`` (default: ``0``)
	Explicitly set the namespace identifier.

	``uuid`` (default: autogenerated)
	Set the UUID of the namespace. This will be reported as a "Namespace UUID"
	descriptor in the Namespace Identification Descriptor List.

	``eui64``
	Set the EUI-64 of the namespace. This will be reported as a "IEEE Extended
	Unique Identifier" descriptor in the Namespace Identification Descriptor List.
	Since machine type 6.1 a non-zero default value is used if the parameter
	is not provided. For earlier machine types the field defaults to 0.

	``bus``
	If there are more ``nvme`` devices defined, this parameter may be used to
	attach the namespace to a specific ``nvme`` device (identified by an ``id``
	parameter on the controller device).

	NVM Subsystems
	--------------

	Additional features becomes available if the controller device (``nvme``) is
	linked to an NVM Subsystem device (``nvme-subsys``).

	The NVM Subsystem emulation allows features such as shared namespaces and
	multipath I/O.

	.. code-block:: console

	-device nvme-subsys,id=nvme-subsys-0,nqn=subsys0
	-device nvme,serial=deadbeef,subsys=nvme-subsys-0
	-device nvme,serial=deadbeef,subsys=nvme-subsys-0

	This will create an NVM subsystem with two controllers. Having controllers
	linked to an ``nvme-subsys`` device allows additional ``nvme-ns`` parameters:

	``shared`` (default: ``on`` since 6.2)
	Specifies that the namespace will be attached to all controllers in the
	subsystem. If set to ``off``, the namespace will remain a private namespace
	and may only be attached to a single controller at a time. Shared namespaces
	are always automatically attached to all controllers (also when controllers
	are hotplugged).

	``detached`` (default: ``off``)
	If set to ``on``, the namespace will be be available in the subsystem, but
	not attached to any controllers initially. A shared namespace with this set
	to ``on`` will never be automatically attached to controllers.

	Thus, adding

	.. code-block:: console

	-drive file=nvm-1.img,if=none,id=nvm-1
	-device nvme-ns,drive=nvm-1,nsid=1
	-drive file=nvm-2.img,if=none,id=nvm-2
	-device nvme-ns,drive=nvm-2,nsid=3,shared=off,detached=on

	will cause NSID 1 will be a shared namespace that is initially attached to both
	controllers. NSID 3 will be a private namespace due to ``shared=off`` and only
	attachable to a single controller at a time. Additionally it will not be
	attached to any controller initially (due to ``detached=on``) or to hotplugged
	controllers.

	Optional Features
	=================

	Controller Memory Buffer
	------------------------

	``nvme`` device parameters related to the Controller Memory Buffer support:

	``cmb_size_mb=UINT32`` (default: ``0``)
	This adds a Controller Memory Buffer of the given size at offset zero in BAR
	2.

	``legacy-cmb`` (default: ``off``)
	By default, the device uses the "v1.4 scheme" for the Controller Memory
	Buffer support (i.e, the CMB is initially disabled and must be explicitly
	enabled by the host). Set this to ``on`` to behave as a v1.3 device wrt. the
	CMB.

	Simple Copy
	-----------

	The device includes support for TP 4065 ("Simple Copy Command"). A number of
	additional ``nvme-ns`` device parameters may be used to control the Copy
	command limits:

	``mssrl=UINT16`` (default: ``128``)
	Set the Maximum Single Source Range Length (``MSSRL``). This is the maximum
	number of logical blocks that may be specified in each source range.

	``mcl=UINT32`` (default: ``128``)
	Set the Maximum Copy Length (``MCL``). This is the maximum number of logical
	blocks that may be specified in a Copy command (the total for all source
	ranges).

	``msrc=UINT8`` (default: ``127``)
	Set the Maximum Source Range Count (``MSRC``). This is the maximum number of
	source ranges that may be used in a Copy command. This is a 0's based value.

	Zoned Namespaces
	----------------

	A namespaces may be "Zoned" as defined by TP 4053 ("Zoned Namespaces"). Set
	``zoned=on`` on an ``nvme-ns`` device to configure it as a zoned namespace.

	The namespace may be configured with additional parameters

	``zoned.zone_size=SIZE`` (default: ``128MiB``)
	Define the zone size (``ZSZE``).

	``zoned.zone_capacity=SIZE`` (default: ``0``)
	Define the zone capacity (``ZCAP``). If left at the default (``0``), the zone
	capacity will equal the zone size.

	``zoned.descr_ext_size=UINT32`` (default: ``0``)
	Set the Zone Descriptor Extension Size (``ZDES``). Must be a multiple of 64
	bytes.

	``zoned.cross_read=BOOL`` (default: ``off``)
	Set to ``on`` to allow reads to cross zone boundaries.

	``zoned.max_active=UINT32`` (default: ``0``)
	Set the maximum number of active resources (``MAR``). The default (``0``)
	allows all zones to be active.

	``zoned.max_open=UINT32`` (default: ``0``)
	Set the maximum number of open resources (``MOR``). The default (``0``)
	allows all zones to be open. If ``zoned.max_active`` is specified, this value
	must be less than or equal to that.

	``zoned.zasl=UINT8`` (default: ``0``)
	Set the maximum data transfer size for the Zone Append command. Like
	``mdts``, the value is specified as a power of two (2^n) and is in units of
	the minimum memory page size (CAP.MPSMIN). The default value (``0``)
	has this property inherit the ``mdts`` value.

	Metadata
	--------

	The virtual namespace device supports LBA metadata in the form separate
	metadata (``MPTR``-based) and extended LBAs.

	``ms=UINT16`` (default: ``0``)
	Defines the number of metadata bytes per LBA.

	``mset=UINT8`` (default: ``0``)
	Set to ``1`` to enable extended LBAs.

	End-to-End Data Protection
	--------------------------

	The virtual namespace device supports DIF- and DIX-based protection information
	(depending on ``mset``).

	``pi=UINT8`` (default: ``0``)
	Enable protection information of the specified type (type ``1``, ``2`` or
	``3``).

	``pil=UINT8`` (default: ``0``)
	Controls the location of the protection information within the metadata. Set
	to ``1`` to transfer protection information as the first eight bytes of
	metadata. Otherwise, the protection information is transferred as the last
	eight bytes.

	Virtualization Enhancements and SR-IOV (Experimental Support)
	-------------------------------------------------------------

	The ``nvme`` device supports Single Root I/O Virtualization and Sharing
	along with Virtualization Enhancements. The controller has to be linked to
	an NVM Subsystem device (``nvme-subsys``) for use with SR-IOV.

	A number of parameters are present (**please note, that they may be
	subject to change**):

	``sriov_max_vfs`` (default: ``0``)
	Indicates the maximum number of PCIe virtual functions supported
	by the controller. Specifying a non-zero value enables reporting of both
	SR-IOV and ARI (Alternative Routing-ID Interpretation) capabilities
	by the NVMe device. Virtual function controllers will not report SR-IOV.

	``sriov_vq_flexible``
	Indicates the total number of flexible queue resources assignable to all
	the secondary controllers. Implicitly sets the number of primary
	controller's private resources to ``(max_ioqpairs - sriov_vq_flexible)``.

	``sriov_vi_flexible``
	Indicates the total number of flexible interrupt resources assignable to
	all the secondary controllers. Implicitly sets the number of primary
	controller's private resources to ``(msix_qsize - sriov_vi_flexible)``.

	``sriov_max_vi_per_vf`` (default: ``0``)
	Indicates the maximum number of virtual interrupt resources assignable
	to a secondary controller. The default ``0`` resolves to
	``(sriov_vi_flexible / sriov_max_vfs)``

	``sriov_max_vq_per_vf`` (default: ``0``)
	Indicates the maximum number of virtual queue resources assignable to
	a secondary controller. The default ``0`` resolves to
	``(sriov_vq_flexible / sriov_max_vfs)``

	The simplest possible invocation enables the capability to set up one VF
	controller and assign an admin queue, an IO queue, and a MSI-X interrupt.

	.. code-block:: console

	-device nvme-subsys,id=subsys0
	-device nvme,serial=deadbeef,subsys=subsys0,sriov_max_vfs=1,
	sriov_vq_flexible=2,sriov_vi_flexible=1

	The minimum steps required to configure a functional NVMe secondary
	controller are:

	* unbind flexible resources from the primary controller

	.. code-block:: console

	nvme virt-mgmt /dev/nvme0 -c 0 -r 1 -a 1 -n 0
	nvme virt-mgmt /dev/nvme0 -c 0 -r 0 -a 1 -n 0

	* perform a Function Level Reset on the primary controller to actually
	release the resources

	.. code-block:: console

	echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset

	* enable VF

	.. code-block:: console

	echo 1 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs

	* assign the flexible resources to the VF and set it ONLINE

	.. code-block:: console

	nvme virt-mgmt /dev/nvme0 -c 1 -r 1 -a 8 -n 1
	nvme virt-mgmt /dev/nvme0 -c 1 -r 0 -a 8 -n 2
	nvme virt-mgmt /dev/nvme0 -c 1 -r 0 -a 9 -n 0

	* bind the NVMe driver to the VF

	.. code-block:: console

	echo 0000:01:00.1 > /sys/bus/pci/drivers/nvme/bind