| .. |
| Copyright 2019 John Snow <jsnow@redhat.com> and Red Hat, Inc. |
| All rights reserved. |
| |
| This file is licensed via The FreeBSD Documentation License, the full |
| text of which is included at the end of this document. |
| |
| ==================================== |
| Dirty Bitmaps and Incremental Backup |
| ==================================== |
| |
| Dirty Bitmaps are in-memory objects that track writes to block devices. They |
| can be used in conjunction with various block job operations to perform |
| incremental or differential backup regimens. |
| |
| This document explains the conceptual mechanisms, as well as up-to-date, |
| complete and comprehensive documentation on the API to manipulate them. |
| (Hopefully, the "why", "what", and "how".) |
| |
| The intended audience for this document is developers who are adding QEMU |
| backup features to management applications, or power users who run and |
| administer QEMU directly via QMP. |
| |
| .. contents:: |
| |
| Overview |
| -------- |
| |
| Bitmaps are bit vectors where each '1' bit in the vector indicates a modified |
| ("dirty") segment of the corresponding block device. The size of the segment |
| that is tracked is the granularity of the bitmap. If the granularity of a |
| bitmap is 64K, each '1' bit means that a 64K region as a whole may have |
| changed in some way, possibly by as little as one byte. |
| |
| Smaller granularities mean more accurate tracking of modified disk data, but |
| requires more computational overhead and larger bitmap sizes. Larger |
| granularities mean smaller bitmap sizes, but less targeted backups. |
| |
| The size of a bitmap (in bytes) can be computed as such: |
| ``size`` = ceil(ceil(``image_size`` / ``granularity``) / 8) |
| |
| e.g. the size of a 64KiB granularity bitmap on a 2TiB image is: |
| ``size`` = ((2147483648K / 64K) / 8) |
| = 4194304B = 4MiB. |
| |
| QEMU uses these bitmaps when making incremental backups to know which sections |
| of the file to copy out. They are not enabled by default and must be |
| explicitly added in order to begin tracking writes. |
| |
| Bitmaps can be created at any time and can be attached to any arbitrary block |
| node in the storage graph, but are most useful conceptually when attached to |
| the root node attached to the guest's storage device model. |
| |
| That is to say: It's likely most useful to track the guest's writes to disk, |
| but you could theoretically track things like qcow2 metadata changes by |
| attaching the bitmap elsewhere in the storage graph. This is beyond the scope |
| of this document. |
| |
| QEMU supports persisting these bitmaps to disk via the qcow2 image format. |
| Bitmaps which are stored or loaded in this way are called "persistent", |
| whereas bitmaps that are not are called "transient". |
| |
| QEMU also supports the migration of both transient bitmaps (tracking any |
| arbitrary image format) or persistent bitmaps (qcow2) via live migration. |
| |
| Supported Image Formats |
| ----------------------- |
| |
| QEMU supports all documented features below on the qcow2 image format. |
| |
| However, qcow2 is only strictly necessary for the persistence feature, which |
| writes bitmap data to disk upon close. If persistence is not required for a |
| specific use case, all bitmap features excepting persistence are available for |
| any arbitrary image format. |
| |
| For example, Dirty Bitmaps can be combined with the 'raw' image format, but |
| any changes to the bitmap will be discarded upon exit. |
| |
| .. warning:: Transient bitmaps will not be saved on QEMU exit! Persistent |
| bitmaps are available only on qcow2 images. |
| |
| Dirty Bitmap Names |
| ------------------ |
| |
| Bitmap objects need a method to reference them in the API. All API-created and |
| managed bitmaps have a human-readable name chosen by the user at creation |
| time. |
| |
| - A bitmap's name is unique to the node, but bitmaps attached to different |
| nodes can share the same name. Therefore, all bitmaps are addressed via |
| their (node, name) pair. |
| |
| - The name of a user-created bitmap cannot be empty (""). |
| |
| - Transient bitmaps can have JSON unicode names that are effectively not |
| length limited. (QMP protocol may restrict messages to less than 64MiB.) |
| |
| - Persistent storage formats may impose their own requirements on bitmap names |
| and namespaces. Presently, only qcow2 supports persistent bitmaps. See |
| docs/interop/qcow2.txt for more details on restrictions. Notably: |
| |
| - qcow2 bitmap names are limited to between 1 and 1023 bytes long. |
| |
| - No two bitmaps saved to the same qcow2 file may share the same name. |
| |
| - QEMU occasionally uses bitmaps for internal use which have no name. They are |
| hidden from API query calls, cannot be manipulated by the external API, are |
| never persistent, nor ever migrated. |
| |
| Bitmap Status |
| ------------- |
| |
| Dirty Bitmap objects can be queried with the QMP command `query-block |
| <qemu-qmp-ref.html#index-query_002dblock>`_, and are visible via the |
| `BlockDirtyInfo <qemu-qmp-ref.html#index-BlockDirtyInfo>`_ QAPI structure. |
| |
| This struct shows the name, granularity, and dirty byte count for each bitmap. |
| Additionally, it shows several boolean status indicators: |
| |
| - ``recording``: This bitmap is recording writes. |
| - ``busy``: This bitmap is in-use by an operation. |
| - ``persistent``: This bitmap is a persistent type. |
| - ``inconsistent``: This bitmap is corrupted and cannot be used. |
| |
| The ``+busy`` status prohibits you from deleting, clearing, or otherwise |
| modifying a bitmap, and happens when the bitmap is being used for a backup |
| operation or is in the process of being loaded from a migration. Many of the |
| commands documented below will refuse to work on such bitmaps. |
| |
| The ``+inconsistent`` status similarly prohibits almost all operations, |
| notably allowing only the ``block-dirty-bitmap-remove`` operation. |
| |
| There is also a deprecated ``status`` field of type `DirtyBitmapStatus |
| <qemu-qmp-ref.html#index-DirtyBitmapStatus>`_. A bitmap historically had |
| five visible states: |
| |
| #. ``Frozen``: This bitmap is currently in-use by an operation and is |
| immutable. It can't be deleted, renamed, reset, etc. |
| |
| (This is now ``+busy``.) |
| |
| #. ``Disabled``: This bitmap is not recording new writes. |
| |
| (This is now ``-recording -busy``.) |
| |
| #. ``Active``: This bitmap is recording new writes. |
| |
| (This is now ``+recording -busy``.) |
| |
| #. ``Locked``: This bitmap is in-use by an operation, and is immutable. |
| The difference from "Frozen" was primarily implementation details. |
| |
| (This is now ``+busy``.) |
| |
| #. ``Inconsistent``: This persistent bitmap was not saved to disk |
| correctly, and can no longer be used. It remains in memory to serve as |
| an indicator of failure. |
| |
| (This is now ``+inconsistent``.) |
| |
| These states are directly replaced by the status indicators and should not be |
| used. The difference between ``Frozen`` and ``Locked`` is an implementation |
| detail and should not be relevant to external users. |
| |
| Basic QMP Usage |
| --------------- |
| |
| The primary interface to manipulating bitmap objects is via the QMP |
| interface. If you are not familiar, see docs/interop/qmp-intro.txt for a broad |
| overview, and `qemu-qmp-ref <qemu-qmp-ref.html>`_ for a full reference of all |
| QMP commands. |
| |
| Supported Commands |
| ~~~~~~~~~~~~~~~~~~ |
| |
| There are six primary bitmap-management API commands: |
| |
| - ``block-dirty-bitmap-add`` |
| - ``block-dirty-bitmap-remove`` |
| - ``block-dirty-bitmap-clear`` |
| - ``block-dirty-bitmap-disable`` |
| - ``block-dirty-bitmap-enable`` |
| - ``block-dirty-bitmap-merge`` |
| |
| And one related query command: |
| |
| - ``query-block`` |
| |
| Creation: block-dirty-bitmap-add |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| `block-dirty-bitmap-add |
| <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dadd>`_: |
| |
| Creates a new bitmap that tracks writes to the specified node. granularity, |
| persistence, and recording state can be adjusted at creation time. |
| |
| .. admonition:: Example |
| |
| to create a new, actively recording persistent bitmap: |
| |
| .. code-block:: QMP |
| |
| -> { "execute": "block-dirty-bitmap-add", |
| "arguments": { |
| "node": "drive0", |
| "name": "bitmap0", |
| "persistent": true, |
| } |
| } |
| |
| <- { "return": {} } |
| |
| - This bitmap will have a default granularity that matches the cluster size of |
| its associated drive, if available, clamped to between [4KiB, 64KiB]. The |
| current default for qcow2 is 64KiB. |
| |
| .. admonition:: Example |
| |
| To create a new, disabled (``-recording``), transient bitmap that tracks |
| changes in 32KiB segments: |
| |
| .. code-block:: QMP |
| |
| -> { "execute": "block-dirty-bitmap-add", |
| "arguments": { |
| "node": "drive0", |
| "name": "bitmap1", |
| "granularity": 32768, |
| "disabled": true |
| } |
| } |
| |
| <- { "return": {} } |
| |
| Deletion: block-dirty-bitmap-remove |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| `block-dirty-bitmap-remove |
| <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dremove>`_: |
| |
| Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed. |
| |
| - Deleting a bitmap does not impact any other bitmaps attached to the same |
| node, nor does it affect any backups already created from this bitmap or |
| node. |
| |
| - Because bitmaps are only unique to the node to which they are attached, you |
| must specify the node/drive name here, too. |
| |
| - Deleting a persistent bitmap will remove it from the qcow2 file. |
| |
| .. admonition:: Example |
| |
| Remove a bitmap named ``bitmap0`` from node ``drive0``: |
| |
| .. code-block:: QMP |
| |
| -> { "execute": "block-dirty-bitmap-remove", |
| "arguments": { |
| "node": "drive0", |
| "name": "bitmap0" |
| } |
| } |
| |
| <- { "return": {} } |
| |
| Resetting: block-dirty-bitmap-clear |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| `block-dirty-bitmap-clear |
| <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dclear>`_: |
| |
| Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared. |
| |
| - An incremental backup created from an empty bitmap will copy no data, as if |
| nothing has changed. |
| |
| .. admonition:: Example |
| |
| Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``: |
| |
| .. code-block:: QMP |
| |
| -> { "execute": "block-dirty-bitmap-clear", |
| "arguments": { |
| "node": "drive0", |
| "name": "bitmap0" |
| } |
| } |
| |
| <- { "return": {} } |
| |
| Enabling: block-dirty-bitmap-enable |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| `block-dirty-bitmap-enable |
| <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002denable>`_: |
| |
| "Enables" a bitmap, setting the ``recording`` bit to true, causing writes to |
| begin being recorded. ``+busy`` bitmaps cannot be enabled. |
| |
| - Bitmaps default to being enabled when created, unless configured otherwise. |
| |
| - Persistent enabled bitmaps will remember their ``+recording`` status on |
| load. |
| |
| .. admonition:: Example |
| |
| To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``: |
| |
| .. code-block:: QMP |
| |
| -> { "execute": "block-dirty-bitmap-enable", |
| "arguments": { |
| "node": "drive0", |
| "name": "bitmap0" |
| } |
| } |
| |
| <- { "return": {} } |
| |
| Enabling: block-dirty-bitmap-disable |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| `block-dirty-bitmap-disable |
| <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002ddisable>`_: |
| |
| "Disables" a bitmap, setting the ``recording`` bit to false, causing further |
| writes to begin being ignored. ``+busy`` bitmaps cannot be disabled. |
| |
| .. warning:: |
| |
| This is potentially dangerous: QEMU makes no effort to stop any writes if |
| there are disabled bitmaps on a node, and will not mark any disabled bitmaps |
| as ``+inconsistent`` if any such writes do happen. Backups made from such |
| bitmaps will not be able to be used to reconstruct a coherent image. |
| |
| - Disabling a bitmap may be useful for examining which sectors of a disk |
| changed during a specific time period, or for explicit management of |
| differential backup windows. |
| |
| - Persistent disabled bitmaps will remember their ``-recording`` status on |
| load. |
| |
| .. admonition:: Example |
| |
| To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``: |
| |
| .. code-block:: QMP |
| |
| -> { "execute": "block-dirty-bitmap-disable", |
| "arguments": { |
| "node": "drive0", |
| "name": "bitmap0" |
| } |
| } |
| |
| <- { "return": {} } |
| |
| Merging, Copying: block-dirty-bitmap-merge |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| `block-dirty-bitmap-merge |
| <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dmerge>`_: |
| |
| Merges one or more bitmaps into a target bitmap. For any segment that is dirty |
| in any one source bitmap, the target bitmap will mark that segment dirty. |
| |
| - Merge takes one or more bitmaps as a source and merges them together into a |
| single destination, such that any segment marked as dirty in any source |
| bitmap(s) will be marked dirty in the destination bitmap. |
| |
| - Merge does not create the destination bitmap if it does not exist. A blank |
| bitmap can be created beforehand to achieve the same effect. |
| |
| - The destination is not cleared prior to merge, so subsequent merge |
| operations will continue to cumulatively mark more segments as dirty. |
| |
| - If the merge operation should fail, the destination bitmap is guaranteed to |
| be unmodified. The operation may fail if the source or destination bitmaps |
| are busy, or have different granularities. |
| |
| - Bitmaps can only be merged on the same node. There is only one "node" |
| argument, so all bitmaps must be attached to that same node. |
| |
| - Copy can be achieved by merging from a single source to an empty |
| destination. |
| |
| .. admonition:: Example |
| |
| Merge the data from ``bitmap0`` into the bitmap ``new_bitmap`` on node |
| ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves |
| a copy. |
| |
| .. code-block:: QMP |
| |
| -> { "execute": "block-dirty-bitmap-merge", |
| "arguments": { |
| "node": "drive0", |
| "target": "new_bitmap", |
| "bitmaps": [ "bitmap0" ] |
| } |
| } |
| |
| <- { "return": {} } |
| |
| Querying: query-block |
| ~~~~~~~~~~~~~~~~~~~~~ |
| |
| `query-block |
| <qemu-qmp-ref.html#index-query_002dblock>`_: |
| |
| Not strictly a bitmaps command, but will return information about any bitmaps |
| attached to nodes serving as the root for guest devices. |
| |
| - The "inconsistent" bit will not appear when it is false, appearing only when |
| the value is true to indicate there is a problem. |
| |
| .. admonition:: Example |
| |
| Query the block sub-system of QEMU. The following json has trimmed irrelevant |
| keys from the response to highlight only the bitmap-relevant portions of the |
| API. This result highlights a bitmap ``bitmap0`` attached to the root node of |
| device ``drive0``. |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "query-block", |
| "arguments": {} |
| } |
| |
| <- { |
| "return": [ { |
| "dirty-bitmaps": [ { |
| "status": "active", |
| "count": 0, |
| "busy": false, |
| "name": "bitmap0", |
| "persistent": false, |
| "recording": true, |
| "granularity": 65536 |
| } ], |
| "device": "drive0", |
| } ] |
| } |
| |
| Bitmap Persistence |
| ------------------ |
| |
| As outlined in `Supported Image Formats`_, QEMU can persist bitmaps to qcow2 |
| files. Demonstrated in `Creation: block-dirty-bitmap-add`_, passing |
| ``persistent: true`` to ``block-dirty-bitmap-add`` will persist that bitmap to |
| disk. |
| |
| Persistent bitmaps will be automatically loaded into memory upon load, and |
| will be written back to disk upon close. Their usage should be mostly |
| transparent. |
| |
| However, if QEMU does not get a chance to close the file cleanly, the bitmap |
| will be marked as ``+inconsistent`` at next load and considered unsafe to use |
| for any operation. At this point, the only valid operation on such bitmaps is |
| ``block-dirty-bitmap-remove``. |
| |
| Losing a bitmap in this way does not invalidate any existing backups that have |
| been made from this bitmap, but no further backups will be able to be issued |
| for this chain. |
| |
| Transactions |
| ------------ |
| |
| Transactions are a QMP feature that allows you to submit multiple QMP commands |
| at once, being guaranteed that they will all succeed or fail atomically, |
| together. The interaction of bitmaps and transactions are demonstrated below. |
| |
| See `transaction <qemu-qmp.ref.html#index-transaction>`_ in the QMP reference |
| for more details. |
| |
| Justification |
| ~~~~~~~~~~~~~ |
| |
| Bitmaps can generally be modified at any time, but certain operations often |
| only make sense when paired directly with other commands. When a VM is paused, |
| it's easy to ensure that no guest writes occur between individual QMP |
| commands. When a VM is running, this is difficult to accomplish with |
| individual QMP commands that may allow guest writes to occur inbetween each |
| command. |
| |
| For example, using only individual QMP commands, we could: |
| |
| #. Boot the VM in a paused state. |
| #. Create a full drive backup of drive0. |
| #. Create a new bitmap attached to drive0, confident that nothing has been |
| written to drive0 in the meantime. |
| #. Resume execution of the VM. |
| #. At a later point, issue incremental backups from ``bitmap0``. |
| |
| At this point, the bitmap and drive backup would be correctly in sync, and |
| incremental backups made from this point forward would be correctly aligned to |
| the full drive backup. |
| |
| This is not particularly useful if we decide we want to start incremental |
| backups after the VM has been running for a while, for which we would want to |
| perform actions such as the following: |
| |
| #. Boot the VM and begin execution. |
| #. Using a single transaction, perform the following operations: |
| |
| - Create ``bitmap0``. |
| - Create a full drive backup of ``drive0``. |
| |
| #. At a later point, issue incremental backups from ``bitmap0``. |
| |
| .. note:: As a consideration, if ``bitmap0`` is created prior to the full |
| drive backup, incremental backups can still be authored from this |
| bitmap, but they will copy extra segments reflecting writes that |
| occurred prior to the backup operation. Transactions allow us to |
| narrow critical points in time to reduce waste, or, in the other |
| direction, to ensure that no segments are omitted. |
| |
| Supported Bitmap Transactions |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| - ``block-dirty-bitmap-add`` |
| - ``block-dirty-bitmap-clear`` |
| - ``block-dirty-bitmap-enable`` |
| - ``block-dirty-bitmap-disable`` |
| - ``block-dirty-bitmap-merge`` |
| |
| The usages for these commands are identical to their respective QMP commands, |
| but see the sections below for concrete examples. |
| |
| Incremental Backups - Push Model |
| -------------------------------- |
| |
| Incremental backups are simply partial disk images that can be combined with |
| other partial disk images on top of a base image to reconstruct a full backup |
| from the point in time at which the incremental backup was issued. |
| |
| The "Push Model" here references the fact that QEMU is "pushing" the modified |
| blocks out to a destination. We will be using the `drive-backup |
| <qemu-qmp-ref.html#index-drive_002dbackup>`_ and `blockdev-backup |
| <qemu-qmp-ref.html#index-blockdev_002dbackup>`_ QMP commands to create both |
| full and incremental backups. |
| |
| Both of these commands are jobs, which have their own QMP API for querying and |
| management documented in `Background jobs |
| <qemu-qmp-ref.html#Background-jobs>`_. |
| |
| Example: New Incremental Backup Anchor Point |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| As outlined in the Transactions - `Justification`_ section, perhaps we want to |
| create a new incremental backup chain attached to a drive. |
| |
| This example creates a new, full backup of "drive0" and accompanies it with a |
| new, empty bitmap that records writes from this point in time forward. |
| |
| .. note:: Any new writes that happen after this command is issued, even while |
| the backup job runs, will be written locally and not to the backup |
| destination. These writes will be recorded in the bitmap |
| accordingly. |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "transaction", |
| "arguments": { |
| "actions": [ |
| { |
| "type": "block-dirty-bitmap-add", |
| "data": { |
| "node": "drive0", |
| "name": "bitmap0" |
| } |
| }, |
| { |
| "type": "drive-backup", |
| "data": { |
| "device": "drive0", |
| "target": "/path/to/drive0.full.qcow2", |
| "sync": "full", |
| "format": "qcow2" |
| } |
| } |
| ] |
| } |
| } |
| |
| <- { "return": {} } |
| |
| <- { |
| "timestamp": { |
| "seconds": 1555436945, |
| "microseconds": 179620 |
| }, |
| "data": { |
| "status": "created", |
| "id": "drive0" |
| }, |
| "event": "JOB_STATUS_CHANGE" |
| } |
| |
| ... |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "type": "backup", |
| "speed": 0, |
| "len": 68719476736, |
| "offset": 68719476736 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "status": "concluded", |
| "id": "drive0" |
| }, |
| "event": "JOB_STATUS_CHANGE" |
| } |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "status": "null", |
| "id": "drive0" |
| }, |
| "event": "JOB_STATUS_CHANGE" |
| } |
| |
| A full explanation of the job transition semantics and the JOB_STATUS_CHANGE |
| event are beyond the scope of this document and will be omitted in all |
| subsequent examples; above, several more events have been omitted for brevity. |
| |
| .. note:: Subsequent examples will omit all events except BLOCK_JOB_COMPLETED |
| except where necessary to illustrate workflow differences. |
| |
| Omitted events and json objects will be represented by ellipses: |
| ``...`` |
| |
| Example: Resetting an Incremental Backup Anchor Point |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| If we want to start a new backup chain with an existing bitmap, we can also |
| use a transaction to reset the bitmap while making a new full backup: |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "transaction", |
| "arguments": { |
| "actions": [ |
| { |
| "type": "block-dirty-bitmap-clear", |
| "data": { |
| "node": "drive0", |
| "name": "bitmap0" |
| } |
| }, |
| { |
| "type": "drive-backup", |
| "data": { |
| "device": "drive0", |
| "target": "/path/to/drive0.new_full.qcow2", |
| "sync": "full", |
| "format": "qcow2" |
| } |
| } |
| ] |
| } |
| } |
| |
| <- { "return": {} } |
| |
| ... |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "type": "backup", |
| "speed": 0, |
| "len": 68719476736, |
| "offset": 68719476736 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| ... |
| |
| The result of this example is identical to the first, but we clear an existing |
| bitmap instead of adding a new one. |
| |
| .. tip:: In both of these examples, "bitmap0" is tied conceptually to the |
| creation of new, full backups. This relationship is not saved or |
| remembered by QEMU; it is up to the operator or management layer to |
| remember which bitmaps are associated with which backups. |
| |
| Example: First Incremental Backup |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| #. Create a full backup and sync it to a dirty bitmap using any method: |
| |
| - Either of the two live backup method demonstrated above, |
| - Using QMP commands with the VM paused as in the `Justification`_ section, |
| or |
| - With the VM offline, manually copy the image and start the VM in a paused |
| state, careful to add a new bitmap before the VM begins execution. |
| |
| Whichever method is chosen, let's assume that at the end of this step: |
| |
| - The full backup is named ``drive0.full.qcow2``. |
| - The bitmap we created is named ``bitmap0``, attached to ``drive0``. |
| |
| #. Create a destination image for the incremental backup that utilizes the |
| full backup as a backing image. |
| |
| - Let's assume the new incremental image is named ``drive0.inc0.qcow2``: |
| |
| .. code:: bash |
| |
| $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ |
| -b drive0.full.qcow2 -F qcow2 |
| |
| #. Issue an incremental backup command: |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "drive-backup", |
| "arguments": { |
| "device": "drive0", |
| "bitmap": "bitmap0", |
| "target": "drive0.inc0.qcow2", |
| "format": "qcow2", |
| "sync": "incremental", |
| "mode": "existing" |
| } |
| } |
| |
| <- { "return": {} } |
| |
| ... |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "type": "backup", |
| "speed": 0, |
| "len": 68719476736, |
| "offset": 68719476736 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| ... |
| |
| This copies any blocks modified since the full backup was created into the |
| ``drive0.inc0.qcow2`` file. During the operation, ``bitmap0`` is marked |
| ``+busy``. If the operation is successful, ``bitmap0`` will be cleared to |
| reflect the "incremental" backup regimen, which only copies out new changes |
| from each incremental backup. |
| |
| .. note:: Any new writes that occur after the backup operation starts do not |
| get copied to the destination. The backup's "point in time" is when |
| the backup starts, not when it ends. These writes are recorded in a |
| special bitmap that gets re-added to bitmap0 when the backup ends so |
| that the next incremental backup can copy them out. |
| |
| Example: Second Incremental Backup |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| #. Create a new destination image for the incremental backup that points to |
| the previous one, e.g.: ``drive0.inc1.qcow2`` |
| |
| .. code:: bash |
| |
| $ qemu-img create -f qcow2 drive0.inc1.qcow2 \ |
| -b drive0.inc0.qcow2 -F qcow2 |
| |
| #. Issue a new incremental backup command. The only difference here is that we |
| have changed the target image below. |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "drive-backup", |
| "arguments": { |
| "device": "drive0", |
| "bitmap": "bitmap0", |
| "target": "drive0.inc1.qcow2", |
| "format": "qcow2", |
| "sync": "incremental", |
| "mode": "existing" |
| } |
| } |
| |
| <- { "return": {} } |
| |
| ... |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "type": "backup", |
| "speed": 0, |
| "len": 68719476736, |
| "offset": 68719476736 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| ... |
| |
| Because the first incremental backup from the previous example completed |
| successfully, ``bitmap0`` was synchronized with ``drive0.inc0.qcow2``. Here, |
| we use ``bitmap0`` again to create a new incremental backup that targets the |
| previous one, creating a chain of three images: |
| |
| .. admonition:: Diagram |
| |
| .. code:: text |
| |
| +-------------------+ +-------------------+ +-------------------+ |
| | drive0.full.qcow2 |<--| drive0.inc0.qcow2 |<--| drive0.inc1.qcow2 | |
| +-------------------+ +-------------------+ +-------------------+ |
| |
| Each new incremental backup re-synchronizes the bitmap to the latest backup |
| authored, allowing a user to continue to "consume" it to create new backups on |
| top of an existing chain. |
| |
| In the above diagram, neither drive0.inc1.qcow2 nor drive0.inc0.qcow2 are |
| complete images by themselves, but rely on their backing chain to reconstruct |
| a full image. The dependency terminates with each full backup. |
| |
| Each backup in this chain remains independent, and is unchanged by new entries |
| made later in the chain. For instance, drive0.inc0.qcow2 remains a perfectly |
| valid backup of the disk as it was when that backup was issued. |
| |
| Example: Incremental Push Backups without Backing Files |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| Backup images are best kept off-site, so we often will not have the preceding |
| backups in a chain available to link against. This is not a problem at backup |
| time; we simply do not set the backing image when creating the destination |
| image: |
| |
| #. Create a new destination image with no backing file set. We will need to |
| specify the size of the base image, because the backing file isn't |
| available for QEMU to use to determine it. |
| |
| .. code:: bash |
| |
| $ qemu-img create -f qcow2 drive0.inc2.qcow2 64G |
| |
| .. note:: Alternatively, you can omit ``mode: "existing"`` from the push |
| backup commands to have QEMU create an image without a backing |
| file for you, but you lose control over format options like |
| compatibility and preallocation presets. |
| |
| #. Issue a new incremental backup command. Apart from the new destination |
| image, there is no difference from the last two examples. |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "drive-backup", |
| "arguments": { |
| "device": "drive0", |
| "bitmap": "bitmap0", |
| "target": "drive0.inc2.qcow2", |
| "format": "qcow2", |
| "sync": "incremental", |
| "mode": "existing" |
| } |
| } |
| |
| <- { "return": {} } |
| |
| ... |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "type": "backup", |
| "speed": 0, |
| "len": 68719476736, |
| "offset": 68719476736 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| ... |
| |
| The only difference from the perspective of the user is that you will need to |
| set the backing image when attempting to restore the backup: |
| |
| .. code:: bash |
| |
| $ qemu-img rebase drive0.inc2.qcow2 \ |
| -u -b drive0.inc1.qcow2 |
| |
| This uses the "unsafe" rebase mode to simply set the backing file to a file |
| that isn't present. |
| |
| It is also possible to use ``--image-opts`` to specify the entire backing |
| chain by hand as an ephemeral property at runtime, but that is beyond the |
| scope of this document. |
| |
| Example: Multi-drive Incremental Backup |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| Assume we have a VM with two drives, "drive0" and "drive1" and we wish to back |
| both of them up such that the two backups represent the same crash-consistent |
| point in time. |
| |
| #. For each drive, create an empty image: |
| |
| .. code:: bash |
| |
| $ qemu-img create -f qcow2 drive0.full.qcow2 64G |
| $ qemu-img create -f qcow2 drive1.full.qcow2 64G |
| |
| #. Create a full (anchor) backup for each drive, with accompanying bitmaps: |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "transaction", |
| "arguments": { |
| "actions": [ |
| { |
| "type": "block-dirty-bitmap-add", |
| "data": { |
| "node": "drive0", |
| "name": "bitmap0" |
| } |
| }, |
| { |
| "type": "block-dirty-bitmap-add", |
| "data": { |
| "node": "drive1", |
| "name": "bitmap0" |
| } |
| }, |
| { |
| "type": "drive-backup", |
| "data": { |
| "device": "drive0", |
| "target": "/path/to/drive0.full.qcow2", |
| "sync": "full", |
| "format": "qcow2" |
| } |
| }, |
| { |
| "type": "drive-backup", |
| "data": { |
| "device": "drive1", |
| "target": "/path/to/drive1.full.qcow2", |
| "sync": "full", |
| "format": "qcow2" |
| } |
| } |
| ] |
| } |
| } |
| |
| <- { "return": {} } |
| |
| ... |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "type": "backup", |
| "speed": 0, |
| "len": 68719476736, |
| "offset": 68719476736 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| ... |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive1", |
| "type": "backup", |
| "speed": 0, |
| "len": 68719476736, |
| "offset": 68719476736 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| ... |
| |
| #. Later, create new destination images for each of the incremental backups |
| that point to their respective full backups: |
| |
| .. code:: bash |
| |
| $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ |
| -b drive0.full.qcow2 -F qcow2 |
| $ qemu-img create -f qcow2 drive1.inc0.qcow2 \ |
| -b drive1.full.qcow2 -F qcow2 |
| |
| #. Issue a multi-drive incremental push backup transaction: |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "transaction", |
| "arguments": { |
| "actions": [ |
| { |
| "type": "drive-backup", |
| "data": { |
| "device": "drive0", |
| "bitmap": "bitmap0", |
| "format": "qcow2", |
| "mode": "existing", |
| "sync": "incremental", |
| "target": "drive0.inc0.qcow2" |
| } |
| }, |
| { |
| "type": "drive-backup", |
| "data": { |
| "device": "drive1", |
| "bitmap": "bitmap0", |
| "format": "qcow2", |
| "mode": "existing", |
| "sync": "incremental", |
| "target": "drive1.inc0.qcow2" |
| } |
| }, |
| ] |
| } |
| } |
| |
| <- { "return": {} } |
| |
| ... |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "type": "backup", |
| "speed": 0, |
| "len": 68719476736, |
| "offset": 68719476736 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| ... |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive1", |
| "type": "backup", |
| "speed": 0, |
| "len": 68719476736, |
| "offset": 68719476736 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| ... |
| |
| Push Backup Errors & Recovery |
| ----------------------------- |
| |
| In the event of an error that occurs after a push backup job is successfully |
| launched, either by an individual QMP command or a QMP transaction, the user |
| will receive a ``BLOCK_JOB_COMPLETE`` event with a failure message, |
| accompanied by a ``BLOCK_JOB_ERROR`` event. |
| |
| In the case of a job being cancelled, the user will receive a |
| ``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR |
| events. |
| |
| In either failure case, the bitmap used for the failed operation is not |
| cleared. It will contain all of the dirty bits it did at the start of the |
| operation, plus any new bits that got marked during the operation. |
| |
| Effectively, the "point in time" that a bitmap is recording differences |
| against is kept at the issuance of the last successful incremental backup, |
| instead of being moved forward to the start of this now-failed backup. |
| |
| Once the underlying problem is addressed (e.g. more storage space is allocated |
| on the destination), the incremental backup command can be retried with the |
| same bitmap. |
| |
| Example: Individual Failures |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| Incremental Push Backup jobs that fail individually behave simply as |
| described above. This example demonstrates the single-job failure case: |
| |
| #. Create a target image: |
| |
| .. code:: bash |
| |
| $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ |
| -b drive0.full.qcow2 -F qcow2 |
| |
| #. Attempt to create an incremental backup via QMP: |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "drive-backup", |
| "arguments": { |
| "device": "drive0", |
| "bitmap": "bitmap0", |
| "target": "drive0.inc0.qcow2", |
| "format": "qcow2", |
| "sync": "incremental", |
| "mode": "existing" |
| } |
| } |
| |
| <- { "return": {} } |
| |
| #. Receive a pair of events indicating failure: |
| |
| .. code-block:: QMP |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "action": "report", |
| "operation": "write" |
| }, |
| "event": "BLOCK_JOB_ERROR" |
| } |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "speed": 0, |
| "offset": 0, |
| "len": 67108864, |
| "error": "No space left on device", |
| "device": "drive0", |
| "type": "backup" |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| #. Delete the failed image, and re-create it. |
| |
| .. code:: bash |
| |
| $ rm drive0.inc0.qcow2 |
| $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ |
| -b drive0.full.qcow2 -F qcow2 |
| |
| #. Retry the command after fixing the underlying problem, such as |
| freeing up space on the backup volume: |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "drive-backup", |
| "arguments": { |
| "device": "drive0", |
| "bitmap": "bitmap0", |
| "target": "drive0.inc0.qcow2", |
| "format": "qcow2", |
| "sync": "incremental", |
| "mode": "existing" |
| } |
| } |
| |
| <- { "return": {} } |
| |
| #. Receive confirmation that the job completed successfully: |
| |
| .. code-block:: QMP |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "type": "backup", |
| "speed": 0, |
| "len": 67108864, |
| "offset": 67108864 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| Example: Partial Transactional Failures |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| QMP commands like `drive-backup <qemu-qmp-ref.html#index-drive_002dbackup>`_ |
| conceptually only start a job, and so transactions containing these commands |
| may succeed even if the job it created later fails. This might have surprising |
| interactions with notions of how a "transaction" ought to behave. |
| |
| This distinction means that on occasion, a transaction containing such job |
| launching commands may appear to succeed and return success, but later |
| individual jobs associated with the transaction may fail. It is possible that |
| a management application may have to deal with a partial backup failure after |
| a "successful" transaction. |
| |
| If multiple backup jobs are specified in a single transaction, if one of those |
| jobs fails, it will not interact with the other backup jobs in any way by |
| default. The job(s) that succeeded will clear the dirty bitmap associated with |
| the operation, but the job(s) that failed will not. It is therefore not safe |
| to delete any incremental backups that were created successfully in this |
| scenario, even though others failed. |
| |
| This example illustrates a transaction with two backup jobs, where one fails |
| and one succeeds: |
| |
| #. Issue the transaction to start a backup of both drives. |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "transaction", |
| "arguments": { |
| "actions": [ |
| { |
| "type": "drive-backup", |
| "data": { |
| "device": "drive0", |
| "bitmap": "bitmap0", |
| "format": "qcow2", |
| "mode": "existing", |
| "sync": "incremental", |
| "target": "drive0.inc0.qcow2" |
| } |
| }, |
| { |
| "type": "drive-backup", |
| "data": { |
| "device": "drive1", |
| "bitmap": "bitmap0", |
| "format": "qcow2", |
| "mode": "existing", |
| "sync": "incremental", |
| "target": "drive1.inc0.qcow2" |
| } |
| }] |
| } |
| } |
| |
| #. Receive notice that the Transaction was accepted, and jobs were |
| launched: |
| |
| .. code-block:: QMP |
| |
| <- { "return": {} } |
| |
| #. Receive notice that the first job has completed: |
| |
| .. code-block:: QMP |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "type": "backup", |
| "speed": 0, |
| "len": 67108864, |
| "offset": 67108864 |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| #. Receive notice that the second job has failed: |
| |
| .. code-block:: QMP |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive1", |
| "action": "report", |
| "operation": "read" |
| }, |
| "event": "BLOCK_JOB_ERROR" |
| } |
| |
| ... |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "speed": 0, |
| "offset": 0, |
| "len": 67108864, |
| "error": "Input/output error", |
| "device": "drive1", |
| "type": "backup" |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| At the conclusion of the above example, ``drive0.inc0.qcow2`` is valid and |
| must be kept, but ``drive1.inc0.qcow2`` is incomplete and should be |
| deleted. If a VM-wide incremental backup of all drives at a point-in-time is |
| to be made, new backups for both drives will need to be made, taking into |
| account that a new incremental backup for drive0 needs to be based on top of |
| ``drive0.inc0.qcow2``. |
| |
| For this example, an incremental backup for ``drive0`` was created, but not |
| for ``drive1``. The last VM-wide crash-consistent backup that is available in |
| this case is the full backup: |
| |
| .. code:: text |
| |
| [drive0.full.qcow2] <-- [drive0.inc0.qcow2] |
| [drive1.full.qcow2] |
| |
| To repair this, issue a new incremental backup across both drives. The result |
| will be backup chains that resemble the following: |
| |
| .. code:: text |
| |
| [drive0.full.qcow2] <-- [drive0.inc0.qcow2] <-- [drive0.inc1.qcow2] |
| [drive1.full.qcow2] <-------------------------- [drive1.inc1.qcow2] |
| |
| Example: Grouped Completion Mode |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| While jobs launched by transactions normally complete or fail individually, |
| it's possible to instruct them to complete or fail together as a group. QMP |
| transactions take an optional properties structure that can affect the |
| behavior of the transaction. |
| |
| The ``completion-mode`` transaction property can be either ``individual`` |
| which is the default legacy behavior described above, or ``grouped``, detailed |
| below. |
| |
| In ``grouped`` completion mode, no jobs will report success until all jobs are |
| ready to report success. If any job fails, all other jobs will be cancelled. |
| |
| Regardless of if a participating incremental backup job failed or was |
| cancelled, their associated bitmaps will all be held at their existing |
| points-in-time, as in individual failure cases. |
| |
| Here's the same multi-drive backup scenario from `Example: Partial |
| Transactional Failures`_, but with the ``grouped`` completion-mode property |
| applied: |
| |
| #. Issue the multi-drive incremental backup transaction: |
| |
| .. code-block:: QMP |
| |
| -> { |
| "execute": "transaction", |
| "arguments": { |
| "properties": { |
| "completion-mode": "grouped" |
| }, |
| "actions": [ |
| { |
| "type": "drive-backup", |
| "data": { |
| "device": "drive0", |
| "bitmap": "bitmap0", |
| "format": "qcow2", |
| "mode": "existing", |
| "sync": "incremental", |
| "target": "drive0.inc0.qcow2" |
| } |
| }, |
| { |
| "type": "drive-backup", |
| "data": { |
| "device": "drive1", |
| "bitmap": "bitmap0", |
| "format": "qcow2", |
| "mode": "existing", |
| "sync": "incremental", |
| "target": "drive1.inc0.qcow2" |
| } |
| }] |
| } |
| } |
| |
| #. Receive notice that the Transaction was accepted, and jobs were launched: |
| |
| .. code-block:: QMP |
| |
| <- { "return": {} } |
| |
| #. Receive notification that the backup job for ``drive1`` has failed: |
| |
| .. code-block:: QMP |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive1", |
| "action": "report", |
| "operation": "read" |
| }, |
| "event": "BLOCK_JOB_ERROR" |
| } |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "speed": 0, |
| "offset": 0, |
| "len": 67108864, |
| "error": "Input/output error", |
| "device": "drive1", |
| "type": "backup" |
| }, |
| "event": "BLOCK_JOB_COMPLETED" |
| } |
| |
| #. Receive notification that the job for ``drive0`` has been cancelled: |
| |
| .. code-block:: QMP |
| |
| <- { |
| "timestamp": {...}, |
| "data": { |
| "device": "drive0", |
| "type": "backup", |
| "speed": 0, |
| "len": 67108864, |
| "offset": 16777216 |
| }, |
| "event": "BLOCK_JOB_CANCELLED" |
| } |
| |
| At the conclusion of *this* example, both jobs have been aborted due to a |
| failure. Both destination images should be deleted and are no longer of use. |
| |
| The transaction as a whole can simply be re-issued at a later time. |
| |
| .. raw:: html |
| |
| <!-- |
| The FreeBSD Documentation License |
| |
| Redistribution and use in source (ReST) and 'compiled' forms (SGML, HTML, |
| PDF, PostScript, RTF and so forth) with or without modification, are |
| permitted provided that the following conditions are met: |
| |
| Redistributions of source code (ReST) must retain the above copyright notice, |
| this list of conditions and the following disclaimer of this file unmodified. |
| |
| Redistributions in compiled form (transformed to other DTDs, converted to |
| PDF, PostScript, RTF and other formats) must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| |
| THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS |
| IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE |
| IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE |
| ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE |
| LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR |
| CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF |
| SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS |
| INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN |
| CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) |
| ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENTATION, EVEN IF ADVISED OF |
| THE POSSIBILITY OF SUCH DAMAGE. |
| --> |