# Storage versioning

## Current status

This document describes the practices adopted by the local storage team but implementing these rules
has not yet been completed for all storage formats.

## Background and requirements

Our on-disk [storage systems](/docs/concepts/filesystems/filesystems.md) need to persist across
updates and multiple versions of the system may need to interact with this data. There will be some
changes that will not be be backwards-compatible and software must avoid changing data it can not
full understand. There can also be minor updates to the storage format that are backwards-compatible
but may need special handling.

As an example for special handling of different versions of backwards-compatible formats, say there
is a minor bug that was corrected that made some previously valid data layout no longer permitted:

  * Old code should still be allowed to use the device since it can still read all of the data
    and can still write a format understood by newer versions.

  * New code would like to correct the no-longer-valid format to a newer format when it occurs and
    it should be able to tell when this migration is required.

  * Utilities such as `fsck` need to know exactly what format to expect. If it sees the invalid
    structure in a device written only by the newer version of the code, it knows there is a serious
    error. But if the device was written to by older code, it knows that this is expected and to
    continue.

## Concepts

  * **Major version:** The non-backwards-compatible version of the format on device. Different
    major versions are not compatible. Any non-backwards-compatible changes should increment the
    major version.

  * **Minor version:** Backwards-compatible changes to how data is stored on the disk should update
    the minor version.

  * **Oldest minor version:** The oldest minor version of the software that has written to the
    device.

## Requirements

Persistent storage systems should maintain two numbers in the header of their data:

  * Major version.
  * _Oldest_ minor version.

Systems should encode data in a way that allows formats to be added without invalidating older
versions when possible. For example, if compression is supported in a filesystem, the compression
algorithm should be stored on the file. This allows adding additional compression algorithms without
invalidating older data.

To support future updates, there should be some reserved bytes in the metadata if possible, and all
reserved regions in metadata should be zero initialized at format time. Unit tests should verify
this. Verification tools should be lenient about checking reserved sections i.e. they should *not*
check that reserved sections are zeroed. Similarly, length fields that allow structures to expand
should be loosely checked. If it makes sense to do so, consider adding a "strict" option (disabled
by default) that performs these checks.

Metrics for the major version and oldest minor version should be available via Cobalt.

## Maintaining and using versions

#### Upon formatting a device

When a device is initially created, the current on-disk format's major and minor version should be
written to the header.

#### Upon opening a device

Software that opens a device should first check the major version. If the major version is
larger than expected, the operation should fail and no operations should be attempted with the
device.

Software that opens a device for writing should next check the device's current oldest minor
version. If the software minor version is less than the oldest minor version stored on the device,
it should update the device's oldest minor version to the current software minor version and
continue. Software with newer minor versions should not increase the oldest minor version without
performing an update.

#### Upon performing minor updates

Sometimes a migration may need to be done. For the example given in the "Background" section, we may
want to check for and correct the newly-invalid format. In this case, the software can check whether
the update is required by looking at the oldest minor version of the device. If it is before the
minor version with the fix, the software knows that there may be data on the device that has the
older format and should perform an update.

If the persistent data is updated so that none of it can be considered to have been written by an
older minor version, the oldest minor version value should be update to the current value. This will
prevent performing the migration in the future so long as no older minor versions of the software
write to the device.

## Why keep only the oldest minor version?

Why do we keep the oldest minor version rather than the newest one that has written to the disk?
Most systems use only the newest version.

We expect increments to the (backwards-compatible) minor version to be of two general types:
additions of new features that older versions of the software can ignore, and increasing the
strictness of requirements of the format.

In both of these cases, if an older version of the software writes to the disk, assumptions about
the data added in a newer version may become invalid. For example, added tracking information may
get out-of-sync because the older version doesn't know how to update it, or the older version may
re-introduce subsequently disallowed patterns. If version 3.0 writes to a version 3.1 disk, it's
not necessarily version 3.1 any more and we don't want to treat it as such. In these cases, the
newer versions of the sofware will want to perform upgrade-specific checks or migrations.

How can a newer version of the software know if something it made exists if the disk has only the
oldest version? Say a new data format was added or a new table was added to the metadata, how does
the newer version know if the device has these structures? For backwards-compatible additions, it
should just check for the presence of the structures which should always be additive. New data
should be added in 0-initialized reserved regions so this is always possible. As mentioned earlier,
checks on these reserved areas should be lenient. For more subtle changes, bits can be added to the
reserved region that indicates a change happened or something might be present.