Google Git
Sign in
fuchsia / fuchsia / dd04239d5d9374032e1358fc204f405bf21e0025 / . / src / ledger / bin / tests / benchmark
tree: 6477549b45343fc7e52ce733fe9d66b479fa224f [path history] [tgz]
  1. backlog/
  2. convergence/
  3. delete_entry/
  4. disk_space/
  5. fetch/
  6. get_entry/
  7. get_page/
  8. put/
  9. sync/
  10. update_entry/
  11. BUILD.gn
  12. README.md
src/ledger/bin/tests/benchmark/README.md

benchmark

Overview

This directory contains Ledger trace-based benchmarks. Each benchmark is implemented as a client connected to Ledger and using its API to perform one or more of its basic operations (creating a page, writing/deleting entries, ...).

Run benchmarks

Using trace spec files

The easiest way to run a benchmark is using the associated trace spec file, for example:

trace record --spec-file=/pkgfs/packages/ledger_benchmarks/0/data/put.tspec

You can read more on the trace spec file format in the trace-based benchmarking guide.

Tracing benchmark apps directly

You can also trace the app directly:

trace record --categories=benchmark,ledger \
  fuchsia-pkg://fuchsia.com/ledger_benchmarks#meta/put.cmx \
  --entry-count=10 --transaction-size=1 --key-size=100 --value-size=100 \
  --refs=off

That would generate a trace result file (default: /data/trace.json), which can be analysed manually or using special tools.

Benchmark parameters

*.tspec files specify the parameters with which benchmark apps are run. You can override these by passing the “--append-args” argument to the trace record tool. Commonly used among all the ledger benchmarks are the following parameters:

  • entry-count for the number of entries to perform operations on
  • unique-key-count if the number of operations and the number of entries differ (i.e. some entries are overwritten), this denotes the number of unique entries in the page
  • key-size for the size of the key (in bytes)
  • value-size for the size of the value (in bytes)
  • refs with the values on or off for the reference strategy: if set to on, entries will be put using CreateReference/PutAsReference, otherwise they will be treated as inline entries.
  • commit-count for the number of commits made to the page
  • transaction-size for the number of operations in a single transaction

Unless the name of the benchmark suggest otherwise, default values are:

  • 100 entries
  • key size 100
  • value size 1000 Benchmarks under sync and convergence use smaller number of entries and smaller value size.

Benchmarks using sync

Some benchmarks exercise cloud synchronization using cloud_provider_firestore.

To run these, follow the cloud sync set-up instructions to set up a Firestore instance, configure the build environment and obtain the sync parameters needed below.

Then, run the selected benchmark as follows:

trace record --spec-file=/pkgfs/packages/ledger_benchmarks/0/data/sync.tspec

A note regarding benchmark apps

Since the benchmark apps are designed to be used with tracing, running them without a tracing will not generate any results.

List of benchmarks

Some of the benchmark apps have several corresponding tspec files, to exercise different possible scenarios of using Ledger. For example, get_page benchmark is used in add_new_page.tspec to emulate creating many pages with different IDs, and in get_same_page.tspec to create several connections to the same page.

Local benchmarks

  • Get page: How long does it take to establish a new page connection?
    • add_new_page.tspec: connection to the new page with previously unused ID.
    • add_new_page_precached.tspec: same as above, but waits for a precached Ledger Page to be ready before each request.
    • add_new_page_after_clear.tspec: same as add_new_page.tspec but each previous page is populated with one entry, then cleared and evicted.
    • get_same_page.tspec: several connections to the same page
    • get_page_id.tspec: how long does the GetId() call takes on a newly created page?
  • Put: How long does it take to write data to a page? And how long before the client will receive a PageWatcher notification about its own change?
    • put.tspec: basic case
    • put_big_entry.tspec: writing big entries to the page
    • put_as_reference.tspec: entries are put as references (CreateReference + PutReference)
    • transaction.tspec: changes are made in a transaction
    • put_memory.tspec: how much memory is Ledger using after every insertion in the basic case?
  • Update entry: How long does it take to update an existing value in Ledger (make several Put operations with the same key, but different values)?
    • update_entry.tspec: basic case
    • update_big_entry.tspec: put an entry of big size, then update its value
    • update_entry_transactions.tspec: changes are grouped in transactions
  • Get entry: How long does it take to retrieve a single value from a page? This benchmark also measures the time taken by the GetSnapshot call and the GetKeys call to retrieve all keys from the page.
    • get_small_entry.tspec: basic case
    • get_small_entry_inline.tspec: GetInline is used instead of Get
    • get_big_entry.tspec: values of bigger size are used.
  • Delete entry: How long does it take to delete an entry from a page?
    • delete_entry.tspec: each entry is deleted separately (outside of a transaction)
    • delete_big_entry.tspec: same as above, but for big entries
    • delete_entry_transactions.tspec: deletions are grouped in transactions
    • disk_space_cleared_page.tspec: how much space does ledger take after the page was cleared out (all the entries deleted in one transaction)?
  • Disk space: How much disk space does ledger use to store pages, objects and commits?
    • disk_space_empty_ledger.tspec: empty ledger (with no pages)
    • disk_space_empty_pages.tspec: ledger containing only empty pages
    • disk_space_entries.tspec: ledger with one page containing some entries
    • disk_space_small_keys.tspec: same, but with small (10 bytes) keys.
    • disk_space_updates.tspec: ledger with one page containing only one entry, but long commit history
    • disk_space_one_commit_per_entry.tspec: ledger with one page containing several entries, each of them added in a separate commit

Sync benchmarks

These benchmarks exercise synchronisation and need an ID of a Firestore instance passed to them as described in a previous section.

  • Backlog: How long does it take to download all existing data when establishing a new connection to an already populated data?
    • backlog.tspec: basic case
    • backlog_big_entry.tspec: page contains one entry, but of a big size
    • backlog_big_entry_updates.tspec: one big entry, but that was updated several times prior to the new connection (commit history)
    • backlog_many_big_entries.tspec: page contains several big entries
    • backlog_many_small_entries.tspec: many small entries
    • backlog_small_entry_updates.tspec: small entry, but a long commit history
    • disk_space_synced_entries.tspec: how much disk space does ledger take on a writer and a reader device, when several entries have been written (in one commit) on one device and then downloaded on another?
    • disk_space_synced_entries_small_keys.tspec: same, but with small (10 bytes) keys.
    • disk_space_synced_updates.tspec: how much disk space does ledger take on a writer and a reader device, when several commits with updates has been made on one device and then downloaded on another?
  • Convergence: Several devices make concurrent changes to the page. How long does it take for all devices to see each other changes?
    • convergence.tspec: two devices
    • multidevice_convergence: several devices
  • Fetch: How long does it take to fetch a lazy value?
    • fetch.tspec: basic case
    • fetch_partial_big_entry.tspec: using FetchPartial (fetch in several parts) on a big entry
  • Sync: When one device makes changes to the page, how long does it take for another one to receive these changes?
    • sync.tspec: basic case
    • sync_big_change.tspec: syncing a big change (containing several write operations)