src/performance/memory/sampler/README.md

# Memory Sampler

`memory_sampler` is a continuous fleetwide sampling heap profiler:

  * Continuous: continuously capture profiles from a running process.
  * Fleetwide: designed to run on devices in the field; profiles are attached to
    crash reports, and can be similarly accessed.
  * Sampling: uses a poisson process to randomly sample allocations based on
    size; this reduces the overhead of the profiler, and makes it suitable to
    get a view of the more interesting allocations without capturing every
    single allocation of a process.
  * Heap profiler: produces (`pprof` compatible) profiles of the heap
    allocations and deallocations of the instrumented process.

`memory_sampler` comes with three parts:

  * `fuchsia.memory.sampler.Sampler`: a FIDL API which describes an interface to
    communicate allocation metadata between a process and a profiler.
  * `memory_sampler`: a Rust component which provides an implementation for this
    API.
  * `libsampler_instrumentation`: a shared library that one can link against to
    automatically instrument the allocator and communicate allocation metadata
    through the FIDL API.

`memory_sampler` supports profiling arbitrary processes that use the platform
default allocator, Scudo. This includes any C, C++ and Rust program built via
the SDK that don't configure an alternative allocator.

## Getting started

`memory_sampler` can be used to continuously profile in-tree
components. Out-of-tree components are not supported yet, but will once the
relevant parts have been added to the Fuchsia SDK.

### In-tree components

To profile a component with `memory_profiler`:

  1. Add the `memory_sampler` Fuchsia package target to your build (e.g. `fx set
     ... --with //src/performance/memory/sampler:memory_sampler`).
  2. Add this package in an appropriate realm.  See
     `//src/performance/memory/sampler/meta/memory_sampler.core_shard.cml` for
     an example shard to include in the `core` realm manifest, to include
     `memory_sampler` as a `core` component. `memory_sampler`'s URL is
     `fuchsia-pkg://fuchsia.com/memory_sampler#meta/memory_sampler.cm`, and it
     depends on the `fuchsia.feedback.CrashReporter` and
     `fuchsia.feedback.CrashReportingProductRegister` capabilities, both offered
     by the `feedback` `core` component.

  3. Route the `fuchsia.memory.sampler.Sampler` capability from `memory_sampler`
     to the instrumented process.

  4. Add the `//src/performance/memory/sampler/instrumentation:lib` shared
     library as a dependency of your binary.

When your component starts, `component_manager` will ensure the configured
instance of `memory_sampler` is running, and your process will regularly
communicate allocation information to `memory_sampler`.

At most once an hour, `memory_sampler` will file a crash report with the
`feedback` service that contains a `pprof`-compatible profile. This profile can
be symbolized and visualized by running the `fx pprof -flame
<path_to_the_profile>` command.

Note: on `core`, `feedback` will write the filed crash reports in a temporary
storage on the device. They can be found via the `fx shell "find /tmp /data
-name <your_program_name>*"` command.

### Out-of-tree components

Currently, out-of-tree components won't have access to `memory_sampler` because
it is entirely internal. However, we do plan to make it generally available via
the SDK; stay tuned.

## Memory Profiles

### Content

The profiles produced by `memory_sampler` contain 6 types of samples:

  * `residual_allocated_objects`: a count of (sampled) allocations that are
    still alive at the time of the production of the profile.
  * `residual_allocated_space`: the size of (sampled) allocations that are still
    alive at the time of the production of the profile.
  * `allocated_objects`: a count of (sampled) allocations observed over the
    duration of the profile (both alive and dead).
  * `allocated_space`: the size of (sampled) allocations observed over the
    duration of the profile (both alive and dead).
  * `deallocated_objects`: a count of (sampled) deallocations observed over the
    duration of the profile.
  * `deallocated_space`: the size of (sampled) deallocations observed over the
    duration of the profile.

Note: because `memory_sampler` is a sampling profiler, it only observes a
(randomly selected) subset of allocations. Moreover, the selection is based on
average memory allocated between samples: it skews the distribution towards
larger allocations. For this reason, counts and space are always underestimated,
but how much depends on the allocation profile of the instrumented process
(e.g. a process that only does large allocations is more likely to produce an
accurate profile than a process that performs a lot of very small
allocations). Nevertheless, outliers are still likelier to get sampled; chances
are that if your process suffers from an unforeseen pathological allocation
pattern, they will tend to show up on profiles.

### Profile filing frequency

`memory_sampler` produces profiles under roughly three conditions:

  1. A partial profile once the recorded allocation data reaches a
     certain size, to reduce memory use. This can be somewhat
     unpredictable, because it depends on the (de)allocation patterns
     and call sites.

  2. A partial profile at least once every 12 hours.

  3. A final profile when the instrumented process exits.

### Partial vs final profiles

`memory_sampler` regularly files partial profiles during the lifetime of the
instrumented process (depending on observed allocations), as well as a single
final profile at the end of the process. Both kind of profiles have the same shape, but different semantics:

  * Partial profiles: residual allocations are either leaks *or* allocations
    that have simply not been deallocated yet. Dead allocations correspond to
    allocations that have been deallocated *within the period covered by this
    profile*. In particular, if an allocation was deallocated in profile X, it
    won't appear in the list of dead allocations of profile X+N. This is done to
    restrict the growth of partial profiles.  Partial profiles also come with an
    iteration number; this number is meaningless, except that profiles come in
    numerical order (i.e. profile 1000 was captured earlier than profile 9999
    regardless of when they were captured).
  * Final profiles: residual allocations are allocations that were never
    deallocated within the lifetime of the process. If an instrumented process
    has memory leaks, they are very likely to appear here (if they were
    sampled). Dead allocations correspond to allocations that have been
    deallocated *between the last partial profile and the end of the process*,
    not over the entire lifetime of the process. This limitation is the result
    of a memory optimization.

Note: If one needs a picture of the overall allocations of a process over its
lifetime, one should look into every single profile captured during the lifetime
of the process; it's possible to get an accurate summary by merging every single
partial profile (discarding live allocations) with a final profile, but we don't
provide (yet) a script to perform this task.

## Performance tuning

Currently, `memory_sampler` hard-codes all its performance parameters; on a
private build, we encourage you to tune them to your liking. Some available
performance knobs follow:

  * `memory_sampler::Recorder::kSamplingIntervalBytes`: the average count of
    bytes allocated between two samples. Reducing this value increases the
    accuracy of the profiler, at the expense of the performance of the
    instrumented process. Note that if the value becomes so small that
    `memory_sampler` is unable to handle the amount of messages it receives,
    this would cause the kernel to kill the instrumented process (because of a
    buffer exhaustion in the FIDL channel).
  * `memory_sampler::sampler_service::DEAD_ALLOCATIONS_PROFILE_THRESHOLD`: the
    amount of observed allocations before filing a partial report. Reducing this
    number decreases the memory footprint `memory_sampler`, at the expense of
    storage space and bandwidth (because more, smaller profiles get filed as a
    result).
  * `memory_sampler::sampler_service::MAX_DURATION_BETWEEN_PROFILES`:
    the maximum elpased time between two partial profiles. Reducing
    this duration will increase the frequency of partial profiles,
    which should in turn reduce the memory consumption of the
    profiler.

Note also that `memory_sampler` comes with built-in throttling of filed
profiles, to limit the rate of filing crash reports; in an `eng` build,
`feedback` does not upload any profile, so it is safe to modify
`memory_sampler::crash_reporter::setup_crash_reporter` to file profiles more
often (both to reduce latency between capture and consumption of profiles, as
well as to increase the sampling rate without significantly increasing
`memory_sampler`'s memory footprint).