blob: 304059e32fe8978975bb7f341c73656cbd26a761 [file] [view]
# Memory Sampler
`memory_sampler` is a continuous fleetwide sampling heap profiler:
* Continuous: continuously capture profiles from a running process.
* Fleetwide: designed to run on devices in the field; profiles are attached to
crash reports, and can be similarly accessed.
* Sampling: uses a poisson process to randomly sample allocations based on
size; this reduces the overhead of the profiler, and makes it suitable to
get a view of the more interesting allocations without capturing every
single allocation of a process.
* Heap profiler: produces (`pprof` compatible) profiles of the heap
allocations and deallocations of the instrumented process.
`memory_sampler` comes with three parts:
* `fuchsia.memory.sampler.Sampler`: a FIDL API which describes an interface to
communicate allocation metadata between a process and a profiler.
* `memory_sampler`: a Rust component which provides an implementation for this
API.
* `libsampler_instrumentation`: a shared library that one can link against to
automatically instrument the allocator and communicate allocation metadata
through the FIDL API.
`memory_sampler` supports profiling arbitrary processes that use the platform
default allocator, Scudo. This includes any C, C++ and Rust program built via
the SDK that don't configure an alternative allocator.
## Getting started
`memory_sampler` can be used to continuously profile in-tree
components. Out-of-tree components are not supported yet, but will once the
relevant parts have been added to the Fuchsia SDK.
### In-tree components
To profile a component with `memory_profiler`:
1. Add the `memory_sampler` Fuchsia package target to your build (e.g. `fx set
... --with //src/performance/memory/sampler:memory_sampler`).
2. Add this package in an appropriate realm. See
`//src/performance/memory/sampler/meta/memory_sampler.core_shard.cml` for
an example shard to include in the `core` realm manifest, to include
`memory_sampler` as a `core` component. `memory_sampler`'s URL is
`fuchsia-pkg://fuchsia.com/memory_sampler#meta/memory_sampler.cm`, and it
depends on the `fuchsia.feedback.CrashReporter` and
`fuchsia.feedback.CrashReportingProductRegister` capabilities, both offered
by the `feedback` `core` component.
3. Route the `fuchsia.memory.sampler.Sampler` capability from `memory_sampler`
to the instrumented process.
4. Add the `//src/performance/memory/sampler/instrumentation:lib` shared
library as a dependency of your binary.
When your component starts, `component_manager` will ensure the configured
instance of `memory_sampler` is running, and your process will regularly
communicate allocation information to `memory_sampler`.
At most once an hour, `memory_sampler` will file a crash report with the
`feedback` service that contains a `pprof`-compatible profile. This profile can
be symbolized and visualized by running the `fx pprof -flame
<path_to_the_profile>` command.
Note: on `core`, `feedback` will write the filed crash reports in a temporary
storage on the device. They can be found via the `fx shell "find /tmp /data
-name <your_program_name>*"` command.
Note: profiles are `gzip`-compressed, which substantially reduces the memory
held by profiles awaiting filing, as well as the storage they occupy on the
device. They are named with the `.pb.gz` extension accordingly. Tooling that
understands the `pprof` format, such as `fx pprof`, detects this and
decompresses profiles transparently; ad-hoc tooling that reads profiles as raw
protobuf needs to decompress them first.
### Out-of-tree components
Currently, out-of-tree components won't have access to `memory_sampler` because
it is entirely internal. However, we do plan to make it generally available via
the SDK; stay tuned.
## Memory Profiles
### Content
The profiles produced by `memory_sampler` contain 6 types of samples:
* `residual_allocated_objects`: a count of (sampled) allocations that are
still alive at the time of the production of the profile.
* `residual_allocated_space`: the size of (sampled) allocations that are still
alive at the time of the production of the profile.
* `allocated_objects`: a count of (sampled) allocations observed over the
duration of the profile (both alive and dead).
* `allocated_space`: the size of (sampled) allocations observed over the
duration of the profile (both alive and dead).
* `deallocated_objects`: a count of (sampled) deallocations observed over the
duration of the profile.
* `deallocated_space`: the size of (sampled) deallocations observed over the
duration of the profile.
Note: because `memory_sampler` is a sampling profiler, it only observes a
(randomly selected) subset of allocations. Moreover, the selection is based on
average memory allocated between samples: it skews the distribution towards
larger allocations. For this reason, counts and space are always underestimated,
but how much depends on the allocation profile of the instrumented process
(e.g. a process that only does large allocations is more likely to produce an
accurate profile than a process that performs a lot of very small
allocations). Nevertheless, outliers are still likelier to get sampled; chances
are that if your process suffers from an unforeseen pathological allocation
pattern, they will tend to show up on profiles.
### Profile filing frequency
`memory_sampler` produces profiles under roughly three conditions:
1. A partial profile once the number of recorded stack traces that can
be reclaimed reaches a threshold, to reduce memory use. This can be
somewhat unpredictable, because it depends on the (de)allocation
patterns and call sites.
2. A partial profile at least once every 12 hours.
3. A final profile when the instrumented process exits.
### Partial vs final profiles
`memory_sampler` regularly files partial profiles during the lifetime of the
instrumented process (depending on observed allocations), as well as a single
final profile at the end of the process. Both kind of profiles have the same shape, but different semantics:
* Partial profiles: residual allocations are either leaks *or* allocations
that have simply not been deallocated yet. Dead allocations correspond to
allocations that have been deallocated *within the period covered by this
profile*. In particular, if an allocation was deallocated in profile X, it
won't appear in the list of dead allocations of profile X+N. This is done to
restrict the growth of partial profiles.
* Final profiles: residual allocations are allocations that were never
deallocated within the lifetime of the process. If an instrumented process
has memory leaks, they are very likely to appear here (if they were
sampled). Dead allocations correspond to allocations that have been
deallocated *between the last partial profile and the end of the process*,
not over the entire lifetime of the process. This limitation is the result
of a memory optimization.
Note: Multiple partial profiles (and a final profile) produced by the same
process within a single reporting interval are automatically coalesced into one
profile before being filed. If one needs a picture of the overall allocations of
a process that spans multiple crash reports, one can similarly merge its partial
profiles (discarding their residual allocations) with its final profile.
Note: There is a hard limit on the number of attachments that can fit in a
report. By default, `memory_sampler` adds profiles to a report as separate
attachments, and merges profiles from the same process, but if more distinct
processes than allowed attachments produce profiles within the same reporting
interval, `memory_sampler` will merge the excess profiles into a single
`combined_profiles.tar` archive (using the same `<process_name>_final_<n>.pb.gz`
and `<process_name>_partial_<n>.pb.gz` entry names as individual
attachments). Individual `.pb.gz` profiles can be extracted with `tar -xf
combined_profiles.tar`, and `pprof` can merge multiple extracted profiles when
passed together on the command line.
## Performance tuning
Currently, `memory_sampler` hard-codes all its performance parameters; on a
private build, we encourage you to tune them to your liking. Some available
performance knobs follow:
* `memory_sampler::Recorder::kSamplingIntervalBytes`: the average count of
bytes allocated between two samples. Reducing this value increases the
accuracy of the profiler, at the expense of the performance of the
instrumented process. Note that if the value becomes so small that
`memory_sampler` is unable to handle the amount of messages it receives,
this would cause the kernel to kill the instrumented process (because of a
buffer exhaustion in the FIDL channel).
* `memory_sampler::sampler_service::RECLAIMABLE_STACK_TRACES_PROFILE_THRESHOLD`:
the number of recorded stack traces that triggers a partial report. Reducing
this number decreases the memory footprint of `memory_sampler`, at the
expense of storage space and bandwidth (because more, smaller profiles get
filed as a result).
* `memory_sampler::sampler_service::MAX_DURATION_BETWEEN_PARTIAL_PROFILES`:
the maximum elapsed time between two partial profiles. Reducing
this duration will increase the frequency of partial profiles,
which should in turn reduce the memory consumption of the
profiler.
* `memory_sampler::profile_builder::PROFILE_COMPRESSION_LEVEL`: the `gzip`
compression level applied to profiles. Reducing it lowers the CPU spent
compressing, at the expense of the memory held by profiles awaiting filing;
note that the returns diminish quickly above it.
* `memory_sampler::crash_reporter::MAX_PENDING_REPORT_BYTES`: the maximum
total VMO footprint of profiles held in memory for the next crash report.
Increasing this value allows more profiles to be accumulated and coalesced
within a reporting window before applying backpressure to the queue, at the
expense of `memory_sampler`'s memory footprint and crash report size.
Note also that `memory_sampler` comes with built-in throttling of filed
profiles, to limit the rate of filing crash reports; in an `eng` build,
`feedback` does not upload any profile, so it is safe to reduce
`memory_sampler::crash_reporter::MIN_DURATION_BETWEEN_SNAPSHOTS_HOURS` to file
profiles more often (both to reduce latency between capture and consumption of
profiles, as well as to increase the sampling rate without significantly
increasing `memory_sampler`'s memory footprint).