Fuchsia Standalone Performance Analysis Tool (perf-analyze)

perf-analyze is a host-side tool designed to perform standalone, automated, and human-interactive performance analysis on Fuchsia traces. The tool acts as a CLI orchestrator that delegates trace processing to specialized analysis plugins.

Subcommands

1. query: Executes SQL queries to inspect and extract structured data from traces. Backed by Perfetto's Trace Processor.
2. analyze: Executes specialized analysis plugins to identify performance anomalies (e.g., binder delays, jank, CPU starvation).
3. visualize (Planned): Generates HTML reports, flamegraphs, SVGs, or deep-link trampoline URLs for the Perfetto UI.

Global Options

• --format <json|markdown|text>: Specifies the output format (default: text).
  • text: Unformatted tab-separated values (TSV), ideal for shell scripting.
  • markdown: Formatted Markdown tables, ideal for doc insertion.
  • json: Structured JSON, ideal for automated tool ingestion.

The query Subcommand

The query subcommand executes SQL queries against a trace file using Perfetto's Trace Processor.

Arguments

• --trace <path_or_url> (Required): File path to a local .fxt trace file or a URL to a remote trace.
• --sql <query_string>: Executes a single raw SQL query. Mutually exclusive with --batch.
• --batch <json_string_or_@filepath>: Executes a JSON array of queries in the format [{"name": "query_name", "sql": "select ..."}, ...]. Use @filepath to read the array from a local file. Mutually exclusive with --sql.

Examples

1. Print query help

fx perf-analyze query --help

Output:

usage: perf-analyze query [-h] --trace TRACE (--sql SQL | --batch BATCH)

options:
  -h, --help     show this help message and exit
  --trace TRACE  Trace file path or URL
  --sql SQL      SQL query to run
  --batch BATCH  Batch JSON or @file

2. Run a single SQL query (default TSV format)

fx perf-analyze query \
  --trace src/performance/perf-analyze/test-data/sample_fxt.fxt \
  --sql "select count(*) as cnt from slice"

Output:

cnt
520

3. Run a single SQL query (Markdown format)

fx perf-analyze --format markdown query \
  --trace src/performance/perf-analyze/test-data/sample_fxt.fxt \
  --sql "select count(*) as cnt from slice"

Output:

| cnt |
| --- |
| 520 |

4. Run a batch of queries from a file (JSON format)

fx perf-analyze --format json query \
  --trace src/performance/perf-analyze/test-data/sample_fxt.fxt \
  --batch @src/performance/perf-analyze/test-data/sample_queries.json

Output:

[
  {
    "name": "slice_count",
    "results": [
      {
        "cnt": 520
      }
    ]
  },
  {
    "name": "process_count",
    "results": [
      {
        "cnt": 2
      }
    ]
  }
]

The analyze Subcommand

The analyze subcommand runs specialized analysis plugins against a trace file to identify specific performance anomalies.

Arguments

• --trace <path_or_url> (Required): File path to a local .fxt trace file or a URL to a remote trace.
• --plugin <plugin_name> (Required): Name of the analysis plugin to execute (e.g., binder).
• --list-plugins: Lists all available analysis plugins.

Available Plugins

app-launch (App Launch Critical Path Breakdown)

Decomposes the critical path of Android/Starnix application launches into contiguous chronological phase segments, intra-segment sub-operations, per-segment kernel thread states, and Fuchsia subsystem bottlenecks.

The plugin executes several diagnostic sections in order:

1. Critical Path Phase Breakdown (Sequential Segments): Reports total_launch_ms (the launchingActivity slice) and displayed_ms (from ActivityTaskManager's START to the launch end, which matches logcat Displayed and AtmDisplayTime), and divides each launch into pre_start_ms (intent receipt to START, e.g. intent resolution or lock waits; 0 when the trace has no START marker, as for some hot launches), pre_fork_ms (START to Start proc), process_fork_ms (Zygote fork & specialization), attach_app_ms (ART post-fork init & attachApplication IPC), bind_app_ms (bindApplication), activity_lifecycle_ms (activityStart + activityResume), first_frame_ui_ms (first Choreographer#doFrame UI traversal), first_frame_render_ms (first RenderThread DrawFrame to TTID), and display_present_tail_ms (post-TTID Fuchsia Display::Client::CommitConfig). Warm and hot launches have no fork or bindApplication, so those columns are empty and pre_activity_ms covers the launch start to the first activityStart/activityResume. unattributed_ms is launch time that no segment covers because a milestone is missing, so the segments up to first_frame_render_ms plus unattributed_ms sum to total_launch_ms.
2. CUJ Sub-Stage Breakdown: Quantifies the CUJ sub-stage slices (init, draw, render, traversal, animation, and scenic). Nested slices (e.g. traversal > draw-VRI[...] > Record View#draw()) count only their self time, so no time on a thread is counted twice. Sub-stages on different threads can overlap in time. Reported as raw milliseconds (--scaling raw) or as each sub-stage's share of the attributed work, scaled to total_launch_ms (--scaling proportional).
3. Intra-Segment Sub-Operation Breakdown: Quantifies open_dex_ms, dlopen_ms, verify_class_ms, class_init_ms, get_resources_ms, inflate_ms, relayout_window_ms, dequeue_buffer_ms, shader_compile_ms, and queue_present_ms.
4. Critical Path Thread State Breakdown: Breaks down each critical-path segment into running_cpu_ms (Running), runnable_starved_ms (R/R+/W), and sleeping_blocked_ms (S/D).
5. Cross-Layer Contention & Fuchsia Subsystem Attribution: Quantifies main thread Binder calls (BinderIoctl on Starnix), and Java monitor and native lock contention on the main thread and RenderThread. These count only outermost slices, clipped to the launch. It also reports system-wide Fxfs storage page-ins (page_in_chunk), time sysmem was busy (sysmem_busy_ms, not necessarily on the app's critical path), and Starnix SELinux hook overhead in the app process.
6. Critical Path Process & Thread Breakdown: Expands every sleep along the launch's piecewise thread spine via sched.thread_executing_span (critical_path.py) to rank exact critical-path milliseconds (critical_path_ms, pct_of_total, running_ms, runnable_ms, waking_ms, unattributed_sleep_ms, unknown_ms) across the app, system_server, zygote, sysmem, fxfs, and GPU drivers.
7. Critical Path Top Blocking Slices: Ranks the innermost active slices (self time, or <uninstrumented> when executing without trace slices) along each launch's critical path.
8. Critical Path Chronological Span Timeline: Lists coalesced chronological spans along each launch's critical path, annotating handoff_from whenever execution shifts between threads via wakeup or blocked wait.

Plugin-Specific Arguments

• --package <str>: Optional substring filter for package or process name.
• --limit <int>: Maximum rows returned in sections 1-5 (default: 20); in critical-path sections 6-8, caps analyzed launches to limit and returns up to limit threads/slices and max(limit * 3, 50) timeline spans per launch.
• --scaling <raw|proportional>: Report CUJ sub-stage durations as raw milliseconds (raw, default) or as each sub-stage's share of the attributed work, scaled to total_launch_ms (proportional).

Example

fx perf-analyze --format markdown analyze \
  --trace out/e2e-testout/trace.fxt \
  --plugin app-launch \
  --package settings

critical-path (Thread-Execution Critical Path Analyzer)

Computes the scheduler wakeup and trace flow critical path between any two thread-bound points in a trace using Perfetto's sched.thread_executing_span and intervals.intersect standard library modules.

Valid Start and End Trace Elements

Because scheduler wakeup chains (thread_state / sched_slice) are anchored to OS threads (utid), both --start-* and --end-* selectors must resolve to a thread-bound (utid, ts) point:

1. Thread-track slice by ID (--start-slice-id <id>, --end-slice-id <id>): Selects a specific row in slice whose track is a thread_track.
2. Thread-track slice by glob (--start-slice <glob>, --end-slice <glob>): Matches slice.name GLOB <glob> on thread_track, optionally narrowed with --*-process <glob>, --*-thread <glob>, --*-index <N> (0 for first, -1 for last), and --*-edge <start|end> (defaults: start for the start point, end for the end point).
3. Explicit timestamp + thread (--start-ts <ns> with --start-tid <tid> or --start-utid <utid>, and matching --end-* flags).

Process-scoped or async-track slices (for example launchingActivity#<id> on process_track, where utid IS NULL) are explicitly rejected with an error explaining how to anchor on a thread-track slice (such as Start proc:* or the instant launchingActivity#<id>:completed*:* slice) or pass --*-ts + --*-tid.

When the start and end points lie on different threads, critical-path discovers a contiguous thread spine across [start_ts, end_ts] by walking deterministic scheduler wakeups (COALESCE(waker_id, prev_id)) first, falling back to bounded flow + wakeup graph reachability when handoffs cross already-running event-loop threads.

The plugin outputs 3 sections:

1. Critical Path Process & Thread Summary: Ranks processes and threads by critical_path_ms (pct_of_total, running_ms, runnable_ms, waking_ms, unattributed_sleep_ms, unknown_ms, spans).
2. Critical Path Top Blocking Slices: Ranks the innermost active slices (self time, or <uninstrumented> when executing without trace slices) along the critical path.
3. Critical Path Chronological Span Timeline: Lists chronological spans along the critical path (rel_start_ms, ts, dur_ms, process_name, thread_name, tid, state, cpu, slice_name, handoff_from), coalescing adjacent spans with identical thread/state/slice by default.

Plugin-Specific Arguments

• --start-slice-id <int> / --end-slice-id <int>: Slice ID on a thread_track.
• --start-slice <glob> / --end-slice <glob>: Glob pattern matching a thread_track slice name.
• --start-process <glob> / --end-process <glob>: Optional process name glob filter for slice or tid resolution.
• --start-thread <glob> / --end-thread <glob>: Optional thread name glob filter for slice or tid resolution.
• --start-index <int> / --end-index <int>: 0-based (or negative) match index when multiple slices match --*-slice.
• --start-edge <start|end> / --end-edge <start|end>: Slice boundary to anchor on (start default for start point; end default for end point).
• --start-ts <int> / --end-ts <int>: Explicit timestamp in nanoseconds (requires --*-tid or --*-utid).
• --start-tid <int> / --end-tid <int>: OS thread ID (tid) for --*-ts.
• --start-utid <int> / --end-utid <int>: Perfetto unique thread ID (utid) for --*-ts.
• --limit <int>: Maximum summary/slice rows to return (default: 20).
• --timeline-limit <int>: Maximum timeline spans to return (default: max(limit * 5, 100)).
• --no-coalesce: Disable coalescing of adjacent timeline spans with identical thread, state, and slice.

Examples

Analyze the cross-thread critical path between Start proc and launch completion:

fx perf-analyze --format markdown analyze \
  --trace out/e2e-testout/trace.fxt \
  --plugin critical-path \
  --start-slice 'Start proc:*calculator*' \
  --end-slice 'launchingActivity#*:completed*:*calculator*' \
  --limit 15

Analyze the critical path between two explicit timestamps and thread IDs:

fx perf-analyze --format markdown analyze \
  --trace out/e2e-testout/trace.fxt \
  --plugin critical-path \
  --start-ts 54120000000 --start-tid 1842 \
  --end-ts 54680000000 --end-tid 2109

binder (Starnix Binder Analysis)

Analyzes Starnix binder delays, missed wakeups (scheduling delays), late-spawned wakers (SpawnLooper instant events), thread pool exhaustion (no_available_threads instant events), and process queue depth backlog (process_queue_depth counters).

[!TIP] Trace Category Recommendation: Including the starnix:binder trace category significantly improves the fidelity and depth of binder analysis. With starnix:binder, the plugin reconstructs full round-trip IPC transaction flows, driver queue wait times, thread pool exhaustion signals, and backlog counters. If starnix:binder is not found in the trace, the plugin issues an explicit notification and falls back to best-effort heuristics using binder_ioctl slices and binder:% thread naming patterns.

The plugin executes 5 diagnostic queries:

1. Missed Wakeups (Wakeup Latencies): Identifies scheduling delays (Runnable state duration) for binder threads exceeding the threshold.
2. Binder Delays (Transaction Queue Latencies): Identifies completed and incomplete transaction queue latencies exceeding the threshold.
3. Spawn Looper Events (Late-Spawned Wakers): Identifies SpawnLooper thread-scoped instant events (dur = 0) and their enclosing syscall/ioctl parent slice context when the binder thread pool exhausts available threads.
4. No Available Threads Events (Thread Pool Exhaustion): Identifies no_available_threads process-scoped instant events recorded when an incoming command cannot find an idle thread and queues to the process queue.
5. Binder Process Queue Depth (Counter Summary): Summarizes process_queue_depth counter track peaks, averages, and sample counts per process.

Fallback Behavior Without starnix:binder

When the trace does not contain slices under the starnix:binder category:

• An initial Trace Category Status section is reported recommending the inclusion of starnix:binder.
• Missed Wakeups and Binder Delays fall back to matching binder_ioctl slices and binder:% / binder_% thread names.
• Spawn Looper Events, No Available Threads, and Process Queue Depth return empty result sets with a note indicating the category was not found and that enabling it will strengthen the analysis.

Plugin-Specific Arguments

• --threshold-ms <float>: Threshold for scheduling delay and queue latency in milliseconds (default: 10.0). Does not filter instant events or queue depth counters.
• --complete-only: Only return complete transactions (default: False, includes incomplete transactions).

Example

fx perf-analyze --format markdown analyze \
  --trace "https://ui.perfetto.dev/#!/?s=b3615b084da54e9a0742dd8f6280de355df4f513" \
  --plugin binder \
  --threshold-ms 15.0

cpu (CPU Utilization & Idle Power Diagnostics)

Provides a multi-dimensional diagnostic view of system activity during an idle trace, identifying timer thrashing, excessive wakeups, per-core utilization and frequency scaling, async executor overhead, binder IPC chatter, and suspend/wake lease blockers.

The plugin executes 6 diagnostic queries in prioritized order:

1. Restless Sleepers (Wakeup Counts): Identifies threads with high context switch / wakeup counts preventing deep sleep (VddMin).
2. Per-Core Utilization & Processing Rate: Analyzes core duty cycles (% active non-idle time) and CPU frequency scaling via Fuchsia kernel Rate:CPU:N counters.
3. Top CPU Consumers (Usual Suspects): Ranks threads by total accumulated CPU runtime.
4. Fuchsia Async Executor Overhead: Measures runtime and slice counts for async executors managing futures.
5. Binder IPC Breakdown: Quantifies Starnix/Android IPC transaction traffic and latencies broken down by thread and process.
6. Suspend and Wake Lease Tracking: Verifies SAG suspend attempts and attributes active wake leases to components.

Plugin-Specific Arguments

• --limit <int>: Maximum number of rows returned for ranked queries (default: 15).

Example

fx perf-analyze --format markdown analyze \
  --trace src/performance/perf-analyze/test-data/sample_fxt.fxt \
  --plugin cpu \
  --limit 10

Running Tests

Unit tests are written using standard Python unittest. To build and run all unit tests, execute:

fx test perf_analyze_test app_launch_test critical_path_test binder_test result_formatter_test cpu_test plugins_test