bigtrace: add local DuckDB-streaming Python server and multi-query UI Add a local Python server (python/perfetto/bigtrace/server, tools/bigtrace_server.py) and Bigtrace UI enhancements (ui/src/bigtrace) for querying local trace corpora at scale: - Parallel TraceProcessorPool and C++ streaming ingestion into DuckDB with a shared catalog-wide _trace_uuid ENUM type and on-demand metadata joins. - Single-pass multi-query execution in the Bigtrace UI and server, live DuckDB query progress bar and timer, and DNS rebinding Host validation. - Comprehensive Python server unit tests, DuckDB operator E2E workflow tests (intervals, trees, and Java heap dump dominator/BFS trees), Playwright E2E tests, and end-to-end Cookbook documentation.
diff --git a/ai/skills/README.md b/ai/skills/README.md index 7a89392..9ec5ea8 100644 --- a/ai/skills/README.md +++ b/ai/skills/README.md
@@ -117,3 +117,9 @@ install from that branch; see [`docs/getting-started/using-ai.md`](../../docs/getting-started/using-ai.md). To try the local tree, `ai/evals/setup_assets.py` bundles it the same way. + +## Repo-local skills + +[`bigtrace/`](bigtrace/SKILL.md) brings up the Bigtrace UI over a directory +or archive of traces. It drives `tools/bigtrace_server.py` and the UI dev +server, so it needs a Perfetto checkout and is not bundled into the release.
diff --git a/ai/skills/bigtrace/SKILL.md b/ai/skills/bigtrace/SKILL.md new file mode 100644 index 0000000..06ea9f9 --- /dev/null +++ b/ai/skills/bigtrace/SKILL.md
@@ -0,0 +1,117 @@ +--- +name: bigtrace +description: >- + Open many Perfetto traces at once in the interactive Bigtrace UI, to run + PerfettoSQL across all of them and filter, sort and compare traces by their + metadata. Use when the user has a directory or a zip/tar archive of traces + and wants to explore or query them together, not one at a time. Needs a + Perfetto source checkout. +--- + +# Bigtrace + +`tools/bigtrace_server.py` serves a directory or archive of traces to the +Bigtrace UI, which runs a PerfettoSQL query on every trace and shows the +results as one table. Run everything from the root of a Perfetto checkout. + +## Start + +1. Start the server in the background and keep its output. Pass the + directory or archive (zip, tar, tar.gz, ...) the user gave you: + + ```sh + tools/bigtrace_server.py /path/to/traces --generate-manifest + ``` + + Archives are unpacked once to `~/.cache/perfetto/bigtrace/`. + `--generate-manifest` loads each trace once to record metadata (duration, + device, build, process and slice counts) for filtering and sorting; later + runs only process new or changed traces. The server is ready when it + prints `Bigtrace server on http://127.0.0.1:5001`. If port 5001 is taken, + add `-p <port>`. + +2. If nothing is listening on port 10000, start the UI in the background: + + ```sh + ui/run-dev-server --bigtrace + ``` + + The first run builds the UI and takes a few minutes; it is ready when + `curl -sf http://localhost:10000/bigtrace.html` succeeds. + +3. Give the user the URL the server printed: + `http://localhost:10000/bigtrace.html?server=http://127.0.0.1:5001`. + +## Multi-Query Extraction & DuckDB Analysis + +Every trace query attaches `_trace_uuid` to every row and streams its result in +C++ (`pf_scan_tp_stream`) into a DuckDB table (`result_1`, `result_2`, ... or a +custom table name). + +In **PerfettoSQL** mode, click **Add Query** to extract multiple tables (e.g., +`slices`, `sched`, `thread_state`, or `heap_objects`, `heap_refs`, `gc_roots`) +in a **single pass** over the trace corpus; double-click any sub-query tab +before running to rename its DuckDB table inline. Then switch the query engine +to **DuckDB SQL** to run cross-table analysis and joins with `trace_metadata` +using either **PerfettoSQL Next (`|>`) pipeline syntax** (`FROM ... |> ...`, +`INTERVAL INTERSECTION OF ... |> ...`, `TREE ACCUMULATE ...`, `GRAPH DOMINATOR TREE ...`) +or the `pf_interval_*`, `pf_tree_*`, `pf_graph_*` table macros. Always pass +`_trace_uuid` first in `PER (_trace_uuid, ...)` / `per := ['_trace_uuid', ...]`: + +- **PerfettoSQL Next (`|>`) Pipeline Syntax (and Single-Trace Equivalence)**: + ```sql + INTERVAL INTERSECTION OF ( + (FROM slices |> WHERE dur >= 0) AS s, + (FROM thread_state |> WHERE dur >= 0) AS st, + (FROM sched |> WHERE dur >= 0) AS c + ) PER (_trace_uuid, utid) + |> AGGREGATE SUM(dur) / 1e6 AS dur_ms + GROUP BY s.name, st.state, c.cpu + |> ORDER BY dur_ms DESC + |> LIMIT 25; + ``` + (To run the exact same `|>` query on a single trace in PerfettoSQL, simply + drop `_trace_uuid` from `PER (_trace_uuid, ...)`.) +- **Intervals (`pf_interval_*`)**: + - 2-way / 3-way intersection (`slices` × `sched` × `thread_state`): + ```sql + SELECT s.name, st.state, c.cpu, sum(dur) / 1e6 AS dur_ms + FROM pf_interval_intersection( + ['(SELECT * FROM slices WHERE dur >= 0)', '(SELECT * FROM thread_state WHERE dur >= 0)', '(SELECT * FROM sched WHERE dur >= 0)'], + ['s', 'st', 'c'], + per := ['_trace_uuid', 'utid'], + full_row := true + ) + GROUP BY ALL ORDER BY dur_ms DESC LIMIT 25; + ``` + - Flattening per track or process: + `pf_interval_flatten_sum('(SELECT * FROM slices WHERE dur >= 0)', ['depth'], per := ['_trace_uuid', 'track_id'])` + and `pf_interval_flatten(..., 'count(*) AS concurrency, arg_max(name, depth) AS deepest', per := ['_trace_uuid', 'track_id'])`. +- **Trees (`pf_tree_*`)**: + - Subtree rollup: `pf_tree_accumulate_up('slices', 'dur', per := ['_trace_uuid'])` + - Ancestor / descendant filtering: + `pf_tree_keep_if('slices', 'descendant', '(SELECT _trace_uuid, id FROM slices WHERE name GLOB ''*doFrame*'')', per := ['_trace_uuid'])` + - Callstack flamegraphs: `pf_tree_flatten_recursion('slices', ['name'], per := ['_trace_uuid'])` followed by `pf_tree_merge_siblings(..., ['name'], ['sum(dur) AS total_dur', 'count(*) AS call_count'], per := ['_trace_uuid'])`. +- **Graphs & Heap Dumps (`pf_graph_*` + `pf_tree_*`)**: + - Single-pass 3-query extraction (`heap_objects`, `heap_refs` excluding `android.memory.heap_graph.excluded_refs`, and `gc_roots`), then: + - Dominator tree + retained size: + `pf_graph_dominator_tree('heap_objects', 'heap_refs', 'gc_roots', per := ['_trace_uuid', 'upid', 'graph_sample_ts'])` → `pf_tree_accumulate_up(..., 'self_size', per := ['_trace_uuid', 'upid', 'graph_sample_ts'])`. + - Shortest GC root paths: + `pf_graph_bfs_tree(...)` → `pf_tree_flatten_recursion(..., ['class_name'], ...)` → `pf_tree_key_siblings(..., ['class_name', 'heap_type'], ...)`. +- **Fleet & Experiment Aggregation**: + - `JOIN trace_metadata USING (_trace_uuid)` and compute `quantile_cont(metric, [0.5, 0.9, 0.99])` grouped by build, device, or `is_treatment`. + +See `examples/bigtrace/README.md` for the complete Cookbook of copy-pasteable +Step 1 + Step 2 recipes. + +Parquet export is off by default; pass `--parquet-dir DIR` to export and reload +query tables as Parquet files, or `--duckdb-path PATH` to persist DuckDB tables +and history across restarts. + +## Stop + +Stop the server with Ctrl-C or SIGTERM, not SIGKILL: it shuts down its +`trace_processor_shell` children on exit, and SIGKILL leaves them running. + +All options: `tools/bigtrace_server.py --help` and +`examples/bigtrace/README.md`.
diff --git a/examples/bigtrace/README.md b/examples/bigtrace/README.md new file mode 100644 index 0000000..4dc7cfe --- /dev/null +++ b/examples/bigtrace/README.md
@@ -0,0 +1,643 @@ +# Bigtrace local server + +`tools/bigtrace_server.py` serves the Bigtrace UI API over traces on local +disk, so the Bigtrace UI can query many traces without a cloud backend. It needs +only Python 3 and the `perfetto` package in this repo. + +The manifests in this directory are format examples and test data for +`python/test/bigtrace_server_*_unittest.py`. + +## Quick start + +```bash +tools/bigtrace_server.py /path/to/traces --generate-manifest +ui/run-dev-server --bigtrace # in another shell +``` + +Open `http://localhost:10000/bigtrace.html?server=http://127.0.0.1:5001`. +`?server=` sets the backend endpoint, same as editing it in the connection +panel. + +`/path/to/traces` is a directory or an archive. Zip and tar files (plain, gz, +bz2 or xz) are recognised by content, whatever their name, and unpacked once +to `~/.cache/perfetto/bigtrace/`; the unpacked copy, with its manifest and +query history, is reused until the archive changes. + +Agents can do all of this with the [`bigtrace` skill](../../ai/skills/bigtrace/SKILL.md). + +## Options + +| Flag | Default | Description | +| --- | --- | --- | +| `TRACES` | `.` | Directory or archive of traces. | +| `-r`, `--recursive` | off (on for archives) | Scan subdirectories. | +| `-m`, `--manifest PATH` | `bigtrace_manifest.json` or `metadata.db` in the trace directory, if present | Manifest to load; CSV if it ends in `.csv`, a trace_fetch SQLite index if it ends in `.db`. | +| `-p`, `--port` | `5001` | Port to listen on. | +| `--host` | `127.0.0.1` | Address to bind. | +| `-w`, `--workers` | CPU count, at most one per 2 GB of RAM | Traces queried in parallel. | +| `--shell-path PATH` | auto | `trace_processor_shell` to use. | +| `--parquet-dir DIR` | off | Optional directory to export each query's DuckDB table as `<name>.parquet` and reload existing `.parquet` files on startup. | +| `--duckdb-path PATH` | `:memory:` | Optional file path for persistent DuckDB storage (tables and query history). | +| `--origin URL` | `https://ui.perfetto.dev` | Perfetto UI that trace links open in. | +| `--cors-origin ORIGIN` | | Extra origin allowed to call the server; repeatable. | +| `--generate-manifest [OUT]` | `bigtrace_manifest.json` in the trace directory | Update the manifest before serving. | +| `-f`, `--force` | off | With `--generate-manifest`: re-extract every trace. | +| `--metadata-query SQL` | | With `--generate-manifest`: extra columns per trace. | + +## Manifests + +Without a manifest the trace list only has file columns (name, size, link). +`--generate-manifest` loads every trace in parallel before serving and records +its bounds, duration, process, thread and slice counts, and the build/OS rows +of the `metadata` table, so the list can be filtered and sorted by them. Later +runs pick up `bigtrace_manifest.json` from the trace directory without the +flag. + +With the flag, only new or changed traces (by size and mtime) are loaded and +deleted ones are dropped, so passing it every time is cheap; `--force` +re-extracts everything. A running server re-reads its source on +`POST /refresh`. + +`--metadata-query` adds the first row of a query to each trace: + +```bash +tools/bigtrace_server.py /path/to/traces --generate-manifest \ + --metadata-query "SELECT count(1) AS jank_slices FROM slice WHERE name GLOB '*Choreographer*'" +tools/bigtrace_server.py /path/to/traces --generate-manifest \ + --metadata-query "$(cat metadata.sql)" +``` + +## Query execution, multi-query pass, and DuckDB tables + +Each trace query runs in parallel across all selected traces (`--workers` at a +time). Python acts only as the control plane: each worker hands its +`trace_processor_shell` HTTP endpoint to the C++ DuckDB extension +(`pf_scan_tp_stream` in `python/perfetto/bigtrace/duckdb_ext/src/proto_ingest.cc`), +which issues `POST /query`, decodes the chunked `QueryResult` protobuf batches +directly into columnar C++ chunks, and streams them into a DuckDB table +(`result_1`, `result_2`, ... or a custom name set via the `Table:` field in the +UI toolbar). Every result table shares a catalog-wide `_trace_uuid` `ENUM` +column (`uint16`, 2 bytes/row), and per-trace metadata columns from +`trace_metadata` are joined on demand when paging in the UI. + +In **PerfettoSQL** mode, you can click **Add Query** to define multiple +PerfettoSQL queries (for example, `#1 slices` and `#2 sched`) that execute in a +**single pass** over the trace corpus—loading each trace once and populating a +separate DuckDB table for each query. **Double-click any sub-query tab** in the +sub-query bar (before it has been run) to edit its DuckDB table name inline +(just like renaming a top-level query tab); if you choose a table name that +already exists in DuckDB or collides with another slot, clicking **Run** prompts +with the standard **Overwrite Table** modal (**Cancel**, **Save as new**, or +**Overwrite**). + +After running, switch the query engine to **DuckDB SQL** to run cross-table +analysis over your extracted tables using either **PerfettoSQL Next (`|>`) +pipeline syntax** (`docs/spec.md` §§2–7) or the `pf_interval_*`, `pf_tree_*`, +and `pf_graph_*` table macros. Always lead `PER (_trace_uuid, ...)` / `per := +['_trace_uuid', ...]` with `_trace_uuid` so partitions are solved independently +per trace in parallel: + +```sql +-- PerfettoSQL Next |> pipeline syntax (compiles automatically in DuckDB SQL mode): +INTERVAL INTERSECTION OF ( + (FROM slices |> WHERE dur >= 0) AS s, + (FROM sched |> WHERE dur >= 0) AS c +) PER (_trace_uuid, utid) +|> AGGREGATE SUM(dur) AS cpu_dur_ns, COUNT(*) AS segments + GROUP BY _trace_uuid, s.name +|> ORDER BY cpu_dur_ns DESC; + +-- Or using the pf_interval_intersection table macro directly: +SELECT * +FROM pf_interval_intersection( + ['(SELECT * FROM "slices" WHERE dur >= 0)', '(SELECT * FROM "sched" WHERE dur >= 0)'], + ['s', 'c'], + per := ['_trace_uuid', 'utid'], + full_row := true +); +``` + +### Copy-Pasting Between Multi-Trace DuckDB and Single-Trace PerfettoSQL + +Because **DuckDB SQL** mode supports the full PerfettoSQL Next (`|>`) pipeline +syntax (`FROM ... |> WHERE ... |> EXTEND ... |> SET ... |> DROP ... |> RENAME ... |> AGGREGATE ... |> FORK AS ...`, +duration literals like `16ms`/`500us`/`1.5s`, and all `INTERVAL`, `TREE`, and +`GRAPH` operators), you can copy a `|>` pipeline directly from Bigtrace's +**DuckDB SQL** editor into a single-trace **Perfetto UI** tab (or vice versa): + +- **Multi-trace -> Single-trace**: Remove `_trace_uuid` from `PER (_trace_uuid, ...)` + (or omit `PER (_trace_uuid)` when `_trace_uuid` was the only partition key) and + remove `_trace_uuid` from `GROUP BY` / `SELECT`. +- **Single-trace -> Multi-trace**: Prepend `_trace_uuid` to every `PER (...)` + clause (e.g. `PER (utid)` -> `PER (_trace_uuid, utid)`, or add + `PER (_trace_uuid)` to `TREE` / `GRAPH` stages). + +Or join across any number of query tables and `trace_metadata`: + +```sql +SELECT + r1._trace_uuid, + m.device_name, + r1.startup_dur_ns, + r2.binder_count +FROM result_1 r1 +JOIN result_2 r2 USING (_trace_uuid) +JOIN trace_metadata m USING (_trace_uuid) +ORDER BY r1.startup_dur_ns DESC; +``` + +Passing `--duckdb-path PATH` persists both DuckDB tables and query execution +history to a DuckDB file across restarts. Passing `--parquet-dir DIR` exports +each query's DuckDB table to `<DIR>/<table_name>.parquet` and reloads existing +`.parquet` files into DuckDB on startup. + +## Cookbook: End-to-End Multi-Trace Analysis Recipes + +Every recipe below follows a two-step workflow: +1. **Step 1 — Single-Pass PerfettoSQL Extraction**: In **PerfettoSQL** mode, use + **Add Query** to extract raw per-trace tables in a single pass over the trace + corpus. Every output table automatically receives a `_trace_uuid` column. +2. **Step 2 — DuckDB Multi-Trace Analysis**: Switch to **DuckDB SQL** mode and + run `pf_interval_*`, `pf_tree_*`, and `pf_graph_*` operators across all + traces in parallel. Always pass `_trace_uuid` first in `per := ['_trace_uuid', ...]`. + +--- + +### 1. Thread Slices × CPU Scheduling (`sched`) & Thread State (`thread_state`) + +#### Step 1: Single-Pass PerfettoSQL Extraction (3 queries) + +- **Query `#1` (Table: `slices`)**: + ```sql + SELECT + s.id, + s.parent_id, + s.ts, + s.dur, + s.depth, + s.name, + s.track_id, + tt.utid, + t.upid + FROM slice s + JOIN thread_track tt ON s.track_id = tt.id + JOIN thread t USING (utid) + WHERE s.dur >= 0; + ``` +- **Query `#2` (Table: `sched`)**: + ```sql + SELECT id, ts, dur, cpu, utid + FROM sched + WHERE utid != 0 AND dur >= 0; + ``` +- **Query `#3` (Table: `thread_state`)**: + ```sql + SELECT id, ts, dur, utid, state, io_wait + FROM thread_state + WHERE dur >= 0; + ``` + +#### Step 2a: 2-Way Intersection (`slices` × `sched`) — On-CPU vs. Wall Duration + +```sql +WITH on_cpu AS ( + SELECT + _trace_uuid, + s.id AS slice_id, + s.name AS slice_name, + s.dur AS wall_dur_ns, + sum(dur) AS cpu_dur_ns + FROM pf_interval_intersection( + ['(SELECT * FROM slices WHERE dur >= 0)', '(SELECT * FROM sched WHERE dur >= 0)'], + ['s', 'c'], + per := ['_trace_uuid', 'utid'], + full_row := true + ) + GROUP BY _trace_uuid, s.id, s.name, s.dur +) +SELECT + slice_name, + count(*) AS slice_count, + sum(wall_dur_ns) / 1e6 AS total_wall_ms, + sum(cpu_dur_ns) / 1e6 AS total_cpu_ms, + round(100.0 * sum(cpu_dur_ns) / nullif(sum(wall_dur_ns), 0), 2) AS on_cpu_pct +FROM on_cpu +GROUP BY slice_name +ORDER BY total_cpu_ms DESC +LIMIT 25; +``` + +#### Step 2b: 3-Way Intersection (`slices` × `thread_state` × `sched`) — Slice Breakdown by State & CPU + +```sql +SELECT + s.name AS slice_name, + st.state AS thread_state, + c.cpu AS cpu, + count(*) AS intersections, + sum(dur) / 1e6 AS total_dur_ms +FROM pf_interval_intersection( + [ + '(SELECT * FROM slices WHERE dur >= 0)', + '(SELECT * FROM thread_state WHERE dur >= 0)', + '(SELECT * FROM sched WHERE dur >= 0)' + ], + ['s', 'st', 'c'], + per := ['_trace_uuid', 'utid'], + full_row := true +) +GROUP BY s.name, st.state, c.cpu +ORDER BY total_dur_ms DESC +LIMIT 50; +``` + +#### Step 2c: Slice Self-Intersection & Interval Flattening (`pf_interval_flatten` / `pf_interval_flatten_sum`) + +```sql +-- Self-intersection of slices per track (finds all overlapping/nested slice pairs): +SELECT + _trace_uuid, + track_id, + a.name AS outer_Or_first_slice, + b.name AS inner_or_second_slice, + sum(dur) / 1e6 AS overlap_ms +FROM pf_interval_intersection( + ['(SELECT * FROM slices WHERE dur >= 0)', '(SELECT * FROM slices WHERE dur >= 0)'], + ['a', 'b'], + per := ['_trace_uuid', 'track_id'], + full_row := true +) +WHERE a.id != b.id +GROUP BY ALL +ORDER BY overlap_ms DESC +LIMIT 25; + +-- Fast-path interval flattening per track (concurrency and sum of depth): +SELECT + _trace_uuid, + track_id, + count(*) AS segment_count, + max(count) AS max_concurrency, + sum(dur) / 1e6 AS active_track_ms +FROM pf_interval_flatten_sum( + '(SELECT * FROM slices WHERE dur >= 0)', + ['depth'], + per := ['_trace_uuid', 'track_id'] +) +GROUP BY _trace_uuid, track_id +ORDER BY active_track_ms DESC; + +-- General interval flattening per process (upid) with custom DuckDB aggregates: +SELECT + _trace_uuid, + upid, + ts, + dur, + concurrency, + deepest_slice +FROM pf_interval_flatten( + '(SELECT * FROM slices WHERE dur >= 0 AND upid IS NOT NULL)', + 'count(*) AS concurrency, arg_max(name, depth) AS deepest_slice', + per := ['_trace_uuid', 'upid'] +) +ORDER BY dur DESC +LIMIT 50; +``` + +--- + +### 2. Slice Tree Hierarchies (`pf_tree_*`) + +#### Step 1: PerfettoSQL Extraction (Table: `slices`) + +```sql +SELECT + s.id, + s.parent_id, + s.ts, + s.dur, + s.depth, + s.name, + s.track_id, + tt.utid +FROM slice s +JOIN thread_track tt ON s.track_id = tt.id +WHERE s.dur >= 0; +``` + +#### Step 2a: Inclusive Subtree Time (`pf_tree_accumulate_up`) & Exclusive Self-Time (`dur - children_dur`) + +```sql +-- Inclusive subtree sum (note: for strict nested slice intervals, self + descendants +-- sums each node's dur up the tree; for exclusive self-time, subtract direct children's dur): +WITH child_sums AS ( + SELECT + _trace_uuid, + parent_id AS id, + sum(dur) AS children_dur + FROM slices + WHERE parent_id IS NOT NULL + GROUP BY _trace_uuid, parent_id +), +acc AS ( + SELECT _trace_uuid, id, total AS subtree_dur_sum + FROM pf_tree_accumulate_up('slices', 'dur', per := ['_trace_uuid']) +) +SELECT + s.name, + count(*) AS invocations, + sum(s.dur) / 1e6 AS inclusive_wall_ms, + sum(s.dur - coalesce(c.children_dur, 0)) / 1e6 AS exclusive_self_ms, + sum(a.subtree_dur_sum) / 1e6 AS accumulated_subtree_ms +FROM slices s +JOIN acc a USING (_trace_uuid, id) +LEFT JOIN child_sums c USING (_trace_uuid, id) +GROUP BY s.name +ORDER BY exclusive_self_ms DESC +LIMIT 25; +``` + +#### Step 2b: Filtering Descendants & Ancestors (`pf_tree_keep_if`) + +```sql +-- All slices strictly descended from 'Choreographer#doFrame': +SELECT + name, + count(*) AS count_under_do_frame, + sum(dur) / 1e6 AS total_dur_ms +FROM pf_tree_keep_if( + 'slices', + 'descendant', + '(SELECT _trace_uuid, id FROM slices WHERE name GLOB ''*Choreographer#doFrame*'')', + per := ['_trace_uuid'] +) +GROUP BY name +ORDER BY total_dur_ms DESC +LIMIT 25; + +-- All ancestor slices enclosing a 'binder transaction': +SELECT + name, + count(*) AS count_enclosing_binder, + sum(dur) / 1e6 AS total_dur_ms +FROM pf_tree_keep_if( + 'slices', + 'ancestor', + '(SELECT _trace_uuid, id FROM slices WHERE name = ''binder transaction'')', + per := ['_trace_uuid'] +) +GROUP BY name +ORDER BY count_enclosing_binder DESC +LIMIT 25; +``` + +#### Step 2c: Collapsing Recursive Frames & Building Aggregated Callstack Flamegraphs + +```sql +-- 1. Collapse direct recursion (A -> A -> A becomes siblings under A's parent), +-- 2. Unify identical root-to-node callstack paths per trace with pf_tree_merge_siblings: +CREATE OR REPLACE TEMP VIEW slices_norec AS +SELECT * +FROM pf_tree_flatten_recursion('slices', ['name'], per := ['_trace_uuid']); + +SELECT + _trace_uuid, + id, + parent_id, + name, + total_dur / 1e6 AS total_dur_ms, + call_count +FROM pf_tree_merge_siblings( + 'slices_norec', + ['name'], + ['sum(dur) AS total_dur', 'count(*) AS call_count'], + per := ['_trace_uuid'] +) +ORDER BY total_dur DESC +LIMIT 50; +``` + +--- + +### 3. Java Heap Dump Analysis (`pf_graph_*` + `pf_tree_*`) + +#### Step 1: Single-Pass 3-Query PerfettoSQL Extraction + +To match Trace Processor's `android.memory.heap_graph.dominator_tree` exactly, +exclude weak, soft, finalizer, and phantom reference edges via +`android.memory.heap_graph.excluded_refs`: + +- **Query `#1` (Table: `heap_objects`)**: + ```sql + SELECT + o.id, + o.upid, + o.graph_sample_ts, + o.self_size, + o.native_size, + o.heap_type, + coalesce(c.deobfuscated_name, c.name) AS class_name + FROM heap_graph_object o + JOIN heap_graph_class c ON o.type_id = c.id + WHERE o.reachable != 0; + ``` +- **Query `#2` (Table: `heap_refs`)**: + ```sql + INCLUDE PERFETTO MODULE android.memory.heap_graph.excluded_refs; + + SELECT + o.upid, + o.graph_sample_ts, + r.owner_id AS src, + r.owned_id AS dst + FROM heap_graph_reference r + JOIN heap_graph_object o ON r.owner_id = o.id + WHERE o.reachable != 0 + AND r.owned_id IS NOT NULL + AND r.id NOT IN _excluded_refs; + ``` +- **Query `#3` (Table: `gc_roots`)**: + ```sql + SELECT + id, + upid, + graph_sample_ts + FROM heap_graph_object + WHERE root_type IS NOT NULL; + ``` + +#### Step 2a: Exact Retained Heap Size (`pf_graph_dominator_tree` → `pf_tree_accumulate_up`) + +```sql +CREATE OR REPLACE TEMP VIEW dom_tree AS +SELECT * +FROM pf_graph_dominator_tree( + 'heap_objects', + 'heap_refs', + 'gc_roots', + per := ['_trace_uuid', 'upid', 'graph_sample_ts'], + full_row := true +); + +WITH retained AS ( + SELECT + _trace_uuid, + upid, + graph_sample_ts, + id, + total AS dominated_size_bytes + FROM pf_tree_accumulate_up( + 'dom_tree', + 'self_size', + per := ['_trace_uuid', 'upid', 'graph_sample_ts'] + ) +) +SELECT + d.class_name, + count(DISTINCT d._trace_uuid) AS traces_present, + count(*) AS object_count, + sum(d.self_size) AS total_shallow_bytes, + max(r.dominated_size_bytes) AS max_object_retained_bytes, + sum(r.dominated_size_bytes) FILTER (WHERE d.parent_id IS NULL) AS root_retained_bytes +FROM dom_tree d +JOIN retained r USING (_trace_uuid, upid, graph_sample_ts, id) +GROUP BY d.class_name +ORDER BY max_object_retained_bytes DESC +LIMIT 25; +``` + +#### Step 2b: Shortest GC Root Retention Paths (`pf_graph_bfs_tree` → `pf_tree_flatten_recursion` → `pf_tree_key_siblings`) + +```sql +CREATE OR REPLACE TEMP VIEW heap_bfs AS +SELECT * +FROM pf_graph_bfs_tree( + 'heap_objects', + 'heap_refs', + 'gc_roots', + per := ['_trace_uuid', 'upid', 'graph_sample_ts'], + full_row := true +); + +CREATE OR REPLACE TEMP VIEW heap_bfs_norec AS +SELECT * +FROM pf_tree_flatten_recursion( + 'heap_bfs', + ['class_name'], + per := ['_trace_uuid', 'upid', 'graph_sample_ts'] +); + +SELECT + _trace_uuid, + node_id, + parent_node_id, + class_name, + heap_type, + count(*) AS instance_count, + sum(self_size) AS shallow_size_bytes +FROM pf_tree_key_siblings( + 'heap_bfs_norec', + ['class_name', 'heap_type'], + key_col := 'node_id', + parent_key_col := 'parent_node_id', + per := ['_trace_uuid', 'upid', 'graph_sample_ts'] +) +GROUP BY _trace_uuid, node_id, parent_node_id, class_name, heap_type +ORDER BY shallow_size_bytes DESC +LIMIT 50; +``` + +--- + +### 4. Cross-Table Fleet & Experiment Aggregation with `trace_metadata` + +Join any DuckDB operator output with `trace_metadata` (`USING (_trace_uuid)`) +to compute fleet-wide P50 / P90 / P99 percentiles across devices, builds, or +A/B experiment arms: + +```sql +WITH per_trace_cpu AS ( + SELECT + _trace_uuid, + s.name AS slice_name, + sum(dur) / 1e6 AS on_cpu_ms + FROM pf_interval_intersection( + ['(SELECT * FROM slices WHERE dur >= 0)', '(SELECT * FROM sched WHERE dur >= 0)'], + ['s', 'c'], + per := ['_trace_uuid', 'utid'], + full_row := true + ) + WHERE s.name GLOB '*Choreographer#doFrame*' + GROUP BY _trace_uuid, s.name +) +SELECT + coalesce(m.android_build_fingerprint, 'unknown') AS build, + m.is_treatment, + count(DISTINCT p._trace_uuid) AS trace_count, + round(quantile_cont(p.on_cpu_ms, 0.50), 2) AS p50_on_cpu_ms, + round(quantile_cont(p.on_cpu_ms, 0.90), 2) AS p90_on_cpu_ms, + round(quantile_cont(p.on_cpu_ms, 0.99), 2) AS p99_on_cpu_ms +FROM per_trace_cpu p +JOIN trace_metadata m USING (_trace_uuid) +GROUP BY ALL +ORDER BY build, m.is_treatment; +``` + +## Opening traces + +Clicking a trace's `link` opens it in the Perfetto UI, which fetches it from +this server: + +``` +https://ui.perfetto.dev/#!/?url=http://127.0.0.1:5001/trace/<uuid>/<name>&referrer=bigtrace_server +``` + +Like `tools/open_trace_in_ui`, `--origin` selects another UI, e.g. +`--origin http://localhost:10000`. + +## Access + +Browsers may only call the server from localhost origins, `https://ui.perfetto.dev` +and `--origin`; add others with `--cors-origin`. Anyone who can reach the port +can read every trace it serves, so keep the default `--host 127.0.0.1` unless you +need otherwise. + +## Manifest formats + +JSON (see `bigtrace_manifest.json`): + +```json +{ + "trace_dir": ".", + "schema": [ + {"name": "duration_ms", "displayName": "Duration (ms)", "type": "FLOAT64", + "defaultVisible": true, "description": "Trace duration in milliseconds"} + ], + "traces": [ + {"file_path": "startup_cold.perfetto-trace", "duration_ms": 2997.996, + "experiment_id": 101, "control_id": 102, "is_treatment": true} + ], + "presets": [ + {"id": "builtin:slice_count", "category": "General", + "name": "Slice count per trace", + "perfettoSql": "SELECT count(1) AS slice_count FROM slice;"} + ], + "experiments": [ + {"experimentId": 101, "experimentName": "PGO", "controlId": 102, + "controlName": "Baseline"} + ] +} +``` + +Only `traces[].file_path` is required. Paths are relative to `trace_dir`, which +is relative to the manifest. Other trace keys become metadata columns. `schema` +sets display names, types and default visibility; unlisted columns are +inferred. `experiment_id`, `control_id` and `is_treatment` drive the UI's +experiment filter. + +CSV (see `bigtrace_manifest.csv`) has one row per trace with the same keys: + +```csv +file_path,device_name,experiment_id,control_id,is_treatment +startup_cold.perfetto-trace,Pixel 8,101,102,true +```
diff --git a/examples/bigtrace/bigtrace_manifest.csv b/examples/bigtrace/bigtrace_manifest.csv new file mode 100644 index 0000000..6cb8837 --- /dev/null +++ b/examples/bigtrace/bigtrace_manifest.csv
@@ -0,0 +1,8 @@ +file_path,device_name,os_version,build_id,test_name,experiment_id,control_id,is_treatment,tag +api24_startup_cold.perfetto-trace,Pixel 6,Android 7.0,NRD90M.220624.014,app_startup_cold,101,102,true,cold +api24_startup_hot.perfetto-trace,Pixel 6,Android 7.0,NRD90M.220624.014,app_startup_hot,101,102,false,hot +api24_startup_warm.perfetto-trace,Pixel 6,Android 7.0,NRD90M.220624.014,app_startup_warm,101,102,true,warm +api31_startup_cold.perfetto-trace,Pixel 7,Android 12,SP1A.210812.016,app_startup_cold,201,202,true,cold +api31_startup_hot.perfetto-trace,Pixel 7,Android 12,SP1A.210812.016,app_startup_hot,201,202,false,hot +api31_startup_warm.perfetto-trace,Pixel 7,Android 12,SP1A.210812.016,app_startup_warm,201,202,true,warm +api34_startup_cold.perfetto-trace,Pixel 8,Android 14,UD1A.230803.022,app_startup_cold,301,302,true,cold
diff --git a/examples/bigtrace/bigtrace_manifest.json b/examples/bigtrace/bigtrace_manifest.json new file mode 100644 index 0000000..2a92322 --- /dev/null +++ b/examples/bigtrace/bigtrace_manifest.json
@@ -0,0 +1,148 @@ +{ + "trace_dir": ".", + "experiments": [ + { + "experiment_id": 101, + "experiment_name": "Profile-Guided Optimization (PGO)", + "control_id": 102, + "control_name": "Default Compiler Optimization", + "is_experiment_denied": false, + "is_control_denied": false + }, + { + "experiment_id": 201, + "experiment_name": "ART JIT Zygote Preload Enabled", + "control_id": 202, + "control_name": "Standard Zygote Startup", + "is_experiment_denied": false, + "is_control_denied": false + }, + { + "experiment_id": 301, + "experiment_name": "Kernel SchedTune Boost", + "control_id": 302, + "control_name": "Energy Aware Scheduling Baseline", + "is_experiment_denied": false, + "is_control_denied": false + } + ], + "presets": [ + { + "id": "preset:app_startup_latency", + "category": "App Performance", + "name": "Application Startup Latency", + "description": "Measures activity resume and application bind duration", + "perfettoSql": "SELECT name, dur / 1e6 AS dur_ms, ts FROM slice WHERE name GLOB '*activity*' OR name GLOB '*bindApplication*' ORDER BY dur DESC;", + "icon": "speed" + }, + { + "id": "preset:top_slices_by_duration", + "category": "Slices", + "name": "Top Slices by Total Duration", + "description": "Aggregates slices by name and sums their duration", + "perfettoSql": "SELECT name, count(1) AS count, sum(dur) / 1e6 AS total_dur_ms, avg(dur) / 1e6 AS avg_dur_ms FROM slice GROUP BY name ORDER BY total_dur_ms DESC LIMIT 50;", + "icon": "analytics" + }, + { + "id": "preset:binder_transactions", + "category": "IPC", + "name": "Binder Transactions (>2ms)", + "description": "Binder transactions taking longer than 2ms", + "perfettoSql": "SELECT name, dur / 1e6 AS dur_ms FROM slice WHERE name GLOB '*binder*' AND dur > 2000000 ORDER BY dur DESC;", + "icon": "sync_alt" + } + ], + "traces": [ + { + "file_path": "api24_startup_cold.perfetto-trace", + "file_name": "api24_startup_cold.perfetto-trace", + "trace_uuid": "c01d0001-0000-0000-0000-000000000001", + "device_name": "Pixel 6", + "os_version": "Android 7.0 (API 24)", + "build_id": "NRD90M.220624.014", + "test_name": "app_startup_cold", + "experiment_id": 101, + "control_id": 102, + "is_treatment": true, + "tag": "cold" + }, + { + "file_path": "api24_startup_hot.perfetto-trace", + "file_name": "api24_startup_hot.perfetto-trace", + "trace_uuid": "c01d0001-0000-0000-0000-000000000002", + "device_name": "Pixel 6", + "os_version": "Android 7.0 (API 24)", + "build_id": "NRD90M.220624.014", + "test_name": "app_startup_hot", + "experiment_id": 101, + "control_id": 102, + "is_treatment": false, + "tag": "hot" + }, + { + "file_path": "api24_startup_warm.perfetto-trace", + "file_name": "api24_startup_warm.perfetto-trace", + "trace_uuid": "c01d0001-0000-0000-0000-000000000003", + "device_name": "Pixel 6", + "os_version": "Android 7.0 (API 24)", + "build_id": "NRD90M.220624.014", + "test_name": "app_startup_warm", + "experiment_id": 101, + "control_id": 102, + "is_treatment": true, + "tag": "warm" + }, + { + "file_path": "api31_startup_cold.perfetto-trace", + "file_name": "api31_startup_cold.perfetto-trace", + "trace_uuid": "c01d0002-0000-0000-0000-000000000001", + "device_name": "Pixel 7", + "os_version": "Android 12 (API 31)", + "build_id": "SP1A.210812.016", + "test_name": "app_startup_cold", + "experiment_id": 201, + "control_id": 202, + "is_treatment": true, + "tag": "cold" + }, + { + "file_path": "api31_startup_hot.perfetto-trace", + "file_name": "api31_startup_hot.perfetto-trace", + "trace_uuid": "c01d0002-0000-0000-0000-000000000002", + "device_name": "Pixel 7", + "os_version": "Android 12 (API 31)", + "build_id": "SP1A.210812.016", + "test_name": "app_startup_hot", + "experiment_id": 201, + "control_id": 202, + "is_treatment": false, + "tag": "hot" + }, + { + "file_path": "api31_startup_warm.perfetto-trace", + "file_name": "api31_startup_warm.perfetto-trace", + "trace_uuid": "c01d0002-0000-0000-0000-000000000003", + "device_name": "Pixel 7", + "os_version": "Android 12 (API 31)", + "build_id": "SP1A.210812.016", + "test_name": "app_startup_warm", + "experiment_id": 201, + "control_id": 202, + "is_treatment": true, + "tag": "warm" + }, + { + "file_path": "api34_startup_cold.perfetto-trace", + "file_name": "api34_startup_cold.perfetto-trace", + "trace_uuid": "c01d0003-0000-0000-0000-000000000001", + "device_name": "Pixel 8", + "os_version": "Android 14 (API 34)", + "build_id": "UD1A.230803.022", + "test_name": "app_startup_cold", + "experiment_id": 301, + "control_id": 302, + "is_treatment": true, + "tag": "cold" + } + ] +}
diff --git a/python/perfetto/bigtrace/server/__init__.py b/python/perfetto/bigtrace/server/__init__.py new file mode 100644 index 0000000..a4aec4b --- /dev/null +++ b/python/perfetto/bigtrace/server/__init__.py
@@ -0,0 +1,33 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""HTTP backend for the Bigtrace UI that runs queries over local traces. + +Traces come from a directory scan, a JSON manifest or a CSV manifest +(catalog.py); zip and tar archives are unpacked to a directory first +(archive.py). A query fans out over a thread pool with one TraceProcessor per +trace (executor.py, trace_processor_pool.py) and streams results into DuckDB, +which also serves UI paging, sorting, filtering, and follow-up DuckDB queries +(duckdb_engine.py). http_handler.py maps the UI's REST API onto these; +server.py wires them up and cli.py is the tools/bigtrace_server.py entry point. +""" + +from perfetto.bigtrace.server.catalog import TraceCatalog +from perfetto.bigtrace.server.cli import main +from perfetto.bigtrace.server.duckdb_engine import DuckDBEngine +from perfetto.bigtrace.server.executor import BigtraceQueryExecutor +from perfetto.bigtrace.server.manifest import (BigtraceManifestGenerator, + generate_manifest) +from perfetto.bigtrace.server.server import BigtraceServer, BigtraceServerConfig +from perfetto.bigtrace.server.trace_processor_pool import TraceProcessorPool +from perfetto.bigtrace.server.util import resolve_trace_processor_bin
diff --git a/python/perfetto/bigtrace/server/archive.py b/python/perfetto/bigtrace/server/archive.py new file mode 100644 index 0000000..74d00a3 --- /dev/null +++ b/python/perfetto/bigtrace/server/archive.py
@@ -0,0 +1,80 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Unpacks zip and tar archives of traces so they can be served.""" + +import hashlib +import os +import shutil +import tarfile +import zipfile +from typing import IO, Iterator, Tuple + +DEFAULT_CACHE_DIR = os.path.join( + os.path.expanduser("~"), ".cache", "perfetto", "bigtrace") + + +def is_archive(path: str) -> bool: + """True for zip and tar files (any compression), judged by content.""" + if not os.path.isfile(path): + return False + return zipfile.is_zipfile(path) or tarfile.is_tarfile(path) + + +def extract_archive(path: str, cache_dir: str = DEFAULT_CACHE_DIR) -> str: + """Extracts `path` once and returns the directory holding its files. + + The directory is keyed by the archive's path, size and mtime, so restarts + reuse it (with its manifest and query history) until the archive changes. + """ + path = os.path.abspath(path) + st = os.stat(path) + key = hashlib.sha1( + f"{path}:{st.st_size}:{st.st_mtime_ns}".encode()).hexdigest()[:12] + out_dir = os.path.join( + os.path.abspath(cache_dir), f"{os.path.basename(path)}-{key}") + if os.path.isdir(out_dir): + return out_dir + + tmp_dir = f"{out_dir}.tmp" + shutil.rmtree(tmp_dir, ignore_errors=True) + os.makedirs(tmp_dir) + try: + for name, src in _members(path): + dest = os.path.normpath(os.path.join(tmp_dir, name)) + # Skip absolute paths and ".." entries that would escape tmp_dir. + if not dest.startswith(tmp_dir + os.sep): + continue + os.makedirs(os.path.dirname(dest), exist_ok=True) + with src, open(dest, "wb") as dst: + shutil.copyfileobj(src, dst) + except BaseException: + # Don't leave a partial copy behind, e.g. when the disk is full. + shutil.rmtree(tmp_dir, ignore_errors=True) + raise + os.rename(tmp_dir, out_dir) + return out_dir + + +def _members(path: str) -> Iterator[Tuple[str, IO[bytes]]]: + """Yields (name, file object) for each regular file in the archive.""" + if zipfile.is_zipfile(path): + with zipfile.ZipFile(path) as zf: + for info in zf.infolist(): + if not info.is_dir(): + yield info.filename, zf.open(info) + return + with tarfile.open(path) as tf: + for member in tf: + if member.isfile(): + yield member.name, tf.extractfile(member)
diff --git a/python/perfetto/bigtrace/server/catalog.py b/python/perfetto/bigtrace/server/catalog.py new file mode 100644 index 0000000..c3c1b42 --- /dev/null +++ b/python/perfetto/bigtrace/server/catalog.py
@@ -0,0 +1,385 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""The set of traces the server queries, loaded from a directory or manifest.""" + +import csv +import dataclasses as dc +import json +import logging +import os +import sqlite3 +import threading +from typing import Any, Dict, List, Optional, Tuple +import urllib.parse + +from perfetto.bigtrace.server.filters import ( + evaluate_filters, + parse_experiment_filter, + parse_order_by, + sort_records, +) +from perfetto.bigtrace.server.util import ( + MANIFEST_NAME, + SQLITE_MANIFEST_NAME, + as_bool, + as_int, + column_schema, + find_trace_files, + generate_trace_uuid, + get_param, + infer_column_type, + iso_time, +) + +logger = logging.getLogger(__name__) + +DEFAULT_UI_ORIGIN = "https://ui.perfetto.dev" + +# Columns every catalog has, whatever the trace source. +CATALOG_COLUMNS = [ + column_schema("link", "Link", "STRING", True, + "Opens the trace in the Perfetto UI"), + column_schema("file_name", "File Name", "STRING", True, "Trace file name"), + column_schema("file_size", "File Size", "INT64", True, + "File size in bytes"), + column_schema("trace_uuid", "Trace UUID", "STRING", True, + "Trace identifier"), + column_schema("file_path", "File Path", "STRING", False, + "Path to the trace"), + column_schema("modified_time", "Modified Time", "TIMESTAMP", False, + "File modification time"), +] + +DEFAULT_PRESETS = [ + { + "id": "builtin:slice_count", + "category": "General", + "name": "Slice count per trace", + "description": "Count total slices in each trace", + "perfettoSql": "SELECT count(1) AS slice_count FROM slice;", + "icon": "analytics", + }, + { + "id": "builtin:slow_slices", + "category": "Performance", + "name": "Slow slices (>10ms)", + "description": "Find slices lasting longer than 10 milliseconds", + "perfettoSql": ("SELECT name, dur / 1e6 AS dur_ms FROM slice " + "WHERE dur > 10000000 ORDER BY dur DESC;"), + "icon": "timer", + }, + { + "id": "builtin:threads_summary", + "category": "Processes", + "name": "Threads per process", + "description": "Summary of thread counts per process", + "perfettoSql": ("SELECT process.name AS process_name, " + "count(thread.utid) AS thread_count FROM process " + "JOIN thread USING (upid) GROUP BY process.name " + "ORDER BY thread_count DESC;"), + "icon": "memory", + }, +] + +# TraceEntry attributes; any other key is looked up in its metadata. +_ENTRY_ATTRS = { + "trace_uuid", "file_path", "file_name", "file_size", "modified_time", "link" +} + + +@dc.dataclass +class TraceEntry: + trace_uuid: str + file_path: str + file_name: str + file_size: int + modified_time: Optional[str] + link: str + metadata: Dict[str, Any] = dc.field(default_factory=dict) + + def get_value(self, key: str) -> Any: + if key == "_trace_uuid": + return self.trace_uuid + if key in _ENTRY_ATTRS: + return getattr(self, key) + return self.metadata.get(key) + + +@dc.dataclass +class SettingsOverrides: + """Trace source settings the UI sends with each request.""" + trace_dir: Optional[str] = None + recursive: Optional[bool] = None + manifest_path: Optional[str] = None + trace_uuids: Optional[List[str]] = None + + @staticmethod + def parse(settings: Any) -> "SettingsOverrides": + """Parses the UI's settings list. Empty values mean "not overridden".""" + out = SettingsOverrides() + if not isinstance(settings, list): + return out + for s in settings: + if not isinstance(s, dict): + continue + values = [str(v).strip() for v in s.get("values") or []] + values = [v for v in values if v] + if not values: + continue + setting_id = get_param(s, "setting_id") + if setting_id == "trace_directory": + out.trace_dir = values[0] + elif setting_id == "recursive": + out.recursive = as_bool(values[0]) + elif setting_id == "trace_manifest": + out.manifest_path = values[0] + elif setting_id == "trace_uuids": + out.trace_uuids = values + return out + + +class TraceCatalog: + """The set of traces queries run over, with their metadata.""" + + def __init__(self, + trace_dir: Optional[str] = None, + recursive: bool = False, + manifest_path: Optional[str] = None, + base_url: str = "", + ui_origin: str = DEFAULT_UI_ORIGIN): + self.trace_dir = os.path.abspath(trace_dir or os.getcwd()) + self.recursive = recursive + self.manifest_path = os.path.abspath( + manifest_path) if manifest_path else None + self.base_url = base_url.rstrip("/") + self.ui_origin = ui_origin.rstrip("/") + self._traces: List[TraceEntry] = [] + self._experiments: List[Dict[str, Any]] = [] + self._presets: List[Dict[str, Any]] = [] + self._manifest_schema: List[Dict[str, Any]] = [] + self._lock = threading.RLock() + self.reload() + + def is_configured_with(self, + trace_dir: Optional[str] = None, + recursive: Optional[bool] = None, + manifest_path: Optional[str] = None) -> bool: + with self._lock: + return ((not trace_dir or os.path.abspath(trace_dir) == self.trace_dir) + and (recursive is None or recursive == self.recursive) and + (not manifest_path or + os.path.abspath(manifest_path) == self.manifest_path)) + + def apply_overrides(self, overrides: SettingsOverrides) -> None: + """Reloads if the request points at a different trace source.""" + with self._lock: + if not self.is_configured_with(overrides.trace_dir, overrides.recursive, + overrides.manifest_path): + self.reload(overrides.trace_dir, overrides.recursive, + overrides.manifest_path) + + def reload(self, + trace_dir: Optional[str] = None, + recursive: Optional[bool] = None, + manifest_path: Optional[str] = None) -> None: + """Re-reads the source. Empty arguments keep the current configuration.""" + with self._lock: + if trace_dir and os.path.abspath(trace_dir) != self.trace_dir: + self.trace_dir = os.path.abspath(trace_dir) + self.manifest_path = None + if recursive is not None: + self.recursive = bool(recursive) + if manifest_path: + self.manifest_path = os.path.abspath(manifest_path) + for name in (MANIFEST_NAME, SQLITE_MANIFEST_NAME): + candidate = os.path.join(self.trace_dir, name) + if not self.manifest_path and os.path.isfile(candidate): + self.manifest_path = candidate + + self._experiments, self._presets, self._manifest_schema = [], [], [] + path = self.manifest_path + if path and os.path.isfile(path): + if path.lower().endswith(".csv"): + self._traces = self._load_csv(path) + elif path.lower().endswith(".db"): + self._traces = self._load_sqlite(path) + else: + self._traces = self._load_manifest(path) + else: + self._traces = [ + self._make_entry({"file_path": p}, self.trace_dir) + for p in find_trace_files(self.trace_dir, self.recursive) + ] + + def _make_entry(self, record: Dict[str, Any], base_dir: str) -> TraceEntry: + path = record.get("file_path") or record.get("path") + if not os.path.isabs(path): + candidate = os.path.normpath(os.path.join(base_dir, path)) + fallback = os.path.normpath(os.path.join(self.trace_dir, path)) + use_fallback = not os.path.exists(candidate) and os.path.exists(fallback) + path = fallback if use_fallback else candidate + try: + st = os.stat(path) + size, mtime = st.st_size, iso_time(st.st_mtime) + except OSError: + size, mtime = 0, record.get("modified_time") + file_name = record.get("file_name") or os.path.basename(path) + trace_uuid = record.get("trace_uuid") or generate_trace_uuid(path) + # The Perfetto UI fetches ?url= traces served from localhost. The URL has + # no query string, which the UI's service worker would reject. + trace_url = (f"{self.base_url}/trace/{trace_uuid}/" + f"{urllib.parse.quote(file_name)}") + link = record.get("link") or ( + f"{self.ui_origin}/#!/?url={trace_url}&referrer=bigtrace_server") + return TraceEntry( + trace_uuid=str(trace_uuid), + file_path=path, + file_name=file_name, + file_size=as_int(record.get("file_size"), size) or size, + modified_time=mtime, + link=link, + metadata={ + k: v + for k, v in record.items() + if k not in _ENTRY_ATTRS and k != "path" and v is not None + }, + ) + + def _load_manifest(self, manifest_file: str) -> List[TraceEntry]: + try: + with open(manifest_file, "r", encoding="utf-8") as f: + data = json.load(f) + except (OSError, ValueError) as e: + logger.error("Failed to parse manifest %s: %s", manifest_file, e) + return [] + base_dir = os.path.dirname(manifest_file) + records = data + if isinstance(data, dict): + records = data.get("traces", []) + self._experiments = data.get("experiments") or [] + self._presets = data.get("presets") or [] + self._manifest_schema = data.get("schema") or [] + base_dir = os.path.normpath( + os.path.join(base_dir, data.get("trace_dir", "."))) + if not isinstance(records, list): + return [] + return [ + self._make_entry(r, base_dir) + for r in records + if isinstance(r, dict) and (r.get("file_path") or r.get("path")) + ] + + def _load_csv(self, csv_file: str) -> List[TraceEntry]: + try: + with open(csv_file, "r", encoding="utf-8", newline="") as f: + rows = list(csv.DictReader(f)) + except (OSError, csv.Error) as e: + logger.error("Failed to load CSV %s: %s", csv_file, e) + return [] + base_dir = os.path.dirname(csv_file) + return [ + self._make_entry(r, base_dir) + for r in rows + if r.get("file_path") or r.get("path") + ] + + def _load_sqlite(self, db_file: str) -> List[TraceEntry]: + """Loads the `traces` table of a trace_fetch metadata.db.""" + try: + conn = sqlite3.connect(f"file:{db_file}?mode=ro", uri=True) + try: + conn.row_factory = sqlite3.Row + rows = [dict(r) for r in conn.execute("SELECT * FROM traces")] + finally: + conn.close() + except sqlite3.Error as e: + logger.error("Failed to load SQLite manifest %s: %s", db_file, e) + return [] + entries = [] + for r in rows: + local_path = r.pop("absolute_local_file_path", None) + r["file_path"] = r.get("file_path") or local_path + if r["file_path"]: + entries.append(self._make_entry(r, os.path.dirname(db_file))) + return entries + + def get_traces(self) -> List[TraceEntry]: + with self._lock: + return list(self._traces) + + def get_experiments(self) -> List[Dict[str, Any]]: + with self._lock: + return list(self._experiments) + + def get_presets(self) -> List[Dict[str, Any]]: + with self._lock: + return list(self._presets) + + def get_schema(self) -> List[Dict[str, Any]]: + """Returns manifest columns, then built-in ones, then discovered keys.""" + with self._lock: + traces = list(self._traces) + manifest_schema = list(self._manifest_schema) + cols: Dict[str, Dict[str, Any]] = {} + for c in manifest_schema + CATALOG_COLUMNS: + if isinstance(c, dict) and "name" in c: + cols.setdefault(c["name"], dict(c)) + for k in sorted({k for t in traces for k in t.metadata}): + if k not in cols: + col_type = infer_column_type(t.metadata.get(k) for t in traces) + cols[k] = column_schema(k, + k.replace("_", " ").title(), col_type, True, + f"Metadata: {k}") + return list(cols.values()) + + def filter_and_sort( + self, + filters: Optional[List[Dict[str, Any]]] = None, + order_by: Optional[str] = None, + experiment_filter: Optional[Dict[str, Any]] = None, + trace_uuids: Optional[List[str]] = None, + limit: Optional[int] = None, + offset: int = 0, + ) -> Tuple[List[TraceEntry], int]: + """Returns (page of traces, number of traces matching before paging). + + A missing or non-positive limit means no limit. + """ + traces = self.get_traces() + if trace_uuids: + wanted = set(trace_uuids) + traces = [ + t for t in traces if t.trace_uuid in wanted or t.file_name in wanted + ] + experiment = parse_experiment_filter(experiment_filter) + if experiment: + traces = [t for t in traces if self._in_experiment_arm(t, experiment)] + if filters: + traces = [t for t in traces if evaluate_filters(t.get_value, filters)] + total = len(traces) + if order_by: + traces = sort_records(traces, lambda t, col: t.get_value(col), + parse_order_by(order_by)) + offset = max(0, offset) + end = offset + limit if limit and limit > 0 else None + return traces[offset:end], total + + @staticmethod + def _in_experiment_arm(trace: TraceEntry, experiment: Dict[str, Any]) -> bool: + if (as_int(trace.get_value("experiment_id"), + None) != experiment["experimentId"] or + as_int(trace.get_value("control_id"), None) != experiment["controlId"]): + return False + arm = trace.get_value("is_treatment") + return arm is None or as_bool(arm) == experiment["isTreatment"]
diff --git a/python/perfetto/bigtrace/server/cli.py b/python/perfetto/bigtrace/server/cli.py new file mode 100644 index 0000000..1a6fffe --- /dev/null +++ b/python/perfetto/bigtrace/server/cli.py
@@ -0,0 +1,159 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Command line entry point: tools/bigtrace_server.py.""" + +import argparse +import logging +import os +from typing import List, Optional + +from perfetto.bigtrace.server.archive import (DEFAULT_CACHE_DIR, + extract_archive, is_archive) +from perfetto.bigtrace.server.catalog import DEFAULT_UI_ORIGIN +from perfetto.bigtrace.server.manifest import generate_manifest +from perfetto.bigtrace.server.server import BigtraceServer, BigtraceServerConfig +from perfetto.bigtrace.server.util import MANIFEST_NAME, default_worker_count + + +def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Serves the Bigtrace UI API over a directory of traces.") + parser.add_argument( + "traces", + nargs="?", + default=".", + metavar="TRACES", + help="Directory of traces, or a zip/tar archive of them, which is " + f"extracted to {DEFAULT_CACHE_DIR} (default: current directory)") + parser.add_argument( + "-r", "--recursive", action="store_true", help="Scan subdirectories") + parser.add_argument( + "-m", + "--manifest", + help="Manifest to load: JSON, CSV (.csv) or a trace_fetch SQLite " + f"metadata.db (.db) (default: {MANIFEST_NAME} or metadata.db in the " + "trace directory, if present)") + parser.add_argument("-p", "--port", type=int, default=5001) + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument( + "-w", + "--workers", + type=int, + help=f"Traces queried in parallel (default: {default_worker_count()})") + parser.add_argument( + "--shell-path", help="Path to trace_processor_shell to use") + parser.add_argument( + "--parquet-dir", + metavar="DIR", + help="Optional directory to export each query's DuckDB table as Parquet " + "and load existing .parquet tables on startup (default: disabled)") + parser.add_argument( + "--duckdb-path", + metavar="PATH", + help="DuckDB database file to keep results in (default: a temporary " + "file under ~/.cache/perfetto/bigtrace/duckdb, deleted on exit, so " + "large results can spill to disk)") + parser.add_argument( + "--cors-origin", + action="append", + default=[], + metavar="ORIGIN", + help="Extra origin allowed to call the server; repeatable. Localhost " + "and https://ui.perfetto.dev are always allowed") + parser.add_argument( + "--origin", + default=DEFAULT_UI_ORIGIN, + help="Perfetto UI that trace links open in (default: %(default)s)") + + manifest = parser.add_argument_group("manifest generation") + manifest.add_argument( + "--generate-manifest", + nargs="?", + const=MANIFEST_NAME, + metavar="OUT", + help="Update the manifest before serving (default OUT: " + f"{MANIFEST_NAME} in the trace directory)") + manifest.add_argument( + "-f", + "--force", + action="store_true", + help="Re-extract every trace instead of reusing cached entries") + manifest.add_argument( + "--metadata-query", help="Extra SQL whose first row is added per trace") + args = parser.parse_args(argv) + if args.shell_path and not (os.path.isfile(args.shell_path) and + os.access(args.shell_path, os.X_OK)): + parser.error(f"--shell-path is not an executable: {args.shell_path}") + return args + + +def main(argv: Optional[List[str]] = None) -> None: + logging.basicConfig( + level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") + args = parse_args(argv) + + trace_dir, recursive = args.traces, args.recursive + if is_archive(trace_dir): + print(f"Unpacking {trace_dir}", flush=True) + trace_dir, recursive = extract_archive(trace_dir), True + + server = BigtraceServer( + BigtraceServerConfig( + trace_dir=trace_dir, + recursive=recursive, + manifest_path=args.manifest, + host=args.host, + port=args.port, + max_workers=args.workers, + bin_path=args.shell_path, + parquet_dir=(os.path.expanduser(args.parquet_dir) + if args.parquet_dir else None), + duckdb_path=(os.path.expanduser(args.duckdb_path) + if args.duckdb_path else None), + cors_origins=tuple(args.cors_origin), + ui_origin=args.origin, + )) + catalog = server.catalog + + if args.generate_manifest: + out = os.path.join(catalog.trace_dir, args.generate_manifest) + generate_manifest( + trace_dir=catalog.trace_dir, + output_path=out, + recursive=recursive, + max_workers=args.workers, + bin_path=args.shell_path, + metadata_query=args.metadata_query, + force=args.force, + ) + catalog.reload(manifest_path=out) + server.duckdb.sync_catalog(catalog) + + url = f"http://{args.host}:{server.server_port}" + source = catalog.manifest_path or catalog.trace_dir + parquet = server.duckdb.parquet_dir or "off" + ext = "loaded" if server.duckdb.extension_loaded else "unavailable" + print( + f"Bigtrace server on {url}\n" + f" Traces: {len(catalog.get_traces())} from {source}\n" + f" Workers: {server.config.max_workers}\n" + f" Parquet: {parquet}\n" + f" DuckDB: {server.duckdb.db_path} (perfetto ext: {ext})\n" + f"Open http://localhost:10000/bigtrace.html?server={url}", + flush=True) + server.start() + + +if __name__ == "__main__": + main()
diff --git a/python/perfetto/bigtrace/server/duckdb_engine.py b/python/perfetto/bigtrace/server/duckdb_engine.py new file mode 100644 index 0000000..e3233bb --- /dev/null +++ b/python/perfetto/bigtrace/server/duckdb_engine.py
@@ -0,0 +1,1164 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""DuckDB engine for storing trace query tables, execution history, and paging.""" + +import datetime +import decimal +import glob +import json +import logging +import os +import re +import shutil +import subprocess +import sys +import threading +from typing import (Any, Callable, Collection, Dict, List, Optional, Sequence, + Tuple) + +from perfetto.bigtrace.server.catalog import TraceCatalog +from perfetto.bigtrace.server.filters import parse_order_by +from perfetto.bigtrace.server.stream_ingest import TpStreamApi +from perfetto.bigtrace.server.util import iso_now + +logger = logging.getLogger(__name__) + +_EXT_DIR = os.path.abspath( + os.path.join(os.path.dirname(__file__), "..", "duckdb_ext")) +_EXT_PATH = os.path.join(_EXT_DIR, "perfetto.duckdb_extension") + +TERMINAL_STATUSES = ("SUCCESS", "FAILED", "CANCELLED") + +STATUS_FIELDS = ( + "queryUuid", + "status", + "startTime", + "endTime", + "processedRows", + "processedTraces", + "totalTraces", + "error", + "errorMessage", +) + +PUBLIC_FIELDS = STATUS_FIELDS + ( + "perfettoSql", + "queryEngine", + "limit", + "traceLimit", + "materialized", + "tableName", + "tableLink", + "schema", + "settings", + "traceFilters", + "traceMetadataColumns", + "traceOrderBy", + "experimentFilter", +) + +_RUNTIME_KEYS = ("_cancel", "_batch_cancel", "_active", "_duck_cursor", + "_deleted", "_ingest") + +_BIGTRACE_TO_DUCKDB_TYPE = { + "INT64": "BIGINT", + "DOUBLE": "DOUBLE", + "FLOAT64": "DOUBLE", + "BOOLEAN": "BOOLEAN", + "TIMESTAMP": "VARCHAR", + "STRING": "VARCHAR", +} + + +def public_view(ex: Dict[str, Any], + fields: Tuple[str, ...] = PUBLIC_FIELDS) -> Dict[str, Any]: + """Returns the wire form of an execution, omitting unset fields.""" + return {k: ex[k] for k in fields if ex.get(k) not in (None, [], {})} + + +def _dedup_column_names(columns: Sequence[str]) -> List[str]: + seen: Dict[str, int] = {} + unique_cols: List[str] = [] + for c in columns: + base = c or "col" + if base in seen: + seen[base] += 1 + unique_cols.append(f"{base}_{seen[base]}") + else: + seen[base] = 0 + unique_cols.append(base) + return unique_cols + + +def sanitize_table_name(name: str) -> str: + cleaned = re.sub(r"[^A-Za-z0-9_]", "_", name.strip()) + if not cleaned: + return "" + if cleaned[0].isdigit(): + cleaned = "t_" + cleaned + return cleaned + + +def _quote_ident(ident: str) -> str: + return '"' + ident.replace('"', '""') + '"' + + +def _quote_str(val: str) -> str: + return "'" + val.replace("'", "''") + "'" + + +def _load_duckdb_module(): + # Extensions resolve DuckDB symbols from the already loaded module, so it + # must be loaded RTLD_GLOBAL. Restore the flags for everyone else. + flags = sys.getdlopenflags() + if hasattr(os, "RTLD_GLOBAL") and hasattr(os, "RTLD_NOW"): + sys.setdlopenflags(os.RTLD_GLOBAL | os.RTLD_NOW) + try: + import duckdb # pylint: disable=import-outside-toplevel + finally: + sys.setdlopenflags(flags) + return duckdb + + +def _extension_is_stale() -> bool: + if not os.path.isfile(_EXT_PATH): + return True + built = os.path.getmtime(_EXT_PATH) + sources = [os.path.join(_EXT_DIR, "build.sh")] + for pattern in ("src/*.cc", "src/*.h", "sql/*.sql"): + sources += glob.glob(os.path.join(_EXT_DIR, pattern)) + return any( + os.path.getmtime(s) > built for s in sources if os.path.isfile(s)) + + +def _ensure_extension_built() -> str: + """Returns the extension path, building it first if missing or stale. + + Building needs g++ (duckdb_ext/build.sh). Without it a stale build is used + as is, and a missing one is an error. + """ + if not _extension_is_stale(): + return _EXT_PATH + build_script = os.path.join(_EXT_DIR, "build.sh") + have_build = os.path.isfile(_EXT_PATH) + if not os.path.isfile(build_script) or not shutil.which("g++"): + if have_build: + logger.warning( + "perfetto.duckdb_extension is older than its sources " + "and g++ is unavailable to rebuild it") + return _EXT_PATH + raise RuntimeError( + f"{_EXT_PATH} is missing; build it with {build_script} (needs g++)") + logger.info("Building perfetto.duckdb_extension") + try: + subprocess.run([build_script], + cwd=_EXT_DIR, + check=True, + capture_output=True, + text=True, + timeout=600) + except subprocess.CalledProcessError as e: + raise RuntimeError(f"Building {_EXT_PATH} failed:\n{e.stderr[-2000:]}") + except (OSError, subprocess.SubprocessError) as e: + raise RuntimeError(f"Building {_EXT_PATH} failed: {e}") from e + return _EXT_PATH + + +def _cgroup_memory_limit_bytes() -> Optional[int]: + """The memory limit of this process's cgroup (v2), if any.""" + try: + with open("/proc/self/cgroup") as f: + path = next( + (l.strip().split(":", 2)[2] for l in f if l.startswith("0::")), + None) + if path is None: + return None + # The tightest limit on the way up to the root applies. + limits = [] + while True: + try: + with open(f"/sys/fs/cgroup{path}/memory.max") as f: + val = f.read().strip() + if val != "max": + limits.append(int(val)) + except (OSError, ValueError): + pass + if path in ("/", ""): + break + path = os.path.dirname(path) + return min(limits) if limits else None + except OSError: + return None + + +def _default_memory_limit_gb() -> int: + # DuckDB spills to disk past this. Trace processors (often 100+ at a time, + # up to GBs each) share the machine, so leave them most of the memory. + try: + total = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") + except (ValueError, OSError, AttributeError): + return 16 + cgroup = _cgroup_memory_limit_bytes() + if cgroup: + total = min(total, cgroup) + return max(2, int((total / (1 << 30)) * 0.40)) + + +def _cleanup_stale_ephemeral_dbs(cache_dir: str) -> None: + try: + for fname in os.listdir(cache_dir): + m = re.match(r"^ephemeral_(\d+)_\d+\.duckdb", fname) + if not m: + continue + pid = int(m.group(1)) + if pid == os.getpid(): + continue + try: + os.kill(pid, 0) + except ProcessLookupError: + try: + os.remove(os.path.join(cache_dir, fname)) + except OSError: + pass + except PermissionError: + pass + except OSError: + pass + + +class DuckDBEngine: + """Manages the embedded DuckDB instance, query tables, history, and paging.""" + + def __init__(self, + db_path: Optional[str] = None, + parquet_dir: Optional[str] = None): + """Opens `db_path`, or if None a temporary database file under + ~/.cache/perfetto/bigtrace/duckdb (deleted on close) rather than an + in-memory one, so that large results can spill to disk.""" + self.db_path = db_path or ":memory:" + self._ephemeral_db_path: Optional[str] = None + if self.db_path == ":memory:": + cache_dir = os.path.join( + os.path.expanduser("~/.cache/perfetto/bigtrace"), "duckdb") + try: + os.makedirs(cache_dir, exist_ok=True) + _cleanup_stale_ephemeral_dbs(cache_dir) + ephem = os.path.join( + cache_dir, f"ephemeral_{os.getpid()}_{id(self)}.duckdb") + self._ephemeral_db_path = ephem + connect_path = ephem + except OSError: + connect_path = ":memory:" + else: + self.db_path = os.path.abspath(os.path.expanduser(self.db_path)) + os.makedirs(os.path.dirname(self.db_path), exist_ok=True) + connect_path = self.db_path + self.parquet_dir = (os.path.abspath(os.path.expanduser(parquet_dir)) + if parquet_dir else None) + if self.parquet_dir: + os.makedirs(self.parquet_dir, exist_ok=True) + self._lock = threading.RLock() + self._counter = 0 + self._table_meta: Dict[str, Dict[str, Any]] = {} + self._latest_table: Optional[str] = None + self._catalog_uuids: List[str] = [] + self._catalog_uuid_set: set[str] = set() + self._duckdb = _load_duckdb_module() + try: + self._conn = self._duckdb.connect( + connect_path, config={"allow_unsigned_extensions": "true"}) + except Exception as e: + wal_path = f"{connect_path}.wal" + if connect_path != ":memory:" and os.path.exists(wal_path): + logger.warning("Removing unreplayable DuckDB WAL %s (%s)", + wal_path, e) + try: + os.remove(wal_path) + except OSError: + pass + self._conn = self._duckdb.connect( + connect_path, config={"allow_unsigned_extensions": "true"}) + else: + raise + self._conn.execute("SET preserve_insertion_order = false") + self._conn.execute( + f"SET memory_limit = '{_default_memory_limit_gb()}GB'") + ext_path = _ensure_extension_built() + self._conn.execute(f"LOAD {_quote_str(ext_path)}") + self.extension_loaded = True + # Streams trace query results into tables; see stream_ingest.py. + self.stream_api: Optional[TpStreamApi] = TpStreamApi(ext_path) + self._init_history_schema() + if self.parquet_dir: + self._load_existing_parquet() + # User DuckDB queries must not reach the file system (they come from any + # page that can reach the server): only Parquet exports are allowed, and + # the configuration is locked so queries cannot undo this. + allowed = [self.parquet_dir] if self.parquet_dir else [] + self._conn.execute("SET allowed_directories = ?", [allowed]) + self._conn.execute("SET enable_external_access = false") + self._conn.execute("PRAGMA disable_print_progress_bar") + self._exec_cursors: List[Any] = [] + for _ in range(8): + c = self._conn.cursor() + c.execute("PRAGMA enable_progress_bar; " + "PRAGMA disable_print_progress_bar; " + "SET progress_bar_time = 0;") + self._exec_cursors.append(c) + self._conn.execute("SET lock_configuration = true") + + def _init_history_schema(self) -> None: + with self._lock: + self._conn.execute(""" + CREATE TABLE IF NOT EXISTS _bigtrace_executions ( + query_uuid VARCHAR PRIMARY KEY, + status VARCHAR NOT NULL, + start_time VARCHAR, + data VARCHAR NOT NULL + )""") + stale = self._conn.execute( + "SELECT data FROM _bigtrace_executions WHERE status = 'IN_PROGRESS'" + ).fetchall() + for (data,) in stale: + ex = json.loads(data) + ex.update(status="CANCELLED", + endTime=iso_now(), + errorMessage="Interrupted by server restart") + self.save_execution(ex) + for row in self.list_executions(): + tname = row.get("tableName") + uid = row.get("queryUuid") + if tname and not tname.startswith("_bigtrace"): + m = re.fullmatch(r"result_(\d+)", tname) + if m: + self._counter = max(self._counter, int(m.group(1))) + if self._physical_table_exists_locked(tname): + self._record_table_meta_locked( + tname, query_uuid=uid, sql=row.get("perfettoSql")) + + def _physical_table_exists_locked(self, table_name: str) -> bool: + row = self._conn.execute( + "SELECT 1 FROM information_schema.tables " + "WHERE table_schema = 'main' AND table_name = ?", + [table_name]).fetchone() + return row is not None + + def _load_existing_parquet(self) -> None: + if not self.parquet_dir or not os.path.isdir(self.parquet_dir): + return + with self._lock: + for fname in sorted(os.listdir(self.parquet_dir)): + if not fname.endswith(".parquet") or fname.startswith("."): + continue + tname = sanitize_table_name(fname[:-len(".parquet")]) + if not tname or tname.startswith("_bigtrace"): + continue + fpath = os.path.join(self.parquet_dir, fname) + try: + self._conn.execute( + f"CREATE OR REPLACE TABLE {_quote_ident(tname)} " + f"AS SELECT * FROM read_parquet({_quote_str(fpath)})") + self._record_table_meta_locked(tname) + m = re.fullmatch(r"result_(\d+)", tname) + if m: + self._counter = max(self._counter, int(m.group(1))) + except self._duckdb.Error as e: + logger.warning("Could not load parquet %s: %s", fpath, e) + + def _checkpoint_locked(self) -> None: + if self.db_path == ":memory:": + return + try: + self._conn.execute("CHECKPOINT") + except self._duckdb.Error: + pass + + # Execution metadata persistence in DuckDB -------------------------------- + + def save_execution(self, ex: Dict[str, Any]) -> None: + doc = {k: v for k, v in ex.items() if k not in _RUNTIME_KEYS} + with self._lock: + self._conn.execute( + "INSERT OR REPLACE INTO _bigtrace_executions VALUES (?, ?, ?, ?)", + [ + doc["queryUuid"], + doc.get("status", "UNKNOWN"), + doc.get("startTime"), + json.dumps(doc, default=str), + ]) + if doc.get("status") != "IN_PROGRESS": + self._checkpoint_locked() + + def get_execution(self, query_uuid: str) -> Optional[Dict[str, Any]]: + with self._lock: + row = self._conn.execute( + "SELECT data FROM _bigtrace_executions WHERE query_uuid = ?", + [query_uuid]).fetchone() + return json.loads(row[0]) if row else None + + def list_executions(self) -> List[Dict[str, Any]]: + with self._lock: + rows = self._conn.execute("SELECT data FROM _bigtrace_executions " + "ORDER BY start_time DESC").fetchall() + return [public_view(json.loads(data)) for (data,) in rows] + + def get_status(self, query_uuid: str) -> Optional[Dict[str, Any]]: + ex = self.get_execution(query_uuid) + return public_view(ex, STATUS_FIELDS) if ex else None + + def mark_cancelled(self, query_uuid: str) -> None: + ex = self.get_execution(query_uuid) + if ex and ex.get("status") == "IN_PROGRESS": + ex.update(status="CANCELLED", endTime=iso_now()) + self.save_execution(ex) + + def delete_execution_record(self, query_uuid: str) -> bool: + with self._lock: + exists = self._conn.execute( + "SELECT 1 FROM _bigtrace_executions WHERE query_uuid = ?", + [query_uuid]).fetchone() + if exists: + self._conn.execute( + "DELETE FROM _bigtrace_executions WHERE query_uuid = ?", + [query_uuid]) + self._checkpoint_locked() + return exists is not None + + # Catalog & Table naming -------------------------------------------------- + + def sync_catalog(self, catalog: TraceCatalog) -> None: + """Materializes `trace_metadata` with shared `_trace_uuid` ENUM type.""" + traces = catalog.get_traces() + schema = catalog.get_schema() + col_names = ["_trace_uuid"] + [ + c["name"] for c in schema if c["name"] != "_trace_uuid" + ] + col_types = {c["name"]: c["type"] for c in schema} + col_types["_trace_uuid"] = "STRING" + uuids = [t.trace_uuid for t in traces] + col_data: List[List[Any]] = [uuids] + for c in col_names[1:]: + col_data.append([t.get_value(c) for t in traces]) + with self._lock: + domain = list(dict.fromkeys(uuids)) + if 0 < len(domain) <= 65535 and domain != self._catalog_uuids: + try: + self._conn.execute('DROP TYPE IF EXISTS "_bt_trace_uuid"') + vals_sql = ", ".join(_quote_str(u) for u in domain) + self._conn.execute( + f'CREATE TYPE "_bt_trace_uuid" AS ENUM ({vals_sql})') + self._catalog_uuids = domain + self._catalog_uuid_set = set(domain) + except self._duckdb.Error as e: + # A persisted result table still uses the existing ENUM type: reuse it. + try: + existing = [ + r[0] for r in self._conn.execute( + 'SELECT unnest(enum_range(NULL::"_bt_trace_uuid"))::VARCHAR' + ).fetchall() + ] + self._catalog_uuids = existing + self._catalog_uuid_set = set(existing) + except self._duckdb.Error: + logger.warning( + "Could not update the trace UUID type: %s", e) + self.create_streaming_table("trace_metadata", col_names, col_types) + self._insert_metadata(col_names, col_types, col_data) + self.finalize_streaming_table("trace_metadata", + sql="-- Trace catalog metadata", + export=False) + + def next_table_name(self, + preferred: Optional[str] = None, + reserved: Optional[Collection[str]] = None) -> str: + with self._lock: + reserved_set = set(reserved or ()) + if preferred: + cleaned = sanitize_table_name(preferred) + if cleaned: + candidate = cleaned + idx = 2 + while candidate in reserved_set: + candidate = f"{cleaned}_{idx}" + idx += 1 + return candidate + while True: + self._counter += 1 + candidate = f"result_{self._counter}" + if (candidate not in self._table_meta and + candidate not in reserved_set and + not self._physical_table_exists_locked(candidate)): + return candidate + + def suggest_table_name(self, + base_name: str, + reserved: Optional[Collection[str]] = None) -> str: + cleaned = sanitize_table_name(base_name) or "tbl" + with self._lock: + reserved_set = set(reserved or ()) + idx = 1 + while True: + candidate = f"{cleaned}_{idx}" + if (candidate not in self._table_meta and + candidate not in reserved_set and + not self._physical_table_exists_locked(candidate)): + return candidate + idx += 1 + + def table_exists(self, table_name: str) -> Tuple[bool, str]: + resolved = sanitize_table_name(table_name) + if not resolved or resolved.startswith("_bigtrace"): + return False, "" + with self._lock: + if resolved in self._table_meta or self._physical_table_exists_locked( + resolved): + return True, resolved + return False, resolved + + # Direct streaming into DuckDB tables ------------------------------------- + + def create_streaming_table(self, + table_name: str, + columns: Sequence[str], + schema_map: Dict[str, str], + query_uuid: Optional[str] = None, + sql: Optional[str] = None, + uuid_enum: bool = True) -> Tuple[str, List[str]]: + """Creates or replaces an empty DuckDB table ready for streaming appends. + + `_trace_uuid` gets the shared `_bt_trace_uuid` ENUM type if `uuid_enum` + and the catalog defines it. + """ + resolved = sanitize_table_name(table_name) or self.next_table_name() + col_names = list(columns) or ["_trace_uuid"] + unique_cols = _dedup_column_names(col_names) + col_defs = [] + col_meta = [] + for orig, uniq in zip(col_names, unique_cols): + btype = schema_map.get(orig, "STRING") + if orig == "_trace_uuid" and uuid_enum and self._catalog_uuids: + dtype = '"_bt_trace_uuid"' + col_defs.append(f"{_quote_ident(uniq)} {dtype}") + col_meta.append({"name": uniq, "type": "STRING"}) + else: + dtype = _BIGTRACE_TO_DUCKDB_TYPE.get(btype, "VARCHAR") + col_defs.append(f"{_quote_ident(uniq)} {dtype}") + col_meta.append({ + "name": uniq, + "type": _duckdb_type_to_bigtrace(dtype) + }) + + qtable = _quote_ident(resolved) + with self._lock: + m = re.fullmatch(r"result_(\d+)", resolved) + if m: + self._counter = max(self._counter, int(m.group(1))) + try: + self._conn.execute(f"DROP VIEW IF EXISTS {qtable}") + except self._duckdb.CatalogException: + pass # A table of that name, which is replaced below. + self._conn.execute( + f"CREATE OR REPLACE TABLE {qtable} ({', '.join(col_defs)})") + prev = self._table_meta.get(resolved, {}) + self._table_meta[resolved] = { + "name": resolved, + "rowCount": 0, + "columns": col_meta, + "queryUuid": query_uuid or prev.get("queryUuid"), + "sql": sql or prev.get("sql"), + } + self._latest_table = resolved + return resolved, unique_cols + + def _insert_metadata(self, columns: Sequence[str], schema_map: Dict[str, + str], + col_data: Sequence[Sequence[Any]]) -> None: + """Fills trace_metadata, typed as created by create_streaming_table.""" + if not col_data or not col_data[0]: + return + types = [schema_map.get(c, "STRING") for c in columns] + rows = [ + tuple(_coerce_metadata_cell(v, t) + for v, t in zip(row, types)) + for row in zip(*col_data) + ] + placeholders = ", ".join(["?"] * len(columns)) + with self._lock: + self._conn.executemany( + f'INSERT INTO "trace_metadata" VALUES ({placeholders})', rows) + + def uuid_enum_covers(self, uuids: Sequence[str]) -> bool: + """Whether all `uuids` are values of the `_bt_trace_uuid` ENUM.""" + with self._lock: + return bool(self._catalog_uuid_set and + all(u in self._catalog_uuid_set for u in uuids)) + + def create_result_table(self, table_name: str, columns: Sequence[str], + types: Sequence[str], with_uuid: bool, + uuid_enum: bool, ord_column: Optional[str], + query_uuid: str, sql: str) -> Tuple[str, List[str]]: + """Creates the empty table a query over traces streams into. + + Columns: the query's, then `_trace_uuid` if `with_uuid`, then + `ord_column` (INTEGER) if set. Returns the table name and the unique + names of the query's columns. + """ + names = list(columns) + (["_trace_uuid"] if with_uuid else []) + schema = dict(zip(columns, types)) + if with_uuid: + schema["_trace_uuid"] = "STRING" + table, unique = self.create_streaming_table(table_name, + names, + schema, + query_uuid=query_uuid, + sql=sql, + uuid_enum=uuid_enum) + if ord_column: + with self._lock: + self._conn.execute( + f"ALTER TABLE {_quote_ident(table)} " + f"ADD COLUMN {_quote_ident(ord_column)} INTEGER") + return table, unique[:len(columns)] + + def insert_from_stream(self, table_name: str, handle: int, + names: Sequence[str], types: Sequence[str], + uuids: Optional[Sequence[str]], uuid_enum: bool, + ord_column: Optional[str], scan_threads: int) -> int: + """Drains stream `handle` into `table_name`; returns the rows inserted. + + Runs on its own cursor so it does not block other DuckDB calls. Rows get + `_trace_uuid` = uuids[ord] if `uuids` is set, and the trace ord in + `ord_column` if set; see create_result_table. + """ + qtable = _quote_ident(sanitize_table_name(table_name)) + proj = [f"p.{_quote_ident(n)}" for n in names] + if uuids is not None: + utype = '"_bt_trace_uuid"' if uuid_enum else "VARCHAR" + proj.append(f'p."_trace_uuid"::{utype}') + if ord_column: + proj.append('p."_trace_ord"') + params = [ + handle, scan_threads, + list(names), + list(types), + list(uuids or []) + ] + with self._lock: + cur = self._conn.cursor() + try: + row = cur.execute( + f"INSERT INTO {qtable} SELECT {', '.join(proj)} FROM " + "pf_scan_tp_stream(?, ?, ?::VARCHAR[], ?::VARCHAR[], ?::VARCHAR[]) p", + params).fetchone() + return int(row[0]) if row else 0 + finally: + cur.close() + + def delete_trace_rows(self, + table_name: str, + uuids: Optional[Sequence[str]] = None, + ords: Optional[Sequence[int]] = None, + ord_column: Optional[str] = None) -> None: + """Deletes the rows of traces whose results failed part way. + + Traces are identified by `_trace_uuid`, or by trace ord in `ord_column`. + """ + qtable = _quote_ident(sanitize_table_name(table_name)) + with self._lock: + if uuids: + self._conn.execute( + f'DELETE FROM {qtable} WHERE "_trace_uuid"::VARCHAR ' + "IN (SELECT unnest(?::VARCHAR[]))", [list(uuids)]) + if ords and ord_column: + self._conn.execute( + f"DELETE FROM {qtable} WHERE {_quote_ident(ord_column)} " + "IN (SELECT unnest(?::INTEGER[]))", [list(ords)]) + + def drop_column(self, table_name: str, column: str) -> None: + with self._lock: + self._conn.execute( + f"ALTER TABLE {_quote_ident(sanitize_table_name(table_name))} " + f"DROP COLUMN {_quote_ident(column)}") + + def row_count(self, table_name: str) -> int: + """Rows of `table_name` as of its last finalize_streaming_table().""" + with self._lock: + meta = self._table_meta.get(sanitize_table_name(table_name)) or {} + return int(meta.get("rowCount", 0)) + + def retype_columns(self, table_name: str, types: Dict[str, str]) -> None: + """Converts VARCHAR columns that only hold numbers to `types`.""" + qtable = _quote_ident(sanitize_table_name(table_name)) + with self._lock: + for col, btype in types.items(): + dtype = _BIGTRACE_TO_DUCKDB_TYPE.get(btype, "VARCHAR") + qcol = _quote_ident(col) + self._conn.execute( + f"ALTER TABLE {qtable} ALTER {qcol} TYPE {dtype} " + f"USING TRY_CAST({qcol} AS {dtype})") + + def finalize_streaming_table(self, + table_name: str, + query_uuid: Optional[str] = None, + sql: Optional[str] = None, + export: bool = True) -> None: + resolved = sanitize_table_name(table_name) + if not resolved: + return + with self._lock: + if not self._physical_table_exists_locked(resolved): + return + if self._latest_table == resolved and resolved != "trace_metadata": + self._conn.execute( + f"CREATE OR REPLACE VIEW latest AS SELECT * FROM {_quote_ident(resolved)}" + ) + if query_uuid: + alias = f"bigtrace_{query_uuid.replace('-', '_')}" + if alias != resolved: + self._conn.execute( + f"CREATE OR REPLACE VIEW {_quote_ident(alias)} " + f"AS SELECT * FROM {_quote_ident(resolved)}") + if export and self.parquet_dir: + pq_path = os.path.join(self.parquet_dir, f"{resolved}.parquet") + tmp_path = os.path.join(self.parquet_dir, + f".{resolved}.parquet.tmp") + try: + self._conn.execute( + f"COPY {_quote_ident(resolved)} TO {_quote_str(tmp_path)} " + "(FORMAT PARQUET, COMPRESSION ZSTD)") + os.replace(tmp_path, pq_path) + except (self._duckdb.Error, OSError) as e: + logger.warning("Parquet export for %s failed: %s", resolved, + e) + try: + os.remove(tmp_path) + except OSError: + pass + self._record_table_meta_locked(resolved, + query_uuid=query_uuid, + sql=sql) + self._checkpoint_locked() + + def _record_table_meta_locked(self, + table_name: str, + query_uuid: Optional[str] = None, + sql: Optional[str] = None) -> None: + qtable = _quote_ident(table_name) + count = self._conn.execute( + f"SELECT count(*) FROM {qtable}").fetchone()[0] + desc = self._conn.execute(f"DESCRIBE {qtable}").fetchall() + cols = [{ + "name": r[0], + "type": _duckdb_type_to_bigtrace(str(r[1])) + } for r in desc] + prev = self._table_meta.get(table_name, {}) + self._table_meta[table_name] = { + "name": table_name, + "rowCount": int(count), + "columns": cols, + "queryUuid": query_uuid or prev.get("queryUuid"), + "sql": sql or prev.get("sql"), + } + + def drop_table(self, + table_name: Optional[str], + query_uuid: Optional[str] = None) -> None: + if not table_name: + return + resolved = sanitize_table_name(table_name) + with self._lock: + owner = self._table_meta.get(resolved, {}).get("queryUuid") + if query_uuid and owner and owner != query_uuid: + resolved = "" + if resolved: + try: + self._conn.execute( + f"DROP TABLE IF EXISTS {_quote_ident(resolved)}") + except self._duckdb.Error as e: + logger.warning("Could not drop table %s: %s", resolved, e) + self._table_meta.pop(resolved, None) + if query_uuid: + alias = f"bigtrace_{query_uuid.replace('-', '_')}" + try: + self._conn.execute( + f"DROP VIEW IF EXISTS {_quote_ident(alias)}") + except self._duckdb.Error as e: + logger.warning("Could not drop view %s: %s", alias, e) + if self.parquet_dir and resolved: + pq_path = os.path.join(self.parquet_dir, f"{resolved}.parquet") + try: + os.remove(pq_path) + except OSError: + pass + self._checkpoint_locked() + + def list_tables(self) -> List[Dict[str, Any]]: + with self._lock: + return [ + dict(meta) + for name, meta in self._table_meta.items() + if not name.startswith("_bigtrace") + ] + + def execute_into_table( + self, + sql: str, + table_name: str, + limit: int, + query_uuid: Optional[str] = None, + on_cursor: Optional[Callable[[Any], None]] = None, + ) -> Tuple[List[str], Dict[str, str], int]: + """Runs DuckDB SQL natively in C++ and materializes directly into `table_name`.""" + resolved = sanitize_table_name(table_name) or self.next_table_name() + qtable = _quote_ident(resolved) + clean_sql = sql.strip().rstrip(";").strip() + with self._lock: + m = re.fullmatch(r"result_(\d+)", resolved) + if m: + self._counter = max(self._counter, int(m.group(1))) + try: + self._conn.execute(f"DROP VIEW IF EXISTS {qtable}") + except self._duckdb.CatalogException: + pass # A table of that name, which is replaced below. + from_pool = bool(self._exec_cursors) + cur = (self._exec_cursors.pop() + if from_pool else self._conn.cursor()) + try: + if not from_pool: + try: + cur.execute("SET enable_progress_bar=true; " + "SET enable_progress_bar_print=false; " + "SET progress_bar_time=0;") + except self._duckdb.Error: + pass + if on_cursor: + on_cursor(cur) + compiled_sql = sql + if self.extension_loaded: + try: + row = cur.execute("SELECT pf_compile_pipe(?)", + [sql]).fetchone() + if row and row[0]: + compiled_sql = str(row[0]) + except self._duckdb.Error: + compiled_sql = sql + clean_sql = compiled_sql.strip().rstrip(";").strip() + # The query runs as a plain read-only SELECT on its own cursor so that + # interrupt() is immediate and concurrent table writes never block on + # it. LIMIT is pushed into the query; only the output rows are streamed + # (as Arrow batches) into the destination table. + limit_sql = f" LIMIT {int(limit)}" if limit > 0 else "" + try: + cur.execute(f"SELECT * FROM ({clean_sql}) AS _bt_q WHERE 1 = 0") + wrappable = True + except self._duckdb.Error: + wrappable = False # E.g. PRAGMA / SHOW / multi-statement scripts. + cur.execute(compiled_sql) + if not cur.description: + return [], {}, 0 + description = cur.description + raw_columns = [d[0] for d in description] + unique_cols = _dedup_column_names(raw_columns) + schema_map = { + u: _duckdb_type_to_bigtrace(str(d[1])) + for u, d in zip(unique_cols, description) + } + select_exprs = [] + for orig, uniq, desc in zip(raw_columns, unique_cols, description): + dtype = str(desc[1]).upper() + qorig, quniq = _quote_ident(orig), _quote_ident(uniq) + if any(t in dtype + for t in ("TIMESTAMP", "DATE", "TIME", "INTERVAL")): + select_exprs.append(f"CAST({qorig} AS VARCHAR) AS {quniq}") + elif orig != uniq: + select_exprs.append(f"{qorig} AS {quniq}") + else: + select_exprs.append(qorig) + select_cols = ", ".join(select_exprs) + if self.stream_api is not None: + self.stream_api.reset_query_progress() + if wrappable: + cur.execute(f"CREATE OR REPLACE TABLE {qtable} AS " + f"SELECT {select_cols} FROM ({clean_sql}) AS _bt_q" + f"{limit_sql}") + else: + rows = cur.fetchmany(limit) if limit > 0 else cur.fetchall() + self.create_streaming_table(resolved, + unique_cols, + schema_map, + query_uuid=query_uuid, + sql=sql, + uuid_enum=False) + if rows: + placeholders = ", ".join(["?"] * len(unique_cols)) + cur.executemany( + f"INSERT INTO {qtable} VALUES ({placeholders})", + [[_to_cell(v) for v in r] for r in rows]) + with self._lock: + self._latest_table = resolved + self.finalize_streaming_table(resolved, + query_uuid=query_uuid, + sql=sql) + total_rows = int( + self._table_meta.get(resolved, {}).get("rowCount", 0)) + finally: + if self.stream_api is not None: + self.stream_api.reset_query_progress() + if from_pool: + with self._lock: + self._exec_cursors.append(cur) + else: + cur.close() + return unique_cols, schema_map, total_rows + + def query_page( + self, + table_name: str, + all_columns: Sequence[str], + out_indices: Sequence[int], + offset: int = 0, + limit: int = 50, + order_by: Optional[str] = None, + filters: Optional[List[Dict[str, Any]]] = None, + ) -> Tuple[int, List[List[Any]]]: + """Streams a filtered, sorted, projected page directly from DuckDB.""" + resolved = sanitize_table_name(table_name) + unique_cols = _dedup_column_names(all_columns) + if not resolved or not unique_cols: + return 0, [] + with self._lock: + if not self._physical_table_exists_locked(resolved): + return 0, [] + cur = self._conn.cursor() + try: + qtable = _quote_ident(resolved) + phys_cols = { + r[0] for r in cur.execute(f"DESCRIBE {qtable}").fetchall() + } + has_meta = (resolved != "trace_metadata" and + self._physical_table_exists_locked("trace_metadata")) + meta_cols = ({ + r[0] + for r in cur.execute('DESCRIBE "trace_metadata"').fetchall() + } if has_meta else set()) + col_set = set(unique_cols) + orig_to_uniq: Dict[str, str] = {} + for orig, uniq in zip(all_columns, unique_cols): + orig_to_uniq.setdefault(orig, uniq) + orig_to_uniq[uniq] = uniq + + needs_meta_join = False + + def _col_expr(uniq: str) -> str: + nonlocal needs_meta_join + if uniq in phys_cols: + return f"_bt_t.{_quote_ident(uniq)}" + if uniq in meta_cols: + needs_meta_join = True + return f"_bt_tm.{_quote_ident(uniq)}" + return "NULL" + + where_parts: List[str] = [] + params: List[Any] = [] + for f in filters or (): + field = f.get("field") + if not field: + continue + op = str(f.get("op", "=")).lower() + target = f.get("value") + col_ident = (_col_expr(orig_to_uniq[field]) + if field in orig_to_uniq else "NULL") + cond, cond_params = _build_filter_sql(col_ident, op, target) + where_parts.append(cond) + params.extend(cond_params) + + order_parts: List[str] = [] + for field, desc in parse_order_by(order_by): + uniq = orig_to_uniq.get(field) + if not uniq or uniq not in col_set: + continue + c = _col_expr(uniq) + direction = "DESC" if desc else "ASC" + is_empty = (f"({c} IS NULL OR CAST({c} AS VARCHAR) = '' " + f"OR UPPER(CAST({c} AS VARCHAR)) = 'NULL')") + num_val = f"TRY_CAST({c} AS DOUBLE)" + if desc: + bucket = (f"CASE WHEN {is_empty} THEN 0 " + f"WHEN {num_val} IS NOT NULL THEN 2 ELSE 1 END") + else: + bucket = (f"CASE WHEN {is_empty} THEN 2 " + f"WHEN {num_val} IS NOT NULL THEN 0 ELSE 1 END") + order_parts.append(f"{bucket} ASC") + order_parts.append(f"{num_val} {direction}") + order_parts.append(f"LOWER(CAST({c} AS VARCHAR)) {direction}") + if order_parts: + order_parts.append("_bt_t.rowid ASC") + order_sql = f" ORDER BY {', '.join(order_parts)}" + else: + order_sql = "" + + proj = ", ".join(f"{_col_expr(unique_cols[i])} AS c_{k}" + for k, i in enumerate(out_indices)) + if not proj: + proj = "1" + + from_base = f"{qtable} AS _bt_t" + join_sql = (' LEFT JOIN "trace_metadata" AS _bt_tm ' + 'ON _bt_t."_trace_uuid" = _bt_tm."_trace_uuid"' + if needs_meta_join else "") + where_sql = f" WHERE {' AND '.join(where_parts)}" if where_parts else "" + count_from = f"{from_base}{join_sql}" if where_parts else from_base + total = cur.execute(f"SELECT count(*) FROM {count_from}{where_sql}", + params).fetchone()[0] + if limit <= 0 or offset >= total: + return int(total), [] + + sql = ( + f"SELECT {proj} FROM {from_base}{join_sql}{where_sql}{order_sql} " + f"LIMIT {int(limit)} OFFSET {int(offset)}") + rows = cur.execute(sql, params).fetchall() + return int(total), [ + [_to_cell(v) for v in r] if out_indices else [] for r in rows + ] + finally: + cur.close() + + def close(self) -> None: + with self._lock: + for c in self._exec_cursors: + try: + c.close() + except Exception: # pylint: disable=broad-except + pass + self._exec_cursors.clear() + self._conn.close() + if self._ephemeral_db_path: + for suffix in ("", ".wal"): + try: + os.remove(self._ephemeral_db_path + suffix) + except OSError: + pass + + +_SUPPORTED_FILTER_OPS = { + "=", "!=", "<", "<=", ">", ">=", "glob", "not glob", "in", "not in", + "is null", "is not null" +} + + +def _sql_equals_expr(col_ident: str, target: Any) -> Tuple[str, List[Any]]: + s_target = "" if target is None else str(target) + try: + f_target = float(target) + return (f"(LOWER(CAST({col_ident} AS VARCHAR)) = LOWER(?) OR " + f"TRY_CAST({col_ident} AS DOUBLE) = ?)", [s_target, f_target]) + except (TypeError, ValueError): + return f"LOWER(CAST({col_ident} AS VARCHAR)) = LOWER(?)", [s_target] + + +def _build_filter_sql(col_ident: str, op: str, + target: Any) -> Tuple[str, List[Any]]: + if op not in _SUPPORTED_FILTER_OPS: + raise ValueError(f"Unsupported filter op: {op}") + is_empty = (f"({col_ident} IS NULL OR CAST({col_ident} AS VARCHAR) = '' " + f"OR UPPER(CAST({col_ident} AS VARCHAR)) = 'NULL')") + if op == "is null": + return is_empty, [] + if op == "is not null": + return f"NOT {is_empty}", [] + if col_ident == "NULL": + return "FALSE", [] + if op == "=": + eq, p = _sql_equals_expr(col_ident, target) + return f"({col_ident} IS NOT NULL AND {eq})", p + if op == "!=": + eq, p = _sql_equals_expr(col_ident, target) + return f"({col_ident} IS NOT NULL AND NOT {eq})", p + if op in ("<", "<=", ">", ">="): + s_target = "" if target is None else str(target) + try: + f_target = float(target) + return ( + f"({col_ident} IS NOT NULL AND CASE " + f"WHEN TRY_CAST({col_ident} AS DOUBLE) IS NOT NULL " + f"THEN TRY_CAST({col_ident} AS DOUBLE) {op} ? " + f"ELSE CAST({col_ident} AS VARCHAR) {op} ? END)", + [f_target, s_target], + ) + except (TypeError, ValueError): + return f"({col_ident} IS NOT NULL AND CAST({col_ident} AS VARCHAR) {op} ?)", [ + s_target + ] + if op in ("glob", "not glob"): + pattern = "" if target is None else str(target) + neg = "NOT " if op == "not glob" else "" + return f"({col_ident} IS NOT NULL AND {neg}(CAST({col_ident} AS VARCHAR) GLOB ?))", [ + pattern + ] + targets = target if isinstance(target, (list, tuple)) else [target] + if not targets: + return ("FALSE" if op == "in" else f"({col_ident} IS NOT NULL)"), [] + parts = [] + params: List[Any] = [] + for t in targets: + eq, p = _sql_equals_expr(col_ident, t) + parts.append(eq) + params.extend(p) + joined = " OR ".join(parts) + if op == "in": + return f"({col_ident} IS NOT NULL AND ({joined}))", params + return f"({col_ident} IS NOT NULL AND NOT ({joined}))", params + + +def _duckdb_type_to_bigtrace(dtype: str) -> str: + u = dtype.upper() + if "INT" in u and "INTERVAL" not in u and "[" not in u and "(" not in u: + return "INT64" + if u.startswith(("DOUBLE", "FLOAT", "DECIMAL", "REAL", "NUMERIC")): + return "DOUBLE" + if u == "BOOLEAN": + return "BOOLEAN" + return "STRING" + + +def _to_cell(v: Any) -> Any: + if v is None or isinstance(v, (bool, int, float, str)): + return v + if isinstance(v, (bytes, bytearray, memoryview)): + import base64 # pylint: disable=import-outside-toplevel + return base64.b64encode(bytes(v)).decode("ascii") + if isinstance(v, decimal.Decimal): + return float(v) + if isinstance(v, (datetime.datetime, datetime.date, datetime.time)): + return v.isoformat() + if isinstance(v, (list, tuple, dict)): + return json.dumps(v, default=str) + return str(v) + + +def _coerce_metadata_cell(value: Any, btype: str) -> Any: + if value is None: + return None + try: + if btype == "INT64": + return int(value) + if btype in ("DOUBLE", "FLOAT64"): + return float(value) + if btype == "BOOLEAN": + if isinstance(value, str): + return value.lower() in ("true", "1") + return bool(value) + return str(value) + except (TypeError, ValueError): + return None
diff --git a/python/perfetto/bigtrace/server/executor.py b/python/perfetto/bigtrace/server/executor.py new file mode 100644 index 0000000..fdc9feb --- /dev/null +++ b/python/perfetto/bigtrace/server/executor.py
@@ -0,0 +1,875 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Runs queries over traces or DuckDB tables, materializing results in DuckDB.""" + +import collections +from concurrent.futures import FIRST_COMPLETED, Future, ThreadPoolExecutor, wait +import logging +import os +import re +import threading +import time +from typing import Any, Callable, Dict, List, Optional, Set, Tuple +import uuid + +from perfetto.trace_processor.api import TraceProcessorConfig +from perfetto.bigtrace.server.catalog import ( + SettingsOverrides, + TraceCatalog, + TraceEntry, +) +from perfetto.bigtrace.server.duckdb_engine import ( + DuckDBEngine, + STATUS_FIELDS, + TERMINAL_STATUSES, + public_view, +) +from perfetto.bigtrace.server.filters import parse_experiment_filter +from perfetto.bigtrace.server import stream_ingest +from perfetto.bigtrace.server.stream_ingest import FeedResult, StreamIngest +from perfetto.bigtrace.server.trace_processor_pool import ( + Shell, + ShellCancelled, + ShellStartError, + TraceProcessorPool, +) +from perfetto.bigtrace.server.util import ( + as_int, + get_param, + iso_now, + to_wire, +) + +logger = logging.getLogger(__name__) + +_DUCKDB_DIRECTIVE = re.compile(r"^\s*--\s*(?:@duckdb|engine\s*:\s*duckdb)\b", + re.IGNORECASE) +# Holds trace ords in result tables whose query has its own `_trace_uuid` +# column, so the rows of failed traces can be deleted; dropped at the end. +_ORD_COLUMN = "_bt_trace_ord" + + +class QueryFailedError(Exception): + pass + + +class BigtraceQueryExecutor: + """Runs PerfettoSQL over traces or DuckDB SQL over materialized tables. + + Every query streams its rows directly into DuckDB (`result_1`, `result_2`, + or a custom `table_name`, plus `latest`) and serves UI pages directly from + DuckDB. + """ + + def __init__( + self, + catalog: TraceCatalog, + tp_config: Optional[TraceProcessorConfig] = None, + max_workers: int = 4, + cache_size: int = 16, + duckdb_engine: Optional[DuckDBEngine] = None, + ): + self.catalog = catalog + self.cache_size = cache_size + self.max_workers = max_workers + self.pool = TraceProcessorPool(tp_config) + self.duckdb = duckdb_engine or DuckDBEngine() + self.duckdb.sync_catalog(self.catalog) + self._workers = ThreadPoolExecutor(max_workers=max_workers) + self._lock = threading.RLock() + self._executions: "collections.OrderedDict[str, Dict[str, Any]]" = ( + collections.OrderedDict()) + self._runs: List[threading.Thread] = [] + + def close(self) -> None: + with self._lock: + exs = list(self._executions.values()) + runs = list(self._runs) + for ex in exs: + self._stop(ex) + self.pool.close() + self._workers.shutdown(wait=True, cancel_futures=True) + for run in runs: + run.join(timeout=10) + self.duckdb.close() + + def _persist(self, write: Callable[..., Any], *args: Any) -> None: + try: + write(*args) + except Exception as e: # pylint: disable=broad-except + logger.warning("Could not save query history: %s", e) + + def _remember(self, ex: Dict[str, Any]) -> None: + with self._lock: + self._executions[ex["queryUuid"]] = ex + self._executions.move_to_end(ex["queryUuid"]) + finished = [ + k for k, v in self._executions.items() + if v["status"] in TERMINAL_STATUSES + ] + for k in finished[:max(0, len(finished) - self.cache_size)]: + del self._executions[k] + + def _load(self, query_uuid: str) -> Dict[str, Any]: + with self._lock: + ex = self._executions.get(query_uuid) + if ex: + self._executions.move_to_end(query_uuid) + return ex + ex = self.duckdb.get_execution(query_uuid) + if ex is None: + raise KeyError(query_uuid) + ex["_cancel"] = threading.Event() + ex["_active"] = set() + self._remember(ex) + return ex + + def list_executions(self) -> List[Dict[str, Any]]: + by_uuid = {ex["queryUuid"]: ex for ex in self.duckdb.list_executions()} + with self._lock: + for query_uuid, ex in self._executions.items(): + by_uuid[query_uuid] = public_view(ex) + return sorted(by_uuid.values(), + key=lambda e: e.get("startTime") or "", + reverse=True) + + def get_execution(self, query_uuid: str) -> Optional[Dict[str, Any]]: + with self._lock: + if query_uuid in self._executions: + return public_view(self._executions[query_uuid]) + ex = self.duckdb.get_execution(query_uuid) + return public_view(ex) if ex else None + + def get_status(self, query_uuid: str) -> Optional[Dict[str, Any]]: + with self._lock: + if query_uuid in self._executions: + ex = self._executions[query_uuid] + if (ex.get("status") == "IN_PROGRESS" and + ex.get("queryEngine") == "duckdb"): + pct = -1.0 + cur = ex.get("_duck_cursor") + if cur is not None and hasattr(cur, "query_progress"): + try: + pct = max(pct, float(cur.query_progress())) + except Exception: # pylint: disable=broad-except + pass + if self.duckdb.stream_api is not None: + try: + pct = max( + pct, + float( + self.duckdb.stream_api.get_query_progress() + ), + ) + except Exception: # pylint: disable=broad-except + pass + if pct >= 0.0: + ex["processedTraces"] = max( + int(ex.get("processedTraces") or 0), + min(99, int(round(pct))), + ) + ex["totalTraces"] = 100 + return public_view(ex, STATUS_FIELDS) + return self.duckdb.get_status(query_uuid) + + def check_table_exists(self, table_name: str) -> Dict[str, Any]: + exists_duck, resolved = self.duckdb.table_exists(table_name) + all_exs = self.list_executions() + in_flight = any( + ex.get("status") == "IN_PROGRESS" and ex.get("tableName") in ( + table_name, resolved) for ex in all_exs) + hist_names = { + ex["tableName"] + for ex in all_exs + if ex.get("tableName") and (ex.get("status") == "IN_PROGRESS" or + (ex.get("processedRows") or 0) > 0) + } + base = resolved or table_name + suggested = self.duckdb.suggest_table_name(base, reserved=hist_names) + return { + "exists": exists_duck or in_flight, + "resolvedTableName": base, + "suggestedTableName": suggested, + } + + def _stop(self, ex: Dict[str, Any]) -> None: + """Stops a running query, killing the trace processors running it.""" + with self._lock: + cancel = ex.get("_cancel") + if cancel: + cancel.set() + batch_cancel = ex.get("_batch_cancel") + if isinstance(batch_cancel, threading.Event): + if all( + e.get("_cancel", threading.Event()).is_set() + for e in self._executions.values() + if e.get("_batch_cancel") is batch_cancel): + batch_cancel.set() + active = list(ex.get("_active", ())) + if "_active" in ex: + ex["_active"].clear() + shared = { + shell for other in self._executions.values() + if other is not ex and not other["_cancel"].is_set() + for shell in other.get("_active", ()) + } + to_close = [shell for shell in active if shell not in shared] + duck_cursor = ex.get("_duck_cursor") + ingest: Optional[StreamIngest] = ex.get("_ingest") + if ingest is not None: + ingest.cancel() + for shell in to_close: + shell.close() + if duck_cursor is not None: + threading.Thread(target=self._interrupt_until_done, + args=(ex,), + daemon=True).start() + + def _interrupt_until_done(self, ex: Dict[str, Any]) -> None: + """Interrupts `ex`'s DuckDB cursor until its run ends. + + A single interrupt() is lost if it lands before the cursor's next + statement starts, so keep interrupting while the run holds a cursor. + """ + deadline = time.monotonic() + 60 + while time.monotonic() < deadline: + with self._lock: + cur = ex.get("_duck_cursor") + if cur is None: + return + try: + cur.interrupt() + except Exception: # pylint: disable=broad-except + pass + time.sleep(0.05) + + def cancel_query(self, query_uuid: str) -> None: + with self._lock: + ex = self._executions.get(query_uuid) + if ex and ex["status"] == "IN_PROGRESS": + ex["status"] = "CANCELLED" + ex["endTime"] = iso_now() + if ex: + self._stop(ex) + self._persist(self.duckdb.mark_cancelled, query_uuid) + + def delete_execution(self, query_uuid: str) -> bool: + table_name = None + with self._lock: + ex = self._executions.pop(query_uuid, None) + if ex: + table_name = ex.get("tableName") + ex["_deleted"] = True + if ex: + self._stop(ex) + if not table_name: + stored = self.duckdb.get_execution(query_uuid) + if stored: + table_name = stored.get("tableName") + self.duckdb.drop_table(table_name, query_uuid) + return self.duckdb.delete_execution_record(query_uuid) or ex is not None + + @staticmethod + def _parse_query_specs( + params: Dict[str, Any], + default_sql: str, + default_table: Optional[str], + default_limit: int, + ) -> List[Tuple[str, Optional[str], int]]: + raw_queries = get_param(params, "queries") + specs: List[Tuple[str, Optional[str], int]] = [] + if isinstance(raw_queries, list) and raw_queries: + for item in raw_queries: + if isinstance(item, dict): + q_sql = str( + get_param(item, "perfetto_sql", "") or + get_param(item, "sql", "")) + q_tbl = get_param(item, "table_name") + q_lim = max(0, + as_int(get_param(item, "limit"), default_limit)) + if q_sql.strip(): + specs.append( + (q_sql, str(q_tbl) if q_tbl else None, q_lim)) + elif isinstance(item, str) and item.strip(): + specs.append((item, None, default_limit)) + if not specs and default_sql.strip(): + specs.append( + (default_sql, str(default_table) if default_table else None, + default_limit)) + return specs + + def execute(self, + params: Dict[str, Any], + is_async: bool = False) -> Dict[str, Any]: + """Starts one or more queries. Sync calls block and return the first page.""" + sql = str(get_param(params, "perfetto_sql", "")) + req_table = get_param(params, "table_name") + limit = max(0, as_int(get_param(params, "limit"), 10000)) + specs = self._parse_query_specs(params, sql, + str(req_table) if req_table else None, + limit) + if not specs: + raise ValueError("perfetto_sql is required") + + first_sql = specs[0][0] + engine = str( + get_param(params, "engine", "") or + get_param(params, "query_engine", "")).lower().strip() + if not engine and _DUCKDB_DIRECTIVE.match(first_sql): + engine = "duckdb" + engine = "duckdb" if engine == "duckdb" else "traces" + + trace_limit = as_int(get_param(params, "trace_limit"), 0) or None + settings = get_param(params, "settings", []) + trace_filters = get_param(params, "trace_filters", []) + trace_order_by = get_param(params, "trace_order_by") + experiment_filter = parse_experiment_filter( + get_param(params, "experiment_filter")) + + traces: List[TraceEntry] = [] + if engine == "traces": + overrides = SettingsOverrides.parse(settings) + if self.catalog.apply_overrides(overrides): + self.duckdb.sync_catalog(self.catalog) + traces, _ = self.catalog.filter_and_sort( + filters=trace_filters, + order_by=trace_order_by, + experiment_filter=experiment_filter, + trace_uuids=overrides.trace_uuids, + limit=trace_limit, + ) + + req_meta = list(get_param(params, "trace_metadata_columns", [])) + if engine == "traces" and "_trace_uuid" not in req_meta: + req_meta = req_meta + ["_trace_uuid"] + + norm_settings = [{ + "settingId": get_param(s, "setting_id", ""), + "values": [str(v) for v in s.get("values") or []], + "category": s.get("category", "TRACE_ADDRESS"), + } for s in settings if isinstance(s, dict)] + + batch_cancel = threading.Event() + executions: List[Dict[str, Any]] = [] + with self._lock: + reserved_tables = { + e["tableName"] + for e in self._executions.values() + if e.get("status") == "IN_PROGRESS" and e.get("tableName") + } + for q_sql, q_table, q_lim in specs: + q_uuid = str(uuid.uuid4()) + t_name = self.duckdb.next_table_name(q_table, + reserved=reserved_tables) + reserved_tables.add(t_name) + ex: Dict[str, Any] = dict( + queryUuid=q_uuid, + status="IN_PROGRESS", + startTime=iso_now(), + endTime=None, + processedRows=0, + processedTraces=0, + totalTraces=len(traces) if engine == "traces" else 1, + perfettoSql=q_sql, + queryEngine=engine, + limit=q_lim, + traceLimit=trace_limit, + materialized=True, + tableName=t_name, + tableLink=None, + schema=[], + settings=norm_settings, + traceFilters=trace_filters, + traceMetadataColumns=req_meta, + traceOrderBy=trace_order_by, + experimentFilter=experiment_filter, + error=None, + errorMessage=None, + _columns=[], + _schema_map={}, + _cancel=threading.Event(), + _batch_cancel=batch_cancel, + _active=set(), + ) + self._remember(ex) + executions.append(ex) + for ex in executions: + self._persist(self.duckdb.save_execution, ex) + + primary = executions[0] + if engine == "duckdb": + target = self._run_duckdb_batch + args: Tuple[Any, ...] = (executions,) + else: + target = self._run_batch + args = (executions, traces) + + if is_async: + run = threading.Thread(target=target, args=args, daemon=True) + with self._lock: + self._runs = [r for r in self._runs if r.is_alive()] + [run] + run.start() + return { + "queryUuid": primary["queryUuid"], + "tableName": primary["tableName"], + "executions": [{ + "queryUuid": e["queryUuid"], + "tableName": e["tableName"], + } for e in executions], + "schema": [], + "rows": [], + "totalFilteredRows": 0, + } + target(*args) + successful = next((e for e in executions if e["status"] == "SUCCESS"), + None) + if not successful and primary["status"] == "FAILED": + raise QueryFailedError(primary["errorMessage"]) + active_primary = successful or primary + sync_limit = (active_primary["limit"] if active_primary["limit"] > 0 + else active_primary["processedRows"]) + res = self.fetch_results(active_primary["queryUuid"], + offset=0, + limit=sync_limit) + res["executions"] = [{ + "queryUuid": e["queryUuid"], + "tableName": e["tableName"], + } for e in executions] + return res + + def _run_duckdb_batch(self, executions: List[Dict[str, Any]]) -> None: + for ex in executions: + self._run_duckdb(ex) + + def _run_duckdb(self, ex: Dict[str, Any]) -> None: + """Runs a DuckDB query, writing its results into a DuckDB table.""" + + def on_cursor(cur: Any) -> None: + with self._lock: + if ex["_cancel"].is_set(): + raise QueryFailedError("cancelled") + ex["_duck_cursor"] = cur + + try: + columns, schema_map, row_count = self.duckdb.execute_into_table( + ex["perfettoSql"], + ex["tableName"], + ex["limit"], + query_uuid=ex["queryUuid"], + on_cursor=on_cursor, + ) + with self._lock: + ex.pop("_duck_cursor", None) + ex["_columns"] = columns + ex["_schema_map"] = schema_map + ex["schema"] = [{ + "name": c, + "type": schema_map.get(c, "STRING") + } for c in columns] + ex["processedTraces"] = 1 + ex["processedRows"] = row_count + if ex["status"] != "CANCELLED": + ex["status"] = "SUCCESS" + except Exception as e: # pylint: disable=broad-except + has_table, _ = self.duckdb.table_exists(ex.get("tableName") or "") + with self._lock: + ex.pop("_duck_cursor", None) + if not has_table: + ex["tableName"] = None + if ex["status"] != "CANCELLED": + ex["status"] = "FAILED" + ex["errorMessage"] = f"DuckDB error: {e}" + with self._lock: + ex["endTime"] = iso_now() + deleted = ex.get("_deleted") + if deleted: + self.duckdb.drop_table(ex.get("tableName"), ex["queryUuid"]) + return + self._persist(self.duckdb.save_execution, ex) + self._remember(ex) + + # Trace queries ------------------------------------------------------------- + + def _new_run(self, ex: Dict[str, Any], traces: List[TraceEntry], + catalog_types: Dict[str, str]) -> "_TraceRun": + assert self.duckdb.stream_api is not None + run = _TraceRun(ex, traces) + uuids = [t.trace_uuid for t in traces] + scan_threads = stream_ingest.default_scan_threads() + + def on_schema(columns: List[str], + types: List[str]) -> Callable[[int], int]: + with_uuid = "_trace_uuid" not in columns + uuid_enum = with_uuid and self.duckdb.uuid_enum_covers(uuids) + table, out_names = self.duckdb.create_result_table( + ex["tableName"], + columns, + types, + with_uuid=with_uuid, + uuid_enum=uuid_enum, + ord_column=None if with_uuid else _ORD_COLUMN, + query_uuid=ex["queryUuid"], + sql=ex["perfettoSql"]) + meta = ["_trace_uuid" + ] + [c for c in catalog_types if c != "_trace_uuid"] + meta = [c for c in meta if c not in columns] + with self._lock: + run.table = table + run.columns = list(columns) + run.with_uuid = with_uuid + ex["tableName"] = table + ex["_columns"] = list(columns) + meta + ex["_schema_map"] = {c: catalog_types[c] for c in meta} + ex["_schema_map"].update(zip(columns, types)) + self._update_schema(ex, columns) + + def insert(handle: int) -> int: + return self.duckdb.insert_from_stream( + table, + handle, + out_names, + types, + uuids=uuids if with_uuid else None, + uuid_enum=uuid_enum, + ord_column=None if with_uuid else _ORD_COLUMN, + scan_threads=scan_threads) + + return insert + + run.ingest = StreamIngest(self.duckdb.stream_api, + ex["perfettoSql"], + on_schema, + row_limit=ex["limit"]) + with self._lock: + ex["_ingest"] = run.ingest + return run + + def _query_trace(self, runs: List["_TraceRun"], trace: TraceEntry, + trace_ord: int, + batch_cancel: threading.Event) -> List["_Outcome"]: + """Loads `trace` once and feeds each run's query on it.""" + outcomes: List[_Outcome] = [(None, None)] * len(runs) + if not any(r.wants_more() for r in runs): + return outcomes + if not os.path.isfile(trace.file_path): + return [(None, FileNotFoundError("trace file not found")) + ] * len(runs) + try: + shell = self.pool.acquire(trace.file_path, batch_cancel) + except ShellCancelled: + return outcomes + except ShellStartError as e: + return [(None, e)] * len(runs) + with self._lock: + for r in runs: + r.ex["_active"].add(shell) + try: + for i, run in enumerate(runs): + if not run.wants_more(): + continue + if shell.closed: + break # Killed by a cancel. + res = run.ingest.feed(shell.host, shell.port, trace_ord) + outcomes[i] = (res, None) + if res.status == stream_ingest.FEED_IO_ERROR: + if shell.proc.poll() is not None: + err = RuntimeError( + f"trace processor died: {res.message}") + for j in range(i + 1, len(runs)): + outcomes[j] = (None, err) + break + return outcomes + finally: + with self._lock: + for r in runs: + r.ex["_active"].discard(shell) + self.pool.release(shell) + + def _run_batch(self, executions: List[Dict[str, Any]], + traces: List[TraceEntry]) -> None: + """Runs 1..N queries per trace in a single pass, streaming into DuckDB.""" + runs: List[_TraceRun] = [] + futures: Dict[Future, Tuple[int, TraceEntry]] = {} + try: + if self.duckdb.stream_api is None: + raise QueryFailedError( + "The perfetto DuckDB extension is not loaded") + self.duckdb.sync_catalog(self.catalog) + catalog_types = { + c["name"]: c["type"] for c in self.catalog.get_schema() + } + catalog_types["_trace_uuid"] = "STRING" + for ex in executions: + runs.append(self._new_run(ex, traces, catalog_types)) + with self._lock: + batch_cancel = executions[0].get( + "_batch_cancel") or threading.Event() + for ex in executions: + ex["_batch_cancel"] = batch_cancel + pending = iter(enumerate(traces)) + + def submit_next() -> bool: + if not any(r.wants_more() for r in runs): + return False + item = next(pending, None) + if item is None: + return False + trace_ord, trace = item + futures[self._workers.submit(self._query_trace, runs, trace, + trace_ord, batch_cancel)] = item + return True + + for _ in range(self.max_workers): + if not submit_next(): + break + while futures: + done, _ = wait(list(futures), return_when=FIRST_COMPLETED) + for future in done: + trace_ord, trace = futures.pop(future) + try: + outcomes = future.result() + except Exception as e: # pylint: disable=broad-except + outcomes = [(None, e)] * len(runs) + for run, outcome in zip(runs, outcomes): + run.record(self._lock, trace_ord, trace, outcome) + submit_next() + if not any(r.wants_more() for r in runs): + batch_cancel.set() # Stops loads in flight. + except BaseException as e: # pylint: disable=broad-except + logger.exception("Trace query failed") + for run in runs: + run.fatal = run.fatal or f"Internal error: {e}" + run.ingest.cancel() + for ex in executions: + if not runs: + ex["errorMessage"] = str(e) + finally: + # Late feeds must not write after the tables are finalized. + for future in futures: + future.cancel() + wait(list(futures)) + for i, ex in enumerate(executions): + run = runs[i] if i < len(runs) else None + try: + self._finish_run(ex, run) + except Exception as e: # pylint: disable=broad-except + logger.exception("Finalizing query %s failed", + ex["queryUuid"]) + with self._lock: + if ex["status"] == "IN_PROGRESS": + ex["status"] = "FAILED" + ex["errorMessage"] = f"Internal error: {e}" + ex["endTime"] = iso_now() + self._persist(self.duckdb.save_execution, ex) + + def _finish_run(self, ex: Dict[str, Any], + run: Optional["_TraceRun"]) -> None: + """Waits for `run`'s inserts, fixes up its table and sets its status.""" + errors: List[str] = [] + table = None + if run is not None: + result = run.ingest.finish() + errors = list(run.errors) + for trace_ord in result.failed_ords: + run.succeeded.discard(trace_ord) + errors += result.errors + if run.fatal: + errors.append(run.fatal) + table = run.table + with self._lock: + cancelled = ex["status"] == "CANCELLED" + deleted = bool(ex.get("_deleted")) + succeeded = bool(run and run.succeeded) + if not cancelled: + status = "FAILED" if errors and not succeeded else "SUCCESS" + else: + status = "CANCELLED" + if errors and not cancelled: + ex["errorMessage"] = (f"{len(errors)} trace(s) failed: " + + "; ".join(errors[:3])) + elif not ex.get("errorMessage") and not run: + ex["errorMessage"] = "Query failed" + materialize = table is not None and not deleted and ( + status == "SUCCESS" or (status == "CANCELLED" and succeeded)) + rows = 0 + if table is not None and materialize: + assert run is not None + if run.partial: + if run.with_uuid: + self.duckdb.delete_trace_rows( + table, + uuids=[run.traces[o].trace_uuid for o in run.partial]) + else: + self.duckdb.delete_trace_rows(table, + ords=sorted(run.partial), + ord_column=_ORD_COLUMN) + if not run.with_uuid: + self.duckdb.drop_column(table, _ORD_COLUMN) + retype = {} + for col, mask in zip(run.columns, result.cell_types): + btype = stream_ingest.type_from_cell_types(mask) + if col in run.ingest.untyped and btype and btype != "STRING": + retype[col] = btype + if retype: + self.duckdb.retype_columns(table, retype) + with self._lock: + ex["_schema_map"].update(retype) + self._update_schema(ex, run.columns) + self.duckdb.finalize_streaming_table(table, + query_uuid=ex["queryUuid"], + sql=ex.get("perfettoSql")) + rows = self.duckdb.row_count(table) + elif table is not None: + self.duckdb.drop_table(table, ex["queryUuid"]) + with self._lock: + ex.pop("_ingest", None) + ex["processedRows"] = rows if materialize else ex["processedRows"] + ex["tableName"] = table if materialize else None + if ex["status"] != "CANCELLED": + ex["status"] = status + ex["endTime"] = iso_now() + if deleted: + self.duckdb.drop_table(table, ex["queryUuid"]) + return + self._persist(self.duckdb.save_execution, ex) + self._remember(ex) + + def _update_schema(self, ex: Dict[str, Any], + sql_columns: List[str]) -> None: + with self._lock: + schema_map = ex["_schema_map"] + visible = sql_columns + [ + c for c in ex["traceMetadataColumns"] if c not in sql_columns + ] + ex["schema"] = [{ + "name": c, + "type": schema_map.get(c, "STRING") + } for c in visible] + + def fetch_results( + self, + query_uuid: str, + offset: int = 0, + limit: int = 50, + order_by: Optional[str] = None, + filters: Optional[List[Dict[str, Any]]] = None, + columns: Optional[List[str]] = None, + ) -> Dict[str, Any]: + """Streams a filtered, sorted and projected page directly from DuckDB.""" + ex = self._load(query_uuid) + with self._lock: + table_name = ex.get("tableName") or "" + all_columns = list(ex.get("_columns") or []) + schema_map = dict(ex.get("_schema_map") or {}) + visible = [s["name"] for s in ex.get("schema") or []] + if not all_columns and visible: + all_columns = list(visible) + for s in ex.get("schema") or []: + schema_map.setdefault(s["name"], s.get("type", "STRING")) + + offset, limit = max(0, offset), max(0, limit) + out_indices = _column_indices(all_columns, columns or visible or + all_columns) + out_columns = [all_columns[i] for i in out_indices] + + total, page = self.duckdb.query_page( + table_name=table_name, + all_columns=all_columns, + out_indices=out_indices, + offset=offset, + limit=limit, + order_by=order_by, + filters=filters, + ) + + return { + "queryUuid": query_uuid, + "tableName": table_name, + "schema": [{ + "name": c, + "type": schema_map.get(c, "STRING") + } for c in out_columns], + "rows": [{ + "values": [to_wire(v) for v in r] + } for r in page], + "totalFilteredRows": total, + "availableColumnNames": all_columns, + } + + +def _column_indices(all_columns: List[str], wanted: List[str]) -> List[int]: + used: set = set() + out = [] + for name in wanted: + idx = next((i for i, c in enumerate(all_columns) + if c == name and i not in used), None) + if idx is not None: + used.add(idx) + out.append(idx) + return out + + +_Outcome = Tuple[Optional[FeedResult], Optional[BaseException]] + + +class _TraceRun: + """State of one query over traces while it runs.""" + + def __init__(self, ex: Dict[str, Any], traces: List[TraceEntry]): + self.ex = ex + self.traces = traces + self.ingest: StreamIngest + self.errors: List[str] = [] + self.succeeded: Set[int] = set() + # Traces that failed after queueing rows, which must be deleted. + self.partial: Set[int] = set() + # The row limit was reached. + self.stopped = False + self.fatal: Optional[str] = None + # Set once the result table exists. + self.table: Optional[str] = None + self.columns: List[str] = [] + self.with_uuid = True + + def wants_more(self) -> bool: + return not self.stopped and not self.ex["_cancel"].is_set() + + def record(self, lock: Any, trace_ord: int, trace: TraceEntry, + outcome: _Outcome) -> None: + res, err = outcome + with lock: + ex = self.ex + ex["processedTraces"] += 1 + cancelled = ex["_cancel"].is_set() + if err is not None: + if not cancelled: + logger.info("Query failed on %s: %s", trace.file_name, err) + self.errors.append(f"{trace.file_name}: {err}") + elif res is not None: + if res.ok: + self.succeeded.add(trace_ord) + ex["processedRows"] += res.rows + if res.status == stream_ingest.FEED_LIMIT: + self.stopped = True + else: + if res.rows: + self.partial.add(trace_ord) + if not (res.status == stream_ingest.FEED_CANCELLED and + cancelled): + msg = res.message or "cancelled" + logger.info("Query failed on %s: %s", trace.file_name, + msg) + self.errors.append(f"{trace.file_name}: {msg}") + if 0 < ex["limit"] <= ex["processedRows"]: + self.stopped = True
diff --git a/python/perfetto/bigtrace/server/filters.py b/python/perfetto/bigtrace/server/filters.py new file mode 100644 index 0000000..bed934c --- /dev/null +++ b/python/perfetto/bigtrace/server/filters.py
@@ -0,0 +1,131 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""DataGrid-style filtering and AIP-132 sorting of records.""" + +import fnmatch +from typing import Any, Callable, Dict, List, Optional, Tuple + +from perfetto.bigtrace.server.util import as_bool, as_int, get_param + + +def is_null(value: Any) -> bool: + return value is None or value == "" or (isinstance(value, str) and + value.upper() == "NULL") + + +def _equals(value: Any, target: Any) -> bool: + if str(value).lower() == str(target).lower(): + return True + try: + return float(value) == float(target) + except (TypeError, ValueError): + return False + + +def _compare(value: Any, target: Any) -> int: + try: + a, b = float(value), float(target) + except (TypeError, ValueError): + a, b = str(value), str(target) + return (a > b) - (a < b) + + +_COMPARISONS = { + "<": lambda c: c < 0, + "<=": lambda c: c <= 0, + ">": lambda c: c > 0, + ">=": lambda c: c >= 0, +} + +_FILTER_OPS = { + "=", "!=", "<", "<=", ">", ">=", "glob", "not glob", "in", "not in", + "is null", "is not null" +} + + +def evaluate_filter_op(value: Any, op: str, target: Any) -> bool: + """Evaluates one DataGrid filter operator against a value.""" + op = op.lower() + if op not in _FILTER_OPS: + raise ValueError(f"Unsupported filter op: {op}") + if op == "is null": + return is_null(value) + if op == "is not null": + return not is_null(value) + if value is None: + return False + if op == "=": + return _equals(value, target) + if op == "!=": + return not _equals(value, target) + if op in _COMPARISONS: + return _COMPARISONS[op](_compare(value, target)) + if op in ("glob", "not glob"): + pattern = "" if target is None else str(target) + return fnmatch.fnmatchcase(str(value), pattern) == (op == "glob") + targets = target if isinstance(target, (list, tuple)) else [target] + return any(_equals(value, t) for t in targets) == (op == "in") + + +def evaluate_filters(get_value: Callable[[str], Any], + filters: Optional[List[Dict[str, Any]]]) -> bool: + return all( + evaluate_filter_op( + get_value(f["field"]), f.get("op", "="), f.get("value")) + for f in filters or [] + if f.get("field")) + + +def parse_order_by(order_by: Optional[str]) -> List[Tuple[str, bool]]: + """Parses an AIP-132 order_by string into [(field, descending)].""" + clauses = [] + for part in (order_by or "").split(","): + tokens = part.split() + if tokens: + clauses.append((tokens[0], len(tokens) > 1 and + tokens[1].lower() == "desc")) + return clauses + + +def sort_records(records: List[Any], get_value: Callable[[Any, str], Any], + clauses: List[Tuple[str, bool]]) -> List[Any]: + """Sorts numbers before strings before nulls, per clause, stably.""" + + def key(value: Any) -> Tuple[int, float, str]: + if is_null(value): + return (2, 0.0, "") + try: + return (0, float(value), "") + except (TypeError, ValueError): + return (1, 0.0, str(value).lower()) + + result = list(records) + for field, descending in reversed(clauses): + result.sort(key=lambda r: key(get_value(r, field)), reverse=descending) + return result + + +def parse_experiment_filter(raw: Any) -> Optional[Dict[str, Any]]: + """Normalizes an experiment filter; None unless both ids are present.""" + if not isinstance(raw, dict): + return None + experiment_id = as_int(get_param(raw, "experiment_id"), None) + control_id = as_int(get_param(raw, "control_id"), None) + if experiment_id is None or control_id is None: + return None + return { + "experimentId": experiment_id, + "controlId": control_id, + "isTreatment": as_bool(get_param(raw, "is_treatment", True)), + }
diff --git a/python/perfetto/bigtrace/server/http_handler.py b/python/perfetto/bigtrace/server/http_handler.py new file mode 100644 index 0000000..1ea36f8 --- /dev/null +++ b/python/perfetto/bigtrace/server/http_handler.py
@@ -0,0 +1,417 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""HTTP request handling and routing for the Bigtrace UI API.""" + +import http.server +import json +import logging +import os +import re +import shutil +from typing import Any, Dict, Optional, Tuple +import urllib.parse + +from perfetto.bigtrace.server.catalog import ( + DEFAULT_PRESETS, + SettingsOverrides, + TraceCatalog, +) +from perfetto.bigtrace.server.executor import ( + BigtraceQueryExecutor, + QueryFailedError, +) +from perfetto.bigtrace.server.util import as_bool, as_int, get_param, to_wire + +logger = logging.getLogger(__name__) + +# Origins that may read responses cross-origin, in addition to any localhost +# origin. Traces can hold private data, so arbitrary websites must not. +DEFAULT_CORS_ORIGINS = ("https://ui.perfetto.dev",) + + +class NotFoundError(Exception): + pass + + +def experiment_to_wire(e: Dict[str, Any]) -> Dict[str, Any]: + return { + "experimentId": + as_int(get_param(e, "experiment_id"), 0), + "experimentName": + str(get_param(e, "experiment_name", "")), + "controlId": + as_int(get_param(e, "control_id"), 0), + "controlName": + str(get_param(e, "control_name", "")), + "isExperimentDenied": + as_bool(get_param(e, "is_experiment_denied", False)), + "isControlDenied": + as_bool(get_param(e, "is_control_denied", False)), + } + + +class BigtraceHandler(http.server.BaseHTTPRequestHandler): + """Serves the Bigtrace REST API; see ROUTES for the endpoints.""" + + executor: BigtraceQueryExecutor + catalog: TraceCatalog + cors_origins: Tuple[str, ...] = DEFAULT_CORS_ORIGINS + configured_host: Optional[str] = None + + def _host_allowed(self) -> bool: + raw_host = self.headers.get("Host") + if not raw_host: + return False + try: + parsed = urllib.parse.urlsplit(f"//{raw_host}") + hostname = (parsed.hostname or "").lower() + except (ValueError, AttributeError): + return False + if not hostname: + return False + + allowed = {"localhost", "127.0.0.1", "::1", "[::1]", "0.0.0.0"} + server = getattr(self, "server", None) + if server: + server_name = getattr(server, "server_name", None) + if server_name: + allowed.add(str(server_name).lower()) + server_address = getattr(server, "server_address", None) + if server_address and isinstance(server_address, + tuple) and server_address: + allowed.add(str(server_address[0]).lower()) + configured_host = getattr(server, "configured_host", None) + if configured_host: + allowed.add(str(configured_host).lower()) + handler_host = getattr(self, "configured_host", None) + if handler_host: + allowed.add(str(handler_host).lower()) + + return hostname in allowed + + def do_OPTIONS(self): + if not self._host_allowed(): + self._send_error(403, "Invalid Host header") + return + self.send_response(200) + self._send_cors_headers() + self.send_header("Content-Length", "0") + self.end_headers() + + def do_HEAD(self): + self.do_OPTIONS() + + def do_GET(self): + self._dispatch("GET") + + def do_POST(self): + self._dispatch("POST") + + def do_DELETE(self): + self._dispatch("DELETE") + + def log_message(self, fmt: str, *args: Any): + logger.debug("%s %s", self.address_string(), fmt % args) + + def _origin_allowed(self, origin: str) -> bool: + if "*" in self.cors_origins or origin in self.cors_origins: + return True + return urllib.parse.urlparse(origin).hostname in ("localhost", "127.0.0.1", + "::1") + + def _send_cors_headers(self): + origin = self.headers.get("Origin") + if origin is None: + # Not a cross-origin browser request (e.g. curl), so nothing to protect. + self.send_header("Access-Control-Allow-Origin", "*") + elif self._origin_allowed(origin): + # The UI sends credentials, which browsers only allow with this header. + self.send_header("Access-Control-Allow-Origin", origin) + self.send_header("Access-Control-Allow-Credentials", "true") + self.send_header("Vary", "Origin") + # Chrome asks before a public site (ui.perfetto.dev) calls localhost. + if self.headers.get("Access-Control-Request-Private-Network") == "true": + self.send_header("Access-Control-Allow-Private-Network", "true") + self.send_header("Access-Control-Allow-Methods", "GET, POST, DELETE") + self.send_header("Access-Control-Allow-Headers", + "Content-Type, Authorization") + + def _send_json(self, status: int, data: Any): + body = json.dumps(data).encode("utf-8") + self.send_response(status) + self._send_cors_headers() + self.send_header("Content-Type", "application/json; charset=utf-8") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def _send_error(self, status: int, detail: str): + self._send_json(status, {"detail": detail}) + + def _read_json_body(self) -> Dict[str, Any]: + length = as_int(self.headers.get("Content-Length"), 0) + if length <= 0: + return {} + body = json.loads(self.rfile.read(length).decode("utf-8") or "{}") + if not isinstance(body, dict): + raise ValueError("Request body must be a JSON object") + return body + + def _normalize_path(self) -> str: + """Strips the query string, duplicate slashes and a /v1 prefix.""" + path = re.sub(r"/+", "/", urllib.parse.urlparse(self.path).path) + if path == "/v1" or path.startswith("/v1/"): + path = path[3:] + return path.rstrip("/") or "/" + + def _dispatch(self, method: str): + if not self._host_allowed(): + self._send_error(403, "Invalid Host header") + return + path = self._normalize_path() + try: + body = self._read_json_body() if method == "POST" else {} + for route_method, pattern, handler in ROUTES: + match = pattern.fullmatch(path) if route_method == method else None + if match: + return handler(self, body, *match.groups()) + raise NotFoundError(f"Route not found: {path}") + except NotFoundError as e: + self._send_error(404, str(e)) + except (BrokenPipeError, ConnectionResetError): + logger.debug("Client closed connection during %s %s", method, path) + except (ValueError, TypeError, AttributeError, QueryFailedError) as e: + logger.info("Bad request %s %s: %s", method, path, e) + self._send_error(400, str(e)) + except Exception as e: # pylint: disable=broad-except + logger.exception("Error handling %s %s", method, path) + self._send_error(500, str(e)) + + # Catalog endpoints ------------------------------------------------------- + + def _root(self, _body): + self._send_json( + 200, { + "status": "ok", + "service": "bigtrace", + "traceCount": len(self.catalog.get_traces()), + "traceDir": self.catalog.trace_dir, + "recursive": self.catalog.recursive, + }) + + def _refresh(self, _body): + self.catalog.reload() + self.executor.duckdb.sync_catalog(self.catalog) + self._send_json(200, { + "status": "ok", + "traceCount": len(self.catalog.get_traces()) + }) + + def _duckdb_tables(self, _body): + self._send_json( + 200, { + "tables": self.executor.duckdb.list_tables(), + "extensionLoaded": self.executor.duckdb.extension_loaded, + }) + + def _execution_config(self, _body): + manifest = self.catalog.manifest_path or "" + self._send_json( + 200, { + "setting": [ + { + "id": "trace_directory", + "name": "Trace directory", + "description": "Directory on the server holding traces", + "category": "TRACE_ADDRESS", + "plainString": { + "defaultValue": self.catalog.trace_dir + }, + }, + { + "id": "recursive", + "name": "Recursive scan", + "description": "Also scan subdirectories", + "category": "TRACE_ADDRESS", + "booleanOptions": { + "defaultValue": self.catalog.recursive + }, + }, + { + "id": "trace_manifest", + "name": "Trace manifest", + "description": "Path to a JSON or CSV manifest", + "category": "TRACE_ADDRESS", + "plainString": { + "defaultValue": manifest + }, + }, + { + "id": "trace_uuids", + "name": "Trace UUIDs", + "description": "Only these trace UUIDs or file names", + "category": "TRACE_ADDRESS", + "disabled": True, + "stringArray": { + "defaultValues": [] + }, + }, + ] + }) + + def _metadata_schema(self, body): + self.catalog.apply_overrides(SettingsOverrides.parse(body.get("settings"))) + self._send_json(200, {"columns": self.catalog.get_schema()}) + + def _metadata(self, body): + overrides = SettingsOverrides.parse(body.get("settings")) + self.catalog.apply_overrides(overrides) + traces, total = self.catalog.filter_and_sort( + filters=body.get("filters"), + order_by=body.get("order_by"), + experiment_filter=get_param(body, "experiment_filter"), + trace_uuids=overrides.trace_uuids, + limit=as_int(body.get("limit"), 50), + offset=as_int(body.get("offset"), 0), + ) + columns = body.get("columns") or [ + c["name"] + for c in self.catalog.get_schema() + if c.get("defaultVisible", True) + ] + self._send_json( + 200, { + "columnNames": columns, + "rows": [{ + "values": [to_wire(t.get_value(c)) for c in columns] + } for t in traces], + "totalFilteredRows": total, + }) + + def _presets(self, _body): + presets = self.catalog.get_presets() or DEFAULT_PRESETS + self._send_json(200, {"tracePresets": presets}) + + def _experiments(self, body): + query = str(body.get("search_query") or "").lower() + limit = max(0, as_int(body.get("limit"), 50)) + offset = max(0, as_int(body.get("offset"), 0)) + matches = [ + e for e in map(experiment_to_wire, self.catalog.get_experiments()) + if query in e["experimentName"].lower() or + query in e["controlName"].lower() + ] + self._send_json(200, {"experiments": matches[offset:offset + limit]}) + + def _experiment_metadata(self, body): + wanted = as_int(body.get("experiment_id"), None) + for exp in map(experiment_to_wire, self.catalog.get_experiments()): + if exp["experimentId"] == wanted: + return self._send_json(200, {"experiment": exp}) + self._send_json(200, {}) + + def _trace_file(self, _body, trace_uuid): + trace = next( + (t for t in self.catalog.get_traces() if t.trace_uuid == trace_uuid), + None) + if not trace or not os.path.isfile(trace.file_path): + raise NotFoundError("Trace file not found") + with open(trace.file_path, "rb") as f: + self.send_response(200) + self._send_cors_headers() + self.send_header("Content-Type", "application/octet-stream") + self.send_header("Content-Disposition", + f'attachment; filename="{trace.file_name}"') + self.send_header("Content-Length", str(os.fstat(f.fileno()).st_size)) + self.end_headers() + shutil.copyfileobj(f, self.wfile, 1 << 16) + + # Query endpoints --------------------------------------------------------- + + def _execute_sync(self, body): + self._send_json(200, self.executor.execute(body, is_async=False)) + + def _execute_async(self, body): + self._send_json(200, self.executor.execute(body, is_async=True)) + + def _list_executions(self, _body): + self._send_json(200, {"queryExecutions": self.executor.list_executions()}) + + def _get_execution(self, _body, query_uuid): + ex = self.executor.get_execution(query_uuid) + if ex is None: + raise NotFoundError(f"Query {query_uuid} not found") + self._send_json(200, ex) + + def _get_status(self, _body, query_uuid): + status = self.executor.get_status(query_uuid) + if status is None: + raise NotFoundError(f"Query {query_uuid} not found") + self._send_json(200, status) + + def _fetch_results(self, body, query_uuid): + try: + result = self.executor.fetch_results( + query_uuid, + offset=as_int(body.get("offset"), 0), + limit=as_int(body.get("limit"), 50), + order_by=body.get("order_by"), + filters=body.get("filters"), + columns=body.get("columns"), + ) + except KeyError: + raise NotFoundError(f"Query {query_uuid} not found") from None + self._send_json(200, result) + + def _cancel(self, _body, query_uuid): + self.executor.cancel_query(query_uuid) + self._send_json(200, {}) + + def _delete(self, _body, query_uuid): + self.executor.delete_execution(query_uuid) + self._send_json(200, {}) + + def _check_table_exists(self, body): + table_name = str(body.get("table_name", "")) + self._send_json(200, self.executor.check_table_exists(table_name)) + + +_UUID = r"([^/:]+)" +ROUTES = [ + (method, re.compile(pattern), handler) for method, pattern, handler in [ + ("GET", r"/", BigtraceHandler._root), + ("POST", r"/refresh", BigtraceHandler._refresh), + ("GET", r"/duckdb/tables", BigtraceHandler._duckdb_tables), + ("GET", r"/trace_presets", BigtraceHandler._presets), + ("GET", rf"/trace/{_UUID}/[^/]+", BigtraceHandler._trace_file), + ("GET", r"/query_executions", BigtraceHandler._list_executions), + ("GET", rf"/query_executions/{_UUID}", BigtraceHandler._get_execution), + ("GET", rf"/query_executions/{_UUID}:status", + BigtraceHandler._get_status), + ("POST", r"/bigtrace_execution_config", + BigtraceHandler._execution_config), + ("POST", r"/trace_metadata_schema", BigtraceHandler._metadata_schema), + ("POST", r"/trace_metadata", BigtraceHandler._metadata), + ("POST", r"/experiments", BigtraceHandler._experiments), + ("POST", r"/experiment_metadata", BigtraceHandler._experiment_metadata), + ("POST", r"/execute_bigtrace_query", BigtraceHandler._execute_sync), + ("POST", r"/execute_bigtrace_query_async", + BigtraceHandler._execute_async), + ("POST", rf"/query_executions/{_UUID}:fetch_results", + BigtraceHandler._fetch_results), + ("POST", rf"/query_executions/{_UUID}:cancel", BigtraceHandler._cancel), + ("POST", r"/check_table_exists", BigtraceHandler._check_table_exists), + ("DELETE", rf"/query_executions/{_UUID}", BigtraceHandler._delete), + ] +]
diff --git a/python/perfetto/bigtrace/server/manifest.py b/python/perfetto/bigtrace/server/manifest.py new file mode 100644 index 0000000..faeb378 --- /dev/null +++ b/python/perfetto/bigtrace/server/manifest.py
@@ -0,0 +1,269 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Generates a manifest of per-trace metadata extracted in parallel.""" + +from concurrent.futures import ThreadPoolExecutor, as_completed +import json +import logging +import os +from typing import Any, Dict, List, Optional + +from perfetto.trace_processor.api import ( + TraceProcessorConfig, + TraceProcessorException, +) +from perfetto.bigtrace.server.catalog import DEFAULT_PRESETS +from perfetto.bigtrace.server.trace_processor_pool import ( + close_quietly, + load_trace, +) +from perfetto.bigtrace.server.util import ( + MANIFEST_NAME, + column_schema, + default_worker_count, + find_trace_files, + generate_trace_uuid, + infer_column_type, + iso_time, + resolve_trace_processor_bin, +) + +logger = logging.getLogger(__name__) + +# Columns the manifest generator extracts, in display order. +MANIFEST_COLUMNS = [ + column_schema("file_name", "File Name", "STRING", True, "Trace file name"), + column_schema("trace_uuid", "Trace UUID", "STRING", True, + "Trace identifier"), + column_schema("duration_ms", "Duration (ms)", "FLOAT64", True, + "Trace duration in milliseconds"), + column_schema("slice_count", "Slice Count", "INT64", True, + "Number of slices"), + column_schema("android_build_fingerprint", "Android Build", "STRING", True, + "Android build fingerprint"), + column_schema("android_sdk_version", "Android SDK", "INT64", True, + "Android SDK version"), + column_schema("process_count", "Process Count", "INT64", False, + "Number of processes"), + column_schema("thread_count", "Thread Count", "INT64", False, + "Number of threads"), + column_schema("system_machine", "Architecture", "STRING", False, + "CPU architecture"), + column_schema("system_name", "OS Name", "STRING", False, + "Operating system"), + column_schema("file_size", "File Size (bytes)", "INT64", False, + "File size in bytes"), + column_schema("start_ts", "Start Timestamp (ns)", "INT64", False, + "Trace start timestamp"), + column_schema("end_ts", "End Timestamp (ns)", "INT64", False, + "Trace end timestamp"), + column_schema("file_path", "File Path", "STRING", False, + "Path to the trace"), + column_schema("modified_time", "Modified Time", "TIMESTAMP", False, + "File modification time"), +] + +# Rows of the trace `metadata` table copied into the manifest. +MANIFEST_METADATA_KEYS = { + "android_build_fingerprint", + "android_incremental_build", + "android_sdk_version", + "benchmark_name", + "story_name", + "system_machine", + "system_name", + "system_release", + "system_version", + "trace_type", +} + + +class BigtraceManifestGenerator: + """Builds a manifest by extracting metadata from every trace in parallel. + + Traces whose size and mtime match an entry of the existing manifest are + reused instead of being loaded again. + """ + + def __init__( + self, + trace_dir: str, + recursive: bool = False, + max_workers: Optional[int] = None, + bin_path: Optional[str] = None, + metadata_query: Optional[str] = None, + ): + self.trace_dir = os.path.abspath(trace_dir) + self.recursive = recursive + self.max_workers = max_workers or default_worker_count() + self.tp_config = TraceProcessorConfig( + bin_path=resolve_trace_processor_bin(bin_path)) + self.metadata_query = metadata_query + + def _extract(self, path: str) -> Dict[str, Any]: + """Returns the manifest record for one trace. Raises for non-traces.""" + st = os.stat(path) + tp = load_trace(path, self.tp_config) + try: + meta: Dict[str, Any] = {} + bounds = next( + iter(tp.query("SELECT start_ts, end_ts FROM trace_bounds")), None) + if bounds is not None and bounds.start_ts is not None: + meta["start_ts"] = bounds.start_ts + meta["end_ts"] = bounds.end_ts + meta["duration_ms"] = round((bounds.end_ts - bounds.start_ts) / 1e6, 3) + counts = next( + iter( + tp.query("SELECT (SELECT count(1) FROM process) AS p, " + "(SELECT count(1) FROM thread) AS t, " + "(SELECT count(1) FROM slice) AS s"))) + meta.update( + process_count=counts.p, thread_count=counts.t, slice_count=counts.s) + trace_uuid = None + for r in tp.query("SELECT name, int_value, str_value FROM metadata"): + value = r.int_value if r.int_value is not None else r.str_value + if value is None: + continue + if r.name == "trace_uuid": + trace_uuid = str(value) + elif r.name in MANIFEST_METADATA_KEYS: + meta[r.name] = value + if self.metadata_query: + try: + result = tp.query(self.metadata_query) + first = next(iter(result), None) + for col in result.column_names if first is not None else []: + if getattr(first, col) is not None: + meta[col] = getattr(first, col) + except TraceProcessorException as e: + logger.warning("Metadata query failed on %s: %s", path, e) + finally: + close_quietly(tp) + return { + "file_name": os.path.basename(path), + "file_path": os.path.relpath(path, self.trace_dir), + "trace_uuid": trace_uuid or generate_trace_uuid(path), + "file_size": st.st_size, + "modified_time": iso_time(st.st_mtime), + **meta, + } + + @staticmethod + def _build_schema(traces: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + keys = {k for t in traces for k in t} + schema = [c for c in MANIFEST_COLUMNS if c["name"] in keys] + known = {c["name"] for c in MANIFEST_COLUMNS} + for key in sorted(keys - known): + display_name = key.replace("_", " ").title() + col_type = infer_column_type(t.get(key) for t in traces) + schema.append( + column_schema(key, display_name, col_type, True, + f"Metadata: {display_name}")) + return schema + + @staticmethod + def _read_manifest(path: str) -> Dict[str, Any]: + if not os.path.isfile(path): + return {} + try: + with open(path, "r", encoding="utf-8") as f: + data = json.load(f) + return data if isinstance(data, dict) else {} + except (OSError, ValueError) as e: + logger.warning("Ignoring unreadable manifest %s: %s", path, e) + return {} + + def generate( + self, + output_path: Optional[str] = None, + existing_manifest_path: Optional[str] = None, + force: bool = False, + ) -> Dict[str, Any]: + """Builds the manifest, writes it to `output_path` if set, returns it.""" + previous = {} if force else self._read_manifest( + existing_manifest_path or output_path or + os.path.join(self.trace_dir, MANIFEST_NAME)) + cached = { + os.path.normpath(t.get("file_path") or t.get("path")): t + for t in previous.get("traces", []) + if isinstance(t, dict) and (t.get("file_path") or t.get("path")) + } + + traces: List[Dict[str, Any]] = [] + to_extract: List[str] = [] + for path in find_trace_files(self.trace_dir, self.recursive): + try: + st = os.stat(path) + except OSError as e: + logger.warning("Cannot stat %s: %s", path, e) + continue + if st.st_size == 0: + continue + old = cached.get(os.path.normpath(os.path.relpath(path, self.trace_dir))) + if old and old.get("file_size") == st.st_size and old.get( + "modified_time") in (None, iso_time(st.st_mtime)): + traces.append(old) + else: + to_extract.append(path) + + logger.info("%d trace(s) cached, %d to extract with %d workers", + len(traces), len(to_extract), self.max_workers) + with ThreadPoolExecutor(max_workers=self.max_workers) as pool: + futures = {pool.submit(self._extract, p): p for p in to_extract} + for done, future in enumerate(as_completed(futures), 1): + name = os.path.basename(futures[future]) + try: + traces.append(future.result()) + logger.info("[%d/%d] Extracted %s", done, len(to_extract), name) + except Exception as e: # pylint: disable=broad-except + logger.info("[%d/%d] Skipped %s: %s", done, len(to_extract), name, e) + + traces.sort(key=lambda t: t.get("file_path", "")) + manifest = { + "trace_dir": ".", + "schema": self._build_schema(traces), + "traces": traces, + "presets": previous.get("presets", DEFAULT_PRESETS), + "experiments": previous.get("experiments", []), + } + if output_path: + out = os.path.abspath(output_path) + os.makedirs(os.path.dirname(out), exist_ok=True) + # Write then rename so a running server never reads a partial file. + tmp = out + ".tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump(manifest, f, indent=2) + os.replace(tmp, out) + logger.info("Wrote %d traces to %s", len(traces), out) + return manifest + + +def generate_manifest(trace_dir: str, + output_path: Optional[str] = None, + existing_manifest_path: Optional[str] = None, + recursive: bool = False, + max_workers: Optional[int] = None, + bin_path: Optional[str] = None, + metadata_query: Optional[str] = None, + force: bool = False) -> Dict[str, Any]: + return BigtraceManifestGenerator( + trace_dir=trace_dir, + recursive=recursive, + max_workers=max_workers, + bin_path=bin_path, + metadata_query=metadata_query, + ).generate( + output_path=output_path, + existing_manifest_path=existing_manifest_path, + force=force)
diff --git a/python/perfetto/bigtrace/server/server.py b/python/perfetto/bigtrace/server/server.py new file mode 100644 index 0000000..100d657 --- /dev/null +++ b/python/perfetto/bigtrace/server/server.py
@@ -0,0 +1,128 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Wires the catalog, DuckDB engine, executor and HTTP server together.""" + +import dataclasses as dc +import http.server +import signal +import threading +import urllib.parse +from typing import Optional, Tuple + +from perfetto.trace_processor.api import TraceProcessorConfig +from perfetto.bigtrace.server.catalog import DEFAULT_UI_ORIGIN, TraceCatalog +from perfetto.bigtrace.server.duckdb_engine import DuckDBEngine +from perfetto.bigtrace.server.executor import BigtraceQueryExecutor +from perfetto.bigtrace.server.http_handler import ( + BigtraceHandler, + DEFAULT_CORS_ORIGINS, +) +from perfetto.bigtrace.server.util import ( + default_worker_count, + resolve_trace_processor_bin, +) + + +def _origin(url: str) -> str: + parts = urllib.parse.urlsplit(url) + return f"{parts.scheme}://{parts.netloc}" + + +@dc.dataclass +class BigtraceServerConfig: + trace_dir: Optional[str] = None + recursive: bool = False + # JSON manifest, CSV (.csv), or SQLite metadata.db (.db). + manifest_path: Optional[str] = None + host: str = "127.0.0.1" + port: int = 5001 + max_workers: Optional[int] = None + bin_path: Optional[str] = None + # Optional Parquet directory for exporting/loading DuckDB tables; None disables. + parquet_dir: Optional[str] = None + # Optional DuckDB database file path; None uses an ephemeral file. + duckdb_path: Optional[str] = None + # Origins allowed besides DEFAULT_CORS_ORIGINS and localhost; "*" for any. + cors_origins: Tuple[str, ...] = () + # Perfetto UI that trace links open in; its origin is allowed to fetch traces. + ui_origin: str = DEFAULT_UI_ORIGIN + + +class BigtraceServer: + + def __init__(self, config: Optional[BigtraceServerConfig] = None): + self.config = config or BigtraceServerConfig() + cfg = self.config + cfg.max_workers = cfg.max_workers or default_worker_count() + + handler = type( + "Handler", (BigtraceHandler,), { + "cors_origins": + DEFAULT_CORS_ORIGINS + tuple(cfg.cors_origins) + + (_origin(cfg.ui_origin),), + "configured_host": + cfg.host, + }) + self.httpd = http.server.ThreadingHTTPServer((cfg.host, cfg.port), handler) + self.httpd.configured_host = cfg.host + self.server_port = self.httpd.server_address[1] + link_host = "127.0.0.1" if cfg.host in ("", "0.0.0.0", "::") else cfg.host + + self.catalog = TraceCatalog( + trace_dir=cfg.trace_dir, + recursive=cfg.recursive, + manifest_path=cfg.manifest_path, + base_url=f"http://{link_host}:{self.server_port}", + ui_origin=cfg.ui_origin, + ) + bin_path = resolve_trace_processor_bin(cfg.bin_path) + self.duckdb = DuckDBEngine( + db_path=cfg.duckdb_path, parquet_dir=cfg.parquet_dir) + self.executor = BigtraceQueryExecutor( + catalog=self.catalog, + tp_config=TraceProcessorConfig(bin_path=bin_path), + max_workers=cfg.max_workers, + duckdb_engine=self.duckdb, + ) + handler.executor = self.executor + handler.catalog = self.catalog + self._thread: Optional[threading.Thread] = None + self._serving = False + self._stopped = False + + def start(self, background: bool = False) -> None: + self._serving = True + if background: + self._thread = threading.Thread( + target=self.httpd.serve_forever, daemon=True) + self._thread.start() + return + signal.signal(signal.SIGTERM, signal.default_int_handler) + try: + self.httpd.serve_forever() + except KeyboardInterrupt: + pass + finally: + self.stop() + + def stop(self) -> None: + if self._stopped: + return + self._stopped = True + if self._serving: + self.httpd.shutdown() + if self._thread: + self._thread.join() + self.httpd.server_close() + self.executor.close()
diff --git a/python/perfetto/bigtrace/server/stream_ingest.py b/python/perfetto/bigtrace/server/stream_ingest.py new file mode 100644 index 0000000..90fc154 --- /dev/null +++ b/python/perfetto/bigtrace/server/stream_ingest.py
@@ -0,0 +1,451 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Streams per-trace query results from trace processors into DuckDB. + +Python only orchestrates: for each trace it hands the extension the address of +a loaded trace_processor_shell. The extension (proto_ingest.cc) POSTs the +query, decodes the reply as it arrives and queues cells that an +`INSERT ... FROM pf_scan_tp_stream()` drains into the result table. + +A query's rows go through a sequence of segments, each one stream plus one +INSERT. Segments rotate every few seconds or traces so that partial results +become visible while the query runs. The first segment starts taking feeds +before the column types are known; its INSERT starts once the cells seen so +far settle them (see pf_tp_stream_wait_typed). +""" + +import ctypes +import dataclasses as dc +import logging +import os +import threading +import time +from typing import Callable, List, Optional, Sequence, Set + +logger = logging.getLogger(__name__) + +FEED_OK = 0 +FEED_SQL_ERROR = 1 +FEED_COLUMN_MISMATCH = 2 +FEED_IO_ERROR = 3 +FEED_CANCELLED = 4 +FEED_CLOSED = 5 +FEED_BAD_HANDLE = 6 +FEED_LIMIT = 7 + +_MSG_CAP = 4096 +_QUEUE_CHUNKS = 512 + +# CellsBatch.CellType values. +_CELL_VARINT = 2 +_CELL_FLOAT64 = 3 +_CELL_STRING = 4 +_CELL_BLOB = 5 + + +@dc.dataclass +class FeedResult: + status: int + rows: int = 0 + message: str = "" + + @property + def ok(self) -> bool: + """Whether all the trace's rows (up to the row limit) were queued.""" + return self.status in (FEED_OK, FEED_LIMIT) + + +def type_from_cell_types(mask: int) -> Optional[str]: + """The column type holding every cell type in `mask`; None if no cells.""" + if mask & ((1 << _CELL_STRING) | (1 << _CELL_BLOB)): + return "STRING" + if mask & (1 << _CELL_FLOAT64): + return "DOUBLE" + if mask & (1 << _CELL_VARINT): + return "INT64" + return None + + +class TpStreamApi: + """ctypes binding of the extension's `pf_tp_stream_*` C ABI. + + Uses the library DuckDB already loaded (dlopen returns the same handle for + the same file), so streams opened here are visible to `pf_scan_tp_stream`. + Calls release the GIL, so feeds block only their own thread. + """ + + def __init__(self, lib_path: str): + lib = ctypes.CDLL(lib_path) + i32, i64 = ctypes.c_int32, ctypes.c_int64 + c_char_pp = ctypes.POINTER(ctypes.c_char_p) + sigs = { + "pf_tp_stream_open": ([ctypes.c_char_p, c_char_pp, i32, i32, i64], i64), + "pf_tp_stream_feed": ([ + i64, ctypes.c_char_p, i32, i32, + ctypes.POINTER(i64), ctypes.c_char_p, i32 + ], i32), + "pf_tp_stream_wait_typed": ([i64, i64], i32), + "pf_tp_stream_columns": ([i64, ctypes.c_char_p, i32], i32), + "pf_tp_stream_cell_types": ([i64, + ctypes.POINTER(ctypes.c_uint32), + i32], None), + "pf_tp_stream_close": ([i64], None), + "pf_tp_stream_cancel": ([i64], None), + "pf_tp_stream_free": ([i64], None), + "pf_query_progress_reset": ([], None), + "pf_query_progress_get": ([ + ctypes.POINTER(i64), + ctypes.POINTER(i64), + ctypes.POINTER(i64), + ctypes.POINTER(i64), + ], i32), + } + for name, (args, res) in sigs.items(): + if hasattr(lib, name): + fn = getattr(lib, name) + fn.argtypes = args + fn.restype = res + self._lib = lib + + def open(self, + sql: str, + columns: Optional[Sequence[str]] = None, + row_limit: int = 0, + queue_chunks: int = _QUEUE_CHUNKS) -> int: + """Opens a stream; `columns` None adopts the first reply's columns.""" + cols = [c.encode("utf-8") for c in columns or []] + arr = (ctypes.c_char_p * max(1, len(cols)))(*cols) + return int( + self._lib.pf_tp_stream_open( + sql.encode("utf-8"), arr, -1 if columns is None else len(cols), + queue_chunks, row_limit)) + + def feed(self, + handle: int, + host: str, + port: int, + trace_ord: Optional[int] = None) -> FeedResult: + if trace_ord is None: + target = host + ord_val = port + if target.startswith("tcp:"): + target = target[4:] + h, p = target.rsplit(":", 1) + return self.feed(handle, h, int(p), ord_val) + rows = ctypes.c_int64(0) + msg = ctypes.create_string_buffer(_MSG_CAP) + status = self._lib.pf_tp_stream_feed(handle, + host.encode("utf-8"), port, trace_ord, + ctypes.byref(rows), msg, _MSG_CAP) + return FeedResult( + int(status), int(rows.value), msg.value.decode("utf-8", "replace")) + + def wait_typed(self, handle: int, timeout_ms: int) -> bool: + return self._lib.pf_tp_stream_wait_typed(handle, timeout_ms) == 1 + + def columns(self, handle: int) -> Optional[List[str]]: + size = 1 << 16 + while True: + buf = ctypes.create_string_buffer(size) + n = self._lib.pf_tp_stream_columns(handle, buf, size) + if n == -2: + size *= 4 + continue + if n < 0: + return None + names = buf.raw.split(b"\0")[:n] + return [c.decode("utf-8", "replace") for c in names] + + def cell_types(self, handle: int, n: int) -> List[int]: + """Per column, the bitmask (1 << CellType) of the cells seen.""" + out = (ctypes.c_uint32 * max(1, n))() + self._lib.pf_tp_stream_cell_types(handle, out, n) + return list(out)[:n] + + def close(self, handle: int) -> None: + self._lib.pf_tp_stream_close(handle) + + def cancel(self, handle: int) -> None: + self._lib.pf_tp_stream_cancel(handle) + + def free(self, handle: int) -> None: + self._lib.pf_tp_stream_free(handle) + + def reset_query_progress(self) -> None: + if hasattr(self._lib, "pf_query_progress_reset"): + self._lib.pf_query_progress_reset() + + def get_query_progress(self) -> int: + if not hasattr(self._lib, "pf_query_progress_get"): + return -1 + r = ctypes.c_int64(0) + ing = ctypes.c_int64(0) + done = ctypes.c_int64(0) + live = ctypes.c_int64(0) + return int( + self._lib.pf_query_progress_get( + ctypes.byref(r), + ctypes.byref(ing), + ctypes.byref(done), + ctypes.byref(live), + )) + + +class _Segment: + """One stream and the INSERT draining it.""" + + def __init__(self, handle: int): + self.handle = handle + self.fed = 0 + self.started_at = time.monotonic() + self.thread: Optional[threading.Thread] = None + self.error: Optional[Exception] = None + self.queued_rows = 0 + self.inserted_rows = 0 + # Traces whose rows all went to this segment. + self.ok_ords: List[int] = [] + + +@dc.dataclass +class FinishResult: + errors: List[str] + # Traces whose rows were lost with a failed INSERT. + failed_ords: Set[int] + # Per column, the cell types seen over the whole query. + cell_types: List[int] + + def __iter__(self): + return iter(self.errors) + + def __len__(self): + return len(self.errors) + + def __eq__(self, other): + if isinstance(other, list): + return self.errors == other + return super().__eq__(other) + + +# Called once the columns and types are known; creates the result table and +# returns insert(handle), which drains a stream into it and returns the rows +# inserted. +OnSchema = Callable[[List[str], List[str]], Callable[[int], int]] + + +class StreamIngest: + """Streams one query's per-trace results into one DuckDB table. + + feed() once per trace from any thread, then finish() (after cancel() to + stop early). Columns no cell typed before the INSERT started are STRING; + `types` and `untyped` say which. + """ + + def __init__(self, + api: TpStreamApi, + sql: str, + on_schema: OnSchema, + columns: Optional[Sequence[str]] = None, + row_limit: int = 0, + segment_traces: int = 256, + segment_secs: float = 10.0): + self.api = api + self.sql = sql + self.row_limit = row_limit + self._on_schema = on_schema + # A row limit spans the whole query, so it needs a single stream. + self.segment_traces = segment_traces if row_limit <= 0 else 1 << 62 + self.segment_secs = segment_secs if row_limit <= 0 else float("inf") + self.columns: Optional[List[str]] = list( + columns) if columns is not None else None + self.types: List[str] = [] + self.untyped: List[str] = [] + self.cell_types: List[int] = [] + self._insert: Optional[Callable[[int], int]] = None + self._lock = threading.Lock() + self._started = threading.Event() + self._force = threading.Event() + self._cancelled = False + self._finished = False + self._schema_error: Optional[Exception] = None + first = _Segment(api.open(sql, columns, row_limit)) + self._segments: List[_Segment] = [first] + self._current = first + self._starter = threading.Thread( + target=self._start_when_typed, + name="bigtrace-stream-start", + daemon=True) + self._starter.start() + + @property + def started(self) -> bool: + return self._started.is_set() + + def start(self, *args, **kwargs) -> None: + pass + + def feed(self, + host: str, + port: int, + trace_ord: Optional[int] = None) -> FeedResult: + """Ingests one trace's result; blocks until all its rows are queued.""" + if trace_ord is None: + target = host + ord_val = port + if target.startswith("tcp:"): + target = target[4:] + h, p = target.rsplit(":", 1) + return self.feed(h, int(p), ord_val) + while True: + with self._lock: + if self._cancelled or self._finished: + return FeedResult(FEED_CANCELLED) + seg = self._current + if self._started.is_set() and (seg.fed >= self.segment_traces or + time.monotonic() - seg.started_at + >= self.segment_secs): + self.api.close(seg.handle) + seg = self._new_segment_locked() + seg.fed += 1 + res = self.api.feed(seg.handle, host, port, trace_ord) + with self._lock: + seg.queued_rows += res.rows + if res.ok: + seg.ok_ords.append(trace_ord) + error = seg.error or self._schema_error + if res.status in (FEED_CLOSED, FEED_BAD_HANDLE) and error is None: + continue # Rotated away; feed the next segment. + if res.status in (FEED_CLOSED, FEED_BAD_HANDLE, FEED_CANCELLED) and error: + return FeedResult(FEED_IO_ERROR, res.rows, f"DuckDB insert failed: " + f"{error}") + return res + + def force_start(self) -> None: + """Starts inserting even if some columns have no typed cell yet.""" + self._force.set() + + def cancel(self) -> None: + """Stops all feeds and inserts promptly; rows inserted so far stay.""" + with self._lock: + self._cancelled = True + segs = list(self._segments) + self._force.set() + for seg in segs: + self.api.cancel(seg.handle) + + def finish(self) -> FinishResult: + """Ends all segments and waits for their inserts.""" + with self._lock: + self._finished = True + segs = list(self._segments) + for seg in segs: + self.api.close(seg.handle) + self._force.set() + self._starter.join() + with self._lock: + segs = list(self._segments) + errors: List[str] = [] + failed: Set[int] = set() + masks = [0] * len(self.columns or []) + if self._schema_error is not None: + errors.append(f"could not create the result table: " + f"{self._schema_error}") + for seg in segs: + self.api.close(seg.handle) + if seg.thread is not None: + seg.thread.join() + for i, m in enumerate(self.api.cell_types(seg.handle, len(masks))): + masks[i] |= m + self.api.free(seg.handle) + lost = seg.error is not None or (seg.thread is None and seg.queued_rows) + if not lost and seg.inserted_rows != seg.queued_rows and ( + not self._cancelled): + logger.error("Stream %d queued %d rows but inserted %d", seg.handle, + seg.queued_rows, seg.inserted_rows) + lost = True + if lost: + failed.update(seg.ok_ords) + if seg.error is not None: + errors.append(f"DuckDB insert failed: {seg.error}") + self.cell_types = masks + return FinishResult(errors, failed, masks) + + def _new_segment_locked(self) -> _Segment: + seg = _Segment(self.api.open(self.sql, self.columns, self.row_limit)) + self._segments.append(seg) + self._current = seg + self._start_insert(seg) + return seg + + def _start_insert(self, seg: _Segment) -> None: + seg.thread = threading.Thread( + target=self._run_insert, + args=(seg,), + name=f"bigtrace-insert-{seg.handle}", + daemon=True) + seg.thread.start() + + def _start_when_typed(self) -> None: + first = self._segments[0] + while True: + typed = self.api.wait_typed(first.handle, 200) + with self._lock: + if self._cancelled: + return + if not typed and not self._force.is_set(): + continue + columns = self.api.columns(first.handle) + if columns is None: + if self._force.is_set(): + return # No trace returned a result: there is no table. + continue + masks = self.api.cell_types(first.handle, len(columns)) + types = [type_from_cell_types(m) or "STRING" for m in masks] + try: + insert = self._on_schema(columns, types) + except Exception as e: # pylint: disable=broad-except + logger.warning("Could not create the result table: %s", e) + with self._lock: + self._schema_error = e + self.cancel() + return + with self._lock: + self.columns = columns + self.types = types + self.untyped = [ + c for c, m in zip(columns, masks) if type_from_cell_types(m) is None + ] + self._insert = insert + self._start_insert(first) + self._started.set() + return + + def _run_insert(self, seg: _Segment) -> None: + assert self._insert is not None + try: + rows = self._insert(seg.handle) + with self._lock: + seg.inserted_rows = rows + except Exception as e: # pylint: disable=broad-except + with self._lock: + cancelled = self._cancelled + seg.error = e + if not cancelled: + logger.warning("Stream insert failed: %s", e) + # Unblocks feeds waiting on a queue nobody drains anymore. + self.api.cancel(seg.handle) + + +def default_scan_threads() -> int: + """DuckDB threads per INSERT; the DuckDB sink bounds streaming throughput.""" + return min(64, max(4, (os.cpu_count() or 8) // 2))
diff --git a/python/perfetto/bigtrace/server/trace_processor_pool.py b/python/perfetto/bigtrace/server/trace_processor_pool.py new file mode 100644 index 0000000..859e989 --- /dev/null +++ b/python/perfetto/bigtrace/server/trace_processor_pool.py
@@ -0,0 +1,236 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Starts a trace_processor_shell per trace. + +The server talks to the shells only over HTTP (the DuckDB extension sends +the queries), so a shell here is just a process and its address. +""" + +import dataclasses as dc +import logging +import socket +import subprocess +import sys +import tempfile +import threading +import time +from typing import Any, List, Optional, Set + +from perfetto.trace_processor.api import ( + PLATFORM_DELEGATE, + TraceProcessor, + TraceProcessorConfig, + TraceProcessorException, +) +from perfetto.trace_processor.process_tree import ( + create_kill_on_close_job, + terminate_process_tree, +) + +logger = logging.getLogger(__name__) + +# Large traces can take minutes to parse. +_LOAD_TIMEOUT_SECS = 600 +_HOST = "127.0.0.1" + + +def load_trace( + trace_path: str, + tp_config: TraceProcessorConfig, + cancel_event: Optional[threading.Event] = None, +) -> TraceProcessor: + """Starts a TraceProcessor with the trace loaded.""" + config = dc.replace( + tp_config, + cancel_event=cancel_event, + extra_flags=list(tp_config.extra_flags or []) + [trace_path], + load_timeout=max(tp_config.load_timeout, _LOAD_TIMEOUT_SECS)) + return TraceProcessor(config=config) + + +def close_quietly(tp: Any) -> None: + try: + tp.close() + except Exception: # pylint: disable=broad-except + logger.debug("Error closing TraceProcessor", exc_info=True) + + +class ShellStartError(TraceProcessorException): + """A shell failed to start or load its trace.""" + + +class ShellCancelled(ShellStartError): + """Startup was cancelled.""" + + +class Shell: + """A running trace_processor_shell serving one trace over HTTP.""" + + def __init__(self, proc: subprocess.Popen, job_handle, port: int, on_close): + self.proc = proc + self._job_handle = job_handle + self.host = _HOST + self.port = port + self._on_close = on_close + self._lock = threading.Lock() + self._closed = False + + @property + def closed(self) -> bool: + return self._closed + + @property + def subprocess(self) -> Optional[subprocess.Popen]: + return None if self._closed else self.proc + + def close(self) -> None: + """Kills the shell; safe to call more than once, from any thread.""" + with self._lock: + if self._closed: + return + self._closed = True + terminate_process_tree(self.proc, self._job_handle) + self._on_close(self) + + +class TraceProcessorPool: + """Starts and stops a trace_processor_shell for each trace of a query.""" + + def __init__(self, tp_config: Optional[TraceProcessorConfig] = None): + self.tp_config = tp_config or TraceProcessorConfig() + self._lock = threading.Lock() + self._shells: Set[Shell] = set() + # Ports of live shells. A shell binds its port only once its trace is + # loaded, so the OS can hand the same free port to another shell in + # between; never reuse a port a live shell owns. + self._ports: Set[int] = set() + self._closed = False + self._bin: Optional[str] = None + + def acquire( + self, + trace_path: str, + cancel_event: Optional[threading.Event] = None, + ) -> Shell: + return self._start(trace_path, cancel_event) + + def release(self, shell: Shell) -> None: + shell.close() + + def shell_flags(self) -> List[str]: + """Flags of the shells this pool starts.""" + cfg = self.tp_config + return ([] if cfg.ingest_ftrace_in_raw else ["--no-ftrace-raw"]) + ( + ["--dev"] if cfg.enable_dev_features else []) + list(cfg.extra_flags or + []) + + def close(self) -> None: + """Kills every live shell.""" + with self._lock: + self._closed = True + shells = list(self._shells) + for shell in shells: + shell.close() + + def _shell_path(self) -> str: + with self._lock: + if self._bin is None: + self._bin = PLATFORM_DELEGATE().get_shell_path( + bin_path=self.tp_config.bin_path, + fetch_latest=self.tp_config.fetch_latest_trace_processor) + return self._bin + + def _reserve_port(self) -> int: + with self._lock: + while True: + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind((_HOST, 0)) + port = s.getsockname()[1] + if port not in self._ports: + self._ports.add(port) + return port + + def _forget(self, shell: Shell) -> None: + with self._lock: + self._shells.discard(shell) + self._ports.discard(shell.port) + + def _start(self, path: str, cancel_event: Optional[threading.Event]) -> Shell: + if self._closed: + raise ShellStartError("Trace processor pool is closed") + port = self._reserve_port() + args = [self._shell_path()] + if sys.platform == "win32" and not args[0].endswith(".exe"): + args = [sys.executable] + args + args += ["-D", "--http-port", str(port)] + self.shell_flags() + [path] + stderr = tempfile.TemporaryFile() + try: + proc = subprocess.Popen( + args, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=stderr, + start_new_session=sys.platform != "win32", + creationflags=(subprocess.CREATE_NEW_PROCESS_GROUP + if sys.platform == "win32" else 0)) + except OSError as e: + stderr.close() + with self._lock: + self._ports.discard(port) + raise ShellStartError(f"Could not start trace processor: {e}") from e + shell = Shell(proc, create_kill_on_close_job(proc), port, self._forget) + with self._lock: + self._shells.add(shell) + closed = self._closed + try: + if closed: + raise ShellStartError("Trace processor pool is closed") + self._wait_ready(shell, cancel_event, stderr) + return shell + except BaseException: + shell.close() + raise + finally: + stderr.close() + + def _wait_ready(self, shell: Shell, cancel_event: Optional[threading.Event], + stderr) -> None: + """Waits until the shell accepts connections, i.e. its trace is loaded. + + Polls with a local TCP connect, which costs microseconds, backing off to + 100 ms for long parses. + """ + deadline = time.monotonic() + max(self.tp_config.load_timeout, + _LOAD_TIMEOUT_SECS) + delay = 0.005 + while True: + if shell.proc.poll() is not None: + stderr.seek(0) + lines = stderr.read().decode("utf-8", "replace").strip().splitlines() + raise ShellStartError( + lines[-1] if lines else f"trace processor exited with " + f"{shell.proc.returncode}") + if shell.closed: + raise ShellCancelled("Trace processor was stopped") + if cancel_event is not None and cancel_event.is_set(): + raise ShellCancelled("Trace processor startup cancelled") + try: + with socket.create_connection((_HOST, shell.port), timeout=1): + return + except OSError: + pass + if time.monotonic() > deadline: + raise ShellStartError("Timed out loading the trace") + time.sleep(delay) + delay = min(delay * 1.5, 0.1)
diff --git a/python/perfetto/bigtrace/server/util.py b/python/perfetto/bigtrace/server/util.py new file mode 100644 index 0000000..9acb022 --- /dev/null +++ b/python/perfetto/bigtrace/server/util.py
@@ -0,0 +1,196 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Helpers shared by the Bigtrace server modules.""" + +from datetime import datetime, timezone +import hashlib +import logging +import os +from typing import Any, Dict, Iterable, List, Optional +import uuid + +logger = logging.getLogger(__name__) + +MANIFEST_NAME = "bigtrace_manifest.json" +# Trace index written by trace_fetch, loaded like a manifest. +SQLITE_MANIFEST_NAME = "metadata.db" + +IGNORED_DIRS = { + ".git", + ".hg", + ".svn", + ".venv", + "venv", + "__pycache__", + "node_modules", + "buildtools", + "out", +} + + +def column_schema(name: str, display_name: str, col_type: str, + default_visible: bool, description: str) -> Dict[str, Any]: + return { + "name": name, + "displayName": display_name, + "type": col_type, + "defaultVisible": default_visible, + "description": description, + } + + +def iso_now() -> str: + return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def iso_time(epoch_secs: float) -> str: + return datetime.fromtimestamp( + epoch_secs, tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def generate_trace_uuid(path: str) -> str: + """Returns a UUID derived from the absolute path, stable across restarts.""" + digest = hashlib.md5(os.path.abspath(path).encode("utf-8")).hexdigest() + return str(uuid.UUID(digest)) + + +def is_manifest_file(file_name: str) -> bool: + base = os.path.basename(file_name).lower() + return base in ("manifest.json", SQLITE_MANIFEST_NAME) or base.endswith( + ("_manifest.json", "_manifest.csv")) + + +def as_int(value: Any, default: Optional[int]) -> Optional[int]: + try: + return int(value) + except (TypeError, ValueError): + return default + + +def as_bool(value: Any) -> bool: + if isinstance(value, str): + return value.strip().lower() in ("1", "true", "yes") + return bool(value) + + +def get_param(body: Dict[str, Any], + snake_name: str, + default: Any = None) -> Any: + """Reads a request field sent either as snake_case or camelCase.""" + value = body.get(snake_name) + if value is None: + head, *rest = snake_name.split("_") + value = body.get(head + "".join(w.title() for w in rest)) + return default if value is None else value + + +def to_wire(value: Any) -> Optional[str]: + """Formats a cell the way the UI expects: strings, or null.""" + if value is None: + return None + if isinstance(value, bool): + return "true" if value else "false" + return str(value) + + +def infer_column_type(values: Iterable[Any]) -> str: + """Infers a Bigtrace column type from the first non-empty value.""" + sample = next((v for v in values if v not in (None, "", "NULL")), None) + if sample is None: + return "STRING" + if isinstance(sample, bool): + return "BOOL" + if isinstance(sample, int): + return "INT64" + if isinstance(sample, float): + return "FLOAT64" + if isinstance(sample, bytes): + return "BYTES" + text = str(sample) + if text.lower() in ("true", "false"): + return "BOOL" + for parse, col_type in ((int, "INT64"), (float, "FLOAT64")): + try: + parse(text) + return col_type + except ValueError: + pass + return "STRING" + + +def default_worker_count() -> int: + """A trace_processor_shell per core, using at most half of the memory. + + A loaded trace typically takes well under 1 GiB. + """ + cpus = os.cpu_count() or 4 + try: + ram_gib = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") / 2**30 + except (AttributeError, ValueError, OSError): + return min(32, cpus) + return max(1, min(cpus, int(ram_gib / 2))) + + +def resolve_trace_processor_bin( + bin_path: Optional[str] = None) -> Optional[str]: + """Finds trace_processor_shell: flag, then env vars, then local builds. + + Returns None to let TraceProcessor download a prebuilt binary. + """ + if bin_path: + return bin_path + repo_root = os.path.abspath( + os.path.join(os.path.dirname(__file__), "..", "..", "..", "..")) + candidates = [ + os.environ.get("SHELL_PATH"), + os.environ.get("TRACE_PROCESSOR_BIN"), + ] + [ + os.path.join(repo_root, "out", d, "trace_processor_shell") + for d in ("release", "linux", "default") + ] + for cand in candidates: + if cand and os.path.isfile(cand) and os.access(cand, os.X_OK): + return cand + return None + + +def find_trace_files(root: str, recursive: bool) -> List[str]: + """Lists candidate trace files. Anything may be a trace, whatever its name. + + Hidden files and manifests are skipped. A non-recursive scan of a directory + that only has subdirectories falls back to a recursive one. + """ + if not os.path.isdir(root): + logger.warning("Trace directory does not exist: %s", root) + return [] + + def is_candidate(path: str) -> bool: + name = os.path.basename(path) + return not name.startswith(".") and not is_manifest_file(name) + + def walk() -> Iterable[str]: + for dir_path, subdirs, files in os.walk(root): + subdirs[:] = [ + d for d in subdirs if d not in IGNORED_DIRS and not d.startswith(".") + ] + for f in files: + yield os.path.join(dir_path, f) + + if recursive: + return sorted(p for p in walk() if is_candidate(p)) + top_level = [ + p for p in (os.path.join(root, f) for f in os.listdir(root)) + if os.path.isfile(p) and is_candidate(p) + ] + return sorted(top_level or (p for p in walk() if is_candidate(p)))
diff --git a/python/perfetto/trace_processor/api.py b/python/perfetto/trace_processor/api.py index 83a43bc..4313b43 100644 --- a/python/perfetto/trace_processor/api.py +++ b/python/perfetto/trace_processor/api.py
@@ -13,6 +13,7 @@ # limitations under the License. import dataclasses as dc +import threading from typing import Any, Dict, List, Optional, Union from perfetto.common.exceptions import PerfettoException @@ -35,6 +36,20 @@ TraceReference = registry.TraceReference +# Building a ProtoFactory parses large descriptors in Python, so it is shared +# by all instances using the same platform delegate. Factories are read-only. +_proto_factories: Dict[type, ProtoFactory] = {} +_proto_factories_lock = threading.Lock() + + +def _get_proto_factory(platform_delegate: PlatformDelegate) -> ProtoFactory: + with _proto_factories_lock: + key = type(platform_delegate) + if key not in _proto_factories: + _proto_factories[key] = ProtoFactory(platform_delegate) + return _proto_factories[key] + + # Custom exception raised if any trace_processor functions return a # response with an error defined TraceProcessorException = PerfettoException @@ -112,6 +127,9 @@ # `remote` or `addr`. enable_sql_file_access: bool = False + # Optional event that cancels startup if set while loading. + cancel_event: Optional[threading.Event] = None + def __init__( self, bin_path: Optional[str] = None, @@ -125,6 +143,7 @@ add_sql_packages: Optional[List[Union[str, SqlPackage]]] = None, fetch_latest_trace_processor: bool = False, enable_sql_file_access: bool = False, + cancel_event: Optional[threading.Event] = None, ): self.bin_path = bin_path self.unique_port = unique_port @@ -137,6 +156,7 @@ self.add_sql_packages = add_sql_packages self.fetch_latest_trace_processor = fetch_latest_trace_processor self.enable_sql_file_access = enable_sql_file_access + self.cancel_event = cancel_event class TraceProcessor: @@ -201,7 +221,7 @@ self.config = config self.platform_delegate = PLATFORM_DELEGATE() - self.protos = ProtoFactory(self.platform_delegate) + self.protos = _get_proto_factory(self.platform_delegate) self.resolver_registry = config.resolver_registry or \ self.platform_delegate.default_resolver_registry() self._backend = self._create_backend(remote, addr)
diff --git a/python/perfetto/trace_processor/platform.py b/python/perfetto/trace_processor/platform.py index c24d1b1..fa8860a 100644 --- a/python/perfetto/trace_processor/platform.py +++ b/python/perfetto/trace_processor/platform.py
@@ -13,6 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. +import collections import contextlib import datetime from datetime import timezone @@ -20,6 +21,7 @@ import socket import stat import tempfile +import threading from typing import Optional, Tuple from urllib import request @@ -31,6 +33,12 @@ # URL to download script to run trace_processor SHELL_URL = 'https://get.perfetto.dev/trace_processor' +# Ports recently handed out by get_bind_addr(). The shell only binds its port +# once the trace is loaded, so until then the OS may hand the same free port to +# another instance, whose client would then talk to the wrong shell. +_recent_ports = collections.deque(maxlen=4096) +_recent_ports_lock = threading.Lock() + class PlatformDelegate: """Abstracts operations which can vary based on platform.""" @@ -88,12 +96,14 @@ if port: return 'localhost', port - free_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - free_socket.bind(('', 0)) - free_socket.listen(5) - port = free_socket.getsockname()[1] - free_socket.close() - return 'localhost', port + with _recent_ports_lock: + while True: + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as free_socket: + free_socket.bind(('', 0)) + port = free_socket.getsockname()[1] + if port not in _recent_ports: + _recent_ports.append(port) + return 'localhost', port def default_resolver_registry(self) -> ResolverRegistry: return ResolverRegistry(resolvers=[PathUriResolver])
diff --git a/python/perfetto/trace_processor/shell.py b/python/perfetto/trace_processor/shell.py index 2f0268e..c876ff9 100644 --- a/python/perfetto/trace_processor/shell.py +++ b/python/perfetto/trace_processor/shell.py
@@ -14,13 +14,14 @@ # limitations under the License. import os +import shutil import subprocess import sys import tempfile +import threading import time -import shutil from typing import List, Optional, Union -from urllib import request, error +from urllib import error, request from perfetto.common.exceptions import PerfettoException from perfetto.trace_processor.http import TraceProcessorHttp @@ -51,6 +52,7 @@ add_sql_packages: Optional[List[Union[str, 'SqlPackage']]] = None, fetch_latest_trace_processor: bool = False, enable_sql_file_access: bool = False, + cancel_event: Optional[threading.Event] = None, ): addr, port = platform_delegate.get_bind_addr( port=0 if unique_port else TP_PORT) @@ -109,24 +111,40 @@ # Set the child up so its whole process tree can be torn down later (see # process_tree). On POSIX, start_new_session puts it in its own session/ # process group; on Windows it is assigned to a Job Object right after. - p = subprocess.Popen( - tp_exec + args, - stdin=subprocess.DEVNULL, - stdout=temp_stdout, - stderr=None if verbose else temp_stderr, - creationflags=creationflags, - start_new_session=sys.platform != 'win32') + try: + p = subprocess.Popen( + tp_exec + args, + stdin=subprocess.DEVNULL, + stdout=temp_stdout, + stderr=None if verbose else temp_stderr, + creationflags=creationflags, + start_new_session=sys.platform != 'win32') + except OSError: + temp_stdout.close() + temp_stderr.close() + raise job_handle = create_kill_on_close_job(p) + cancelled = False success = False - for _ in range(load_timeout + 1): + deadline = time.monotonic() + load_timeout + poll_delay = 0.05 + while True: try: if p.poll() is None: - _ = request.urlretrieve(f'http://{url}/status') + # Not urlretrieve(), which leaves a temp file behind for every call. + request.urlopen(f'http://{url}/status', timeout=1).close() success = True break except (error.URLError, ConnectionError): - time.sleep(1) + if cancel_event and cancel_event.is_set(): + cancelled = True + break + if time.monotonic() > deadline: + break + # Exponential backoff so fast loads wake up quickly without busy-polling. + time.sleep(poll_delay) + poll_delay = min(poll_delay * 2, 1.0) if not success: terminate_process_tree(p, job_handle) @@ -136,6 +154,8 @@ stderr = temp_stderr.read().decode("utf-8") temp_stdout.close() temp_stderr.close() + if cancelled: + raise PerfettoException("Trace processor startup cancelled.") raise PerfettoException("Trace processor failed to start.\n" f"stdout: {stdout}\nstderr: {stderr}\n" "If this is a slow machine or network, try " @@ -166,6 +186,7 @@ config.add_sql_packages, config.fetch_latest_trace_processor, config.enable_sql_file_access, + config.cancel_event, ) super().__init__(url, protos)
diff --git a/python/run_tests.py b/python/run_tests.py index 194f689..e9192b4 100755 --- a/python/run_tests.py +++ b/python/run_tests.py
@@ -20,6 +20,13 @@ from test import api_integrationtest from test import bigtrace_api_integrationtest +from test import bigtrace_server_api_unittest +from test import bigtrace_server_archive_unittest +from test import bigtrace_server_duckdb_unittest +from test import bigtrace_server_execution_unittest +from test import bigtrace_server_manifest_unittest +from test import bigtrace_server_robustness_unittest +from test import bigtrace_server_stream_unittest from test import open_trace_in_ui_unittest from test import query_result_iterator_unittest from test import resolver_unittest @@ -58,6 +65,14 @@ suite.addTests(loader.loadTestsFromModule(api_integrationtest)) suite.addTests(loader.loadTestsFromModule(stdlib_unittest)) suite.addTests(loader.loadTestsFromModule(unix_unittest)) + suite.addTests(loader.loadTestsFromModule(bigtrace_server_api_unittest)) + suite.addTests(loader.loadTestsFromModule(bigtrace_server_archive_unittest)) + suite.addTests(loader.loadTestsFromModule(bigtrace_server_duckdb_unittest)) + suite.addTests(loader.loadTestsFromModule(bigtrace_server_execution_unittest)) + suite.addTests(loader.loadTestsFromModule(bigtrace_server_manifest_unittest)) + suite.addTests( + loader.loadTestsFromModule(bigtrace_server_robustness_unittest)) + suite.addTests(loader.loadTestsFromModule(bigtrace_server_stream_unittest)) if os.path.exists(os.environ["WORKER_PATH"]): suite.addTests(loader.loadTestsFromModule(bigtrace_api_integrationtest))
diff --git a/python/test/bigtrace_server_api_unittest.py b/python/test/bigtrace_server_api_unittest.py new file mode 100644 index 0000000..b5c759e --- /dev/null +++ b/python/test/bigtrace_server_api_unittest.py
@@ -0,0 +1,313 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""HTTP API tests for the Bigtrace server.""" + +import json +import time +import unittest + +from test.bigtrace_server_fixture import BigtraceServerTestCase, request_raw + +API24_FILTER = [{"field": "file_name", "op": "glob", "value": "*api24*"}] +TERMINAL_STATES = ("SUCCESS", "FAILED", "CANCELLED") + + +class BigtraceServerApiTest(BigtraceServerTestCase): + + def _metadata(self, **body): + status, data, _ = self._request("POST", "/trace_metadata", body) + self.assertEqual(status, 200) + return data + + def _wait_until_done(self, query_uuid: str, timeout_s: float = 5) -> str: + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + status, data, _ = self._request("GET", + f"/query_executions/{query_uuid}:status") + self.assertEqual(status, 200) + if data.get("status") in TERMINAL_STATES: + return data["status"] + time.sleep(0.1) + self.fail(f"Query {query_uuid} did not finish within {timeout_s}s") + + def test_options_returns_cors_headers(self): + status, _, headers = self._request("OPTIONS", "/v1/trace_metadata") + self.assertEqual(status, 200) + self.assertIn("Access-Control-Allow-Origin", headers) + self.assertIn("Access-Control-Allow-Methods", headers) + + def test_head_returns_empty_response_with_cors(self): + status, raw, headers = request_raw(self.base_url, "HEAD", "/") + self.assertEqual(status, 200) + self.assertEqual(raw, b"") + self.assertIn("Access-Control-Allow-Origin", headers) + self.assertEqual(headers.get("Content-Length"), "0") + + def test_cors_only_allows_trusted_origins(self): + + def cors_headers(origin): + _, _, headers = request_raw( + self.base_url, "GET", "/", headers={"Origin": origin}) + return (headers.get("Access-Control-Allow-Origin"), + headers.get("Access-Control-Allow-Credentials")) + + # The UI fetches with credentials, so allowed origins need both headers. + for origin in ("http://localhost:10000", "https://ui.perfetto.dev"): + self.assertEqual(cors_headers(origin), (origin, "true")) + self.assertEqual(cors_headers("https://evil.example.com"), (None, None)) + + def test_dns_rebinding_host_header_validation(self): + port = self.server.server_port + for allowed in ("localhost", "127.0.0.1", "[::1]", "0.0.0.0"): + status, _, _ = request_raw( + self.base_url, "GET", "/", headers={"Host": f"{allowed}:{port}"}) + self.assertEqual(status, 200) + + for forbidden in ("evil.example.com", f"evil.example.com:{port}", + "attacker.local"): + status, raw, _ = request_raw( + self.base_url, "GET", "/", headers={"Host": forbidden}) + self.assertEqual(status, 403) + data = json.loads(raw) + self.assertEqual(data.get("detail"), "Invalid Host header") + + def test_execution_config_lists_settings(self): + for path in ("/bigtrace_execution_config", "/v1/bigtrace_execution_config"): + with self.subTest(path=path): + status, data, _ = self._request("POST", path, {}) + self.assertEqual(status, 200) + ids = {s["id"] for s in data["setting"]} + self.assertLessEqual( + {"trace_directory", "recursive", "trace_manifest", "trace_uuids"}, + ids) + + def test_metadata_schema_includes_manifest_columns(self): + status, data, _ = self._request("POST", "/trace_metadata_schema", {}) + self.assertEqual(status, 200) + names = {c["name"] for c in data["columns"]} + self.assertLessEqual( + { + "link", "file_name", "file_size", "trace_uuid", "device_name", + "experiment_id" + }, names) + + def test_metadata_projects_requested_columns(self): + columns = ["file_name", "device_name", "tag"] + data = self._metadata(limit=10, offset=0, columns=columns) + self.assertIn("totalFilteredRows", data) + self.assertEqual(data["columnNames"], columns) + self.assertGreaterEqual(len(data["rows"]), 7) + self.assertEqual(len(data["rows"][0]["values"]), 3) + + def test_metadata_glob_filter(self): + data = self._metadata( + limit=10, + filters=[{ + "field": "file_name", + "op": "glob", + "value": "*cold*" + }]) + for row in data["rows"]: + self.assertIn("cold", " ".join(str(v) for v in row["values"])) + + def test_metadata_equality_filter(self): + data = self._metadata( + limit=10, + filters=[{ + "field": "device_name", + "op": "=", + "value": "Pixel 7" + }]) + self.assertEqual(data["totalFilteredRows"], 3) + + def test_metadata_in_filter(self): + data = self._metadata( + limit=10, + filters=[{ + "field": "device_name", + "op": "in", + "value": ["Pixel 6", "Pixel 8"] + }]) + self.assertEqual(data["totalFilteredRows"], 4) + + def test_metadata_boolean_filter_accepts_lowercase_string(self): + # The UI sends booleans as the string "true". + data = self._metadata( + limit=10, + filters=[{ + "field": "is_treatment", + "op": "=", + "value": "true" + }]) + self.assertGreater(data["totalFilteredRows"], 0) + + def test_metadata_order_by_desc(self): + data = self._metadata( + limit=10, order_by="file_size desc", columns=["file_name", "file_size"]) + sizes = [int(r["values"][1]) for r in data["rows"]] + self.assertEqual(sizes, sorted(sizes, reverse=True)) + + def test_metadata_experiment_filter(self): + data = self._metadata( + limit=10, + experiment_filter={ + "experiment_id": 101, + "control_id": 102, + "is_treatment": True, + }) + self.assertEqual(data["totalFilteredRows"], 2) + + def test_experiment_search_and_metadata(self): + status, data, _ = self._request("POST", "/experiments", + {"search_query": "PGO"}) + self.assertEqual(status, 200) + self.assertEqual([e["experimentId"] for e in data["experiments"]], [101]) + + status, data, _ = self._request("POST", "/experiment_metadata", + {"experiment_id": 101}) + self.assertEqual(status, 200) + self.assertEqual(data["experiment"]["experimentName"], + "Profile-Guided Optimization (PGO)") + + def test_trace_presets(self): + status, data, _ = self._request("GET", "/trace_presets") + self.assertEqual(status, 200) + self.assertIn("preset:app_startup_latency", + [p["id"] for p in data["tracePresets"]]) + + def test_check_table_exists(self): + status, data, _ = self._request("POST", "/check_table_exists", + {"table_name": "my_table"}) + self.assertEqual(status, 200) + self.assertEqual(data["exists"], False) + self.assertEqual(data["resolvedTableName"], "my_table") + + def test_link_opens_trace_in_perfetto_ui(self): + data = self._metadata(limit=1, columns=["link"]) + link = data["rows"][0]["values"][0] + prefix = "https://ui.perfetto.dev/#!/?url=" + self.assertTrue(link.startswith(prefix), link) + trace_url = link[len(prefix):].split("&")[0] + self.assertTrue(trace_url.startswith(self.base_url + "/trace/"), link) + self.assertNotIn("?", trace_url) + + status, raw, headers = request_raw( + self.base_url, + "GET", + trace_url[len(self.base_url):], + headers={"Origin": "https://ui.perfetto.dev"}) + self.assertEqual(status, 200) + self.assertGreater(len(raw), 1000) + self.assertEqual( + headers.get("Access-Control-Allow-Origin"), "https://ui.perfetto.dev") + + def test_unknown_trace_download_is_404(self): + status, _, _ = self._request("GET", "/trace/no-such-uuid/x.pftrace") + self.assertEqual(status, 404) + + def test_sync_query_returns_rows_with_metadata_columns(self): + status, data, _ = self._request( + "POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT count(1) AS slice_count FROM slice;", + "limit": 100, + "trace_filters": API24_FILTER, + "trace_metadata_columns": ["file_name", "device_name"], + }) + self.assertEqual(status, 200) + self.assertIn("queryUuid", data) + self.assertEqual(len(data["rows"]), 3) + self.assertLessEqual({"slice_count", "file_name", "device_name"}, + {s["name"] for s in data["schema"]}) + + def test_sync_query_failure_returns_400(self): + status, data, _ = self._request( + "POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT * FROM definitely_non_existent_table;", + "limit": 10, + }) + self.assertEqual(status, 400) + self.assertIn("detail", data) + + def test_async_query_polling_and_fetch_results(self): + status, data, _ = self._request( + "POST", "/execute_bigtrace_query_async", { + "perfetto_sql": + "SELECT name, dur FROM slice WHERE dur > 10000000 LIMIT 5;", + "limit": + 50, + "trace_filters": + API24_FILTER, + "trace_metadata_columns": ["file_name", "tag"], + }) + self.assertEqual(status, 200) + uid = data["queryUuid"] + self.assertTrue(uid) + self.assertEqual(self._wait_until_done(uid), "SUCCESS") + + status, data, _ = self._request("GET", f"/query_executions/{uid}") + self.assertEqual(status, 200) + self.assertEqual(data["status"], "SUCCESS") + self.assertGreater(data["processedRows"], 0) + + status, data, _ = self._request( + "POST", f"/query_executions/{uid}:fetch_results", { + "limit": 10, + "offset": 0, + "order_by": "dur desc", + "columns": ["name", "dur", "file_name"], + }) + self.assertEqual(status, 200) + self.assertIn("rows", data) + self.assertLessEqual({"name", "dur", "tag"}, + set(data["availableColumnNames"])) + self.assertEqual([s["name"] for s in data["schema"][:2]], ["name", "dur"]) + durs = [int(r["values"][1]) for r in data["rows"] if r["values"][1]] + self.assertEqual(durs, sorted(durs, reverse=True)) + + def test_query_history_listing_and_deletion(self): + _, data, _ = self._request("POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT 1 AS val;", + "limit": 1 + }) + uid = data["queryUuid"] + + status, data, _ = self._request("GET", "/query_executions") + self.assertEqual(status, 200) + self.assertIn(uid, [q["queryUuid"] for q in data["queryExecutions"]]) + + status, _, _ = self._request("DELETE", + f"/query_executions/{uid}?drop_table=true") + self.assertEqual(status, 200) + + status, data, _ = self._request("GET", f"/query_executions/{uid}") + self.assertEqual(status, 404) + self.assertIn("detail", data) + + def test_query_listing_omits_unset_fields(self): + status, _, _ = self._request("POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT 1 AS val;", + "limit": 10 + }) + self.assertEqual(status, 200) + status, data, _ = self._request("GET", "/query_executions") + self.assertEqual(status, 200) + latest = data["queryExecutions"][0] + self.assertIn("materialized", latest) + self.assertIn("schema", latest) + self.assertNotIn("error", latest) + self.assertNotIn("tableLink", latest) + + +if __name__ == "__main__": + unittest.main()
diff --git a/python/test/bigtrace_server_archive_unittest.py b/python/test/bigtrace_server_archive_unittest.py new file mode 100644 index 0000000..81fddca --- /dev/null +++ b/python/test/bigtrace_server_archive_unittest.py
@@ -0,0 +1,108 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Tests for serving zip and tar archives of traces.""" + +import io +import os +import tarfile +import tempfile +import unittest +import zipfile + +from perfetto.bigtrace.server import TraceCatalog +from perfetto.bigtrace.server.archive import extract_archive, is_archive +from test.bigtrace_server_fixture import COLD_TRACE, HOT_TRACE, trace_path + + +def write_tar(path: str, mode: str, files: dict) -> None: + with tarfile.open(path, mode) as tf: + for name, data in files.items(): + info = tarfile.TarInfo(name) + info.size = len(data) + tf.addfile(info, io.BytesIO(data)) + + +def write_zip(path: str, files: dict) -> None: + with zipfile.ZipFile(path, "w") as zf: + for name, data in files.items(): + zf.writestr(name, data) + + +class BigtraceServerArchiveTest(unittest.TestCase): + + def setUp(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.tmp_dir = tmp.name + self.cache_dir = os.path.join(self.tmp_dir, "cache") + + def _path(self, name: str) -> str: + return os.path.join(self.tmp_dir, name) + + def _extract(self, path: str) -> str: + return extract_archive(path, cache_dir=self.cache_dir) + + def test_formats_are_detected_by_content(self): + files = {"a/trace": b"data"} + # Misleading names: the format comes from the bytes, not the extension. + write_zip(self._path("zip.bin"), files) + write_tar(self._path("tar.bin"), "w", files) + write_tar(self._path("targz.zip"), "w:gz", files) + write_tar(self._path("tarxz"), "w:xz", files) + for name in ("zip.bin", "tar.bin", "targz.zip", "tarxz"): + with self.subTest(name): + self.assertTrue(is_archive(self._path(name))) + out = self._extract(self._path(name)) + with open(os.path.join(out, "a", "trace"), "rb") as f: + self.assertEqual(f.read(), b"data") + + def test_traces_and_directories_are_not_archives(self): + self.assertFalse(is_archive(trace_path(COLD_TRACE))) + self.assertFalse(is_archive(trace_path("android_anr.pftrace.gz"))) + self.assertFalse(is_archive(self.tmp_dir)) + self.assertFalse(is_archive(self._path("missing"))) + + def test_entries_escaping_the_output_are_skipped(self): + files = {"../evil": b"x", "/abs": b"x", "ok": b"ok"} + write_zip(self._path("t.zip"), files) + write_tar(self._path("t.tar"), "w", files) + for name in ("t.zip", "t.tar"): + with self.subTest(name): + out = self._extract(self._path(name)) + self.assertEqual(os.listdir(out), ["ok"]) + self.assertFalse(os.path.exists(os.path.join(self.cache_dir, "evil"))) + + def test_extraction_is_reused_until_the_archive_changes(self): + path = self._path("t.tar.gz") + write_tar(path, "w:gz", {"a": b"1"}) + first = self._extract(path) + self.assertEqual(self._extract(path), first) + + write_tar(path, "w:gz", {"a": b"1", "b": b"22"}) + second = self._extract(path) + self.assertNotEqual(second, first) + self.assertEqual(sorted(os.listdir(second)), ["a", "b"]) + + def test_catalog_serves_traces_from_extracted_archive(self): + path = self._path("traces.tgz") + with tarfile.open(path, "w:gz") as tf: + tf.add(trace_path(COLD_TRACE), f"run1/{COLD_TRACE}") + tf.add(trace_path(HOT_TRACE), f"run2/{HOT_TRACE}") + catalog = TraceCatalog(trace_dir=self._extract(path), recursive=True) + names = sorted(t.file_name for t in catalog.get_traces()) + self.assertEqual(names, [COLD_TRACE, HOT_TRACE]) + + +if __name__ == "__main__": + unittest.main()
diff --git a/python/test/bigtrace_server_duckdb_unittest.py b/python/test/bigtrace_server_duckdb_unittest.py new file mode 100644 index 0000000..2ebc257 --- /dev/null +++ b/python/test/bigtrace_server_duckdb_unittest.py
@@ -0,0 +1,1244 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""DuckDB mode tests: result tables, typing, cancellation and concurrency.""" + +import tempfile +import time + +from perfetto.trace_processor import TraceProcessor, TraceProcessorConfig +from test.bigtrace_server_fixture import ( + BigtraceServerTestCase, + COLD_TRACE, + base_url, + copy_trace, + request, + resolve_trace_processor_bin, + running_server, + trace_path, + wait_for, +) + +_COLD_ONLY = [{"field": "file_name", "op": "glob", "value": "*startup_cold*"}] +_HEAP_TRACE = "system-server-heap-graph.pftrace" +# Streams for hours unless interrupted. +_SLOW_DUCKDB_SQL = "SELECT sum(i % 7) FROM range(1000000000000) t(i)" + + +class BigtraceServerDuckdbTest(BigtraceServerTestCase): + + def _traces_query(self, sql, **extra): + status, data, _ = self._request("POST", "/execute_bigtrace_query", { + "perfetto_sql": sql, + "limit": 100000, + "trace_filters": _COLD_ONLY, + **extra + }) + self.assertEqual(status, 200, data) + return data + + def _duckdb_query(self, sql, limit=1000): + return self._request("POST", "/execute_bigtrace_query", { + "engine": "duckdb", + "perfetto_sql": sql, + "limit": limit + }) + + def _status(self, uid): + return self._request("GET", f"/query_executions/{uid}:status")[1] + + def _wait_runs_finished(self, timeout=10.0): + executor = self.server.executor + wait_for(lambda: not any(r.is_alive() for r in executor._runs), timeout) + + def test_result_table_is_queryable_as_soon_as_query_succeeds(self): + status, data, _ = self._request( + "POST", "/execute_bigtrace_query_async", { + "perfetto_sql": "SELECT ts, dur FROM slice LIMIT 50", + "limit": 1000, + "trace_filters": _COLD_ONLY, + }) + self.assertEqual(status, 200, data) + uid, table = data["queryUuid"], data["tableName"] + wait_for(lambda: self._status(uid)["status"] != "IN_PROGRESS") + final = self._status(uid) + self.assertEqual(final["status"], "SUCCESS") + status, res, _ = self._duckdb_query(f"SELECT count(*) AS n FROM {table}") + self.assertEqual(status, 200, res) + self.assertGreater(final["processedRows"], 0) + self.assertEqual(res["rows"][0]["values"], [str(final["processedRows"])]) + + def test_duckdb_query_adds_exactly_one_history_entry(self): + before = { + e["queryUuid"] + for e in self._request("GET", "/query_executions")[1]["queryExecutions"] + } + status, res, _ = self._duckdb_query("SELECT 42 AS answer") + self.assertEqual(status, 200, res) + after = self._request("GET", "/query_executions")[1]["queryExecutions"] + added = [e for e in after if e["queryUuid"] not in before] + self.assertEqual(len(added), 1) + self.assertEqual(added[0]["queryEngine"], "duckdb") + self.assertEqual(added[0]["queryUuid"], res["queryUuid"]) + + def test_values_are_typed_from_the_result_schema(self): + status, res, _ = self._duckdb_query(""" + SELECT 1::BIGINT AS i, 2.5::DECIMAL(10, 2) AS d, + TIMESTAMPTZ '2020-01-01 00:00:00+00' AS tz, [1, 2] AS l, + true AS b, NULL::INTEGER AS n, 'x' AS s""") + self.assertEqual(status, 200, res) + self.assertEqual([(c["name"], c["type"]) for c in res["schema"]], + [("i", "INT64"), ("d", "DOUBLE"), ("tz", "STRING"), + ("l", "STRING"), ("b", "BOOLEAN"), ("n", "INT64"), + ("s", "STRING")]) + values = res["rows"][0]["values"] + self.assertEqual(values[0], "1") + self.assertEqual(float(values[1]), 2.5) + self.assertTrue(values[2].startswith("2020-01-01")) + self.assertEqual(values[3], "[1, 2]") + self.assertIsNone(values[5]) + self.assertEqual(values[6], "x") + + def test_duckdb_error_fails_the_query(self): + status, res, _ = self._duckdb_query("SELECT * FROM no_such_table_xyz") + self.assertEqual(status, 400) + self.assertIn("no_such_table_xyz", res["detail"]) + + def test_cancel_interrupts_a_running_duckdb_query(self): + status, data, _ = self._request("POST", "/execute_bigtrace_query_async", { + "engine": "duckdb", + "perfetto_sql": _SLOW_DUCKDB_SQL, + "limit": 10 + }) + self.assertEqual(status, 200, data) + uid = data["queryUuid"] + self._request("POST", f"/query_executions/{uid}:cancel", {}) + self._wait_runs_finished() + self.assertEqual(self._status(uid)["status"], "CANCELLED") + + def test_trace_queries_finish_while_a_duckdb_query_runs(self): + _, data, _ = self._request("POST", "/execute_bigtrace_query_async", { + "engine": "duckdb", + "perfetto_sql": _SLOW_DUCKDB_SQL, + "limit": 10 + }) + slow_uid = data["queryUuid"] + try: + wait_for(lambda: "_duck_cursor" in self.server.executor._executions.get( + slow_uid, {})) + start = time.monotonic() + res = self._traces_query("SELECT 1 AS one") + self.assertEqual(self._status(res["queryUuid"])["status"], "SUCCESS") + self.assertEqual(self._request("GET", "/query_executions")[0], 200) + self.assertLess(time.monotonic() - start, 10) + finally: + self._request("POST", f"/query_executions/{slow_uid}:cancel", {}) + self._wait_runs_finished() + + def test_deleting_a_query_drops_its_table(self): + res = self._traces_query("SELECT 1 AS one", table_name="to_delete") + tables = lambda: { + t["name"] for t in self._request("GET", "/duckdb/tables")[1]["tables"] + } + self.assertIn("to_delete", tables()) + self._request("DELETE", f"/query_executions/{res['queryUuid']}") + self.assertNotIn("to_delete", tables()) + status, _, _ = self._duckdb_query("SELECT * FROM to_delete") + self.assertEqual(status, 400) + + def test_deleting_an_older_run_keeps_a_newer_table_of_the_same_name(self): + old = self._traces_query("SELECT 1 AS v", table_name="reused") + self._traces_query("SELECT 2 AS v", table_name="reused") + self._request("DELETE", f"/query_executions/{old['queryUuid']}") + status, res, _ = self._duckdb_query("SELECT DISTINCT v FROM reused") + self.assertEqual(status, 200, res) + self.assertEqual([r["values"] for r in res["rows"]], [["2"]]) + + def test_interval_flatten_over_a_slice_result_table(self): + if not self.server.executor.duckdb.extension_loaded: + self.skipTest("perfetto.duckdb_extension not available") + res = self._traces_query( + "SELECT id, ts, dur, depth, track_id FROM slice", + table_name="cold_slices") + total = res["totalFilteredRows"] + self.assertGreater(total, 0) + per = "per := ['_trace_uuid', 'track_id']" + rel = "'(SELECT * FROM cold_slices WHERE dur >= 0)'" + status, flat_sum, _ = self._duckdb_query( + f"SELECT count(*) AS n, max(count) AS max_count " + f"FROM pf_interval_flatten_sum({rel}, ['depth'], {per})") + self.assertEqual(status, 200, flat_sum) + n, max_count = (int(v) for v in flat_sum["rows"][0]["values"]) + self.assertGreater(n, 0) + self.assertGreaterEqual(max_count, 1) + status, flat, _ = self._duckdb_query( + f"SELECT count(*) AS n FROM pf_interval_flatten({rel}, " + f"'count(*) AS concurrency, max(depth) AS max_depth', {per})") + self.assertEqual(status, 200, flat) + self.assertEqual(int(flat["rows"][0]["values"][0]), n) + + +class BigtraceServerIntervalWorkflowsTest(BigtraceServerTestCase): + """End-to-end tests for interval workflows (A, B, C), TP comparison, and edge cases.""" + + def _duckdb_query(self, sql, limit=1000): + return self._request("POST", "/execute_bigtrace_query", { + "engine": "duckdb", + "perfetto_sql": sql, + "limit": limit + }) + + def test_workflow_a_slices_sched_intersection_and_aggregate(self): + if not self.server.executor.duckdb.extension_loaded: + self.skipTest("perfetto.duckdb_extension not available") + status, batch_res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "queries": [ + { + "perfetto_sql": ( + "SELECT s.id, s.ts, s.dur, s.depth, s.name, s.track_id, " + "tt.utid FROM slice s JOIN thread_track tt ON s.track_id = tt.id" + ), + "table_name": "slices" + }, + { + "perfetto_sql": ( + "SELECT id, ts, dur, cpu, utid FROM sched WHERE utid != 0" + ), + "table_name": "sched" + }, + ], + "limit": 100000, + "trace_filters": _COLD_ONLY, + }) + self.assertEqual(status, 200, batch_res) + self.assertEqual(len(batch_res["executions"]), 2) + self.assertEqual([e["tableName"] for e in batch_res["executions"]], + ["slices", "sched"]) + + duck_sql = """ + SELECT s.name, sum(dur) AS running_dur + FROM pf_interval_intersection( + ['(SELECT * FROM slices WHERE dur >= 0)', '(SELECT * FROM sched WHERE dur >= 0)'], + ['s', 'c'], + per := ['_trace_uuid', 'utid'], + full_row := true + ) + GROUP BY s.name + ORDER BY running_dur DESC + LIMIT 10 + """ + status, res, _ = self._duckdb_query(duck_sql) + self.assertEqual(status, 200, res) + self.assertGreater(len(res["rows"]), 0) + top_name = res["rows"][0]["values"][0] + top_dur = float(res["rows"][0]["values"][1]) + self.assertTrue(top_name) + self.assertGreater(top_dur, 0.0) + + # Struct columns projected directly returned to UI as JSON strings + struct_sql = """ + SELECT ts, dur, _trace_uuid, utid, s, c + FROM pf_interval_intersection( + ['(SELECT * FROM slices WHERE dur >= 0)', '(SELECT * FROM sched WHERE dur >= 0)'], + ['s', 'c'], + per := ['_trace_uuid', 'utid'], + full_row := true + ) + LIMIT 2 + """ + status, sres, _ = self._duckdb_query(struct_sql) + self.assertEqual(status, 200, sres) + self.assertGreater(len(sres["rows"]), 0) + s_val = sres["rows"][0]["values"][4] + c_val = sres["rows"][0]["values"][5] + self.assertTrue(s_val.startswith("{") and "id" in s_val and "name" in s_val) + self.assertTrue(c_val.startswith("{") and "id" in c_val and "cpu" in c_val) + + def test_workflow_b_self_interval_intersect_slices(self): + if not self.server.executor.duckdb.extension_loaded: + self.skipTest("perfetto.duckdb_extension not available") + status, res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "perfetto_sql": ( + "SELECT s.id, s.ts, s.dur, s.depth, s.name, s.track_id, tt.utid " + "FROM slice s JOIN thread_track tt ON s.track_id = tt.id"), + "table_name": "slices_b", + "limit": 100000, + "trace_filters": _COLD_ONLY, + }) + self.assertEqual(status, 200, res) + + # Self-intersect per track_id + status, res_track, _ = self._duckdb_query(""" + SELECT count(*) AS cnt, sum(dur) AS total_dur + FROM pf_interval_intersection( + ['(SELECT * FROM slices_b WHERE dur >= 0)', '(SELECT * FROM slices_b WHERE dur >= 0)'], + ['a', 'b'], + per := ['_trace_uuid', 'track_id'] + ) + """) + self.assertEqual(status, 200, res_track) + cnt_track = int(res_track["rows"][0]["values"][0]) + dur_track = float(res_track["rows"][0]["values"][1]) + self.assertGreater(cnt_track, 0) + self.assertGreater(dur_track, 0.0) + + # Self-intersect per utid + status, res_utid, _ = self._duckdb_query(""" + SELECT count(*) AS cnt, sum(dur) AS total_dur + FROM pf_interval_intersection( + ['(SELECT * FROM slices_b WHERE dur >= 0)', '(SELECT * FROM slices_b WHERE dur >= 0)'], + ['a', 'b'], + per := ['_trace_uuid', 'utid'] + ) + """) + self.assertEqual(status, 200, res_utid) + cnt_utid = int(res_utid["rows"][0]["values"][0]) + dur_utid = float(res_utid["rows"][0]["values"][1]) + self.assertGreater(cnt_utid, 0) + self.assertGreater(dur_utid, 0.0) + + def test_workflow_c_flatten_and_flatten_sum_slices(self): + if not self.server.executor.duckdb.extension_loaded: + self.skipTest("perfetto.duckdb_extension not available") + status, res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "perfetto_sql": ( + "SELECT s.id, s.ts, s.dur, s.depth, s.name, s.track_id, tt.utid " + "FROM slice s JOIN thread_track tt ON s.track_id = tt.id"), + "table_name": "slices_c", + "limit": 100000, + "trace_filters": _COLD_ONLY, + }) + self.assertEqual(status, 200, res) + + per = "per := ['_trace_uuid', 'track_id']" + rel = "'(SELECT * FROM slices_c WHERE dur >= 0)'" + status, flat_sum, _ = self._duckdb_query( + f"SELECT count(*) AS n, max(count) AS max_count " + f"FROM pf_interval_flatten_sum({rel}, ['depth'], {per})") + self.assertEqual(status, 200, flat_sum) + n, max_count = (int(v) for v in flat_sum["rows"][0]["values"]) + self.assertGreater(n, 0) + self.assertGreaterEqual(max_count, 1) + + status, flat, _ = self._duckdb_query( + f"SELECT count(*) AS n FROM pf_interval_flatten({rel}, " + f"'count(*) AS concurrency, max(depth) AS max_depth', {per})") + self.assertEqual(status, 200, flat) + self.assertEqual(int(flat["rows"][0]["values"][0]), n) + + def test_correctness_compare_trace_processor_and_duckdb_intersection(self): + if not self.server.executor.duckdb.extension_loaded: + self.skipTest("perfetto.duckdb_extension not available") + + # 1. Compute ground truth directly with TraceProcessor + tp = TraceProcessor( + trace=trace_path(COLD_TRACE), + config=TraceProcessorConfig(bin_path=resolve_trace_processor_bin())) + try: + tp.query("INCLUDE PERFETTO MODULE intervals.intersect;") + tp.query(""" + CREATE PERFETTO TABLE cmp_slices AS + SELECT s.id, s.ts, s.dur, tt.utid + FROM slice s JOIN thread_track tt ON s.track_id = tt.id + WHERE s.dur >= 0; + """) + tp.query(""" + CREATE PERFETTO TABLE cmp_sched AS + SELECT id, ts, dur, utid + FROM sched + WHERE utid != 0 AND dur >= 0; + """) + tp_res = list( + tp.query(""" + SELECT count(*) AS cnt, sum(dur) AS total_dur + FROM _interval_intersect!((cmp_slices, cmp_sched), (utid)); + """)) + expected_cnt = int(tp_res[0].cnt) + expected_dur = int(tp_res[0].total_dur) + finally: + tp.close() + + # 2. Compute via Bigtrace Server + DuckDB pf_interval_intersection + status, batch_res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "queries": [ + { + "perfetto_sql": ( + "SELECT s.id, s.ts, s.dur, s.depth, s.name, s.track_id, " + "tt.utid FROM slice s JOIN thread_track tt ON s.track_id = tt.id" + ), + "table_name": "slices_cmp" + }, + { + "perfetto_sql": ( + "SELECT id, ts, dur, cpu, utid FROM sched WHERE utid != 0" + ), + "table_name": "sched_cmp" + }, + ], + "limit": + 100000, + "trace_filters": [{ + "field": "file_name", + "op": "=", + "value": COLD_TRACE + }], + }) + self.assertEqual(status, 200, batch_res) + + status, duck_res, _ = self._duckdb_query(""" + SELECT count(*) AS cnt, sum(dur) AS total_dur + FROM pf_interval_intersection( + ['(SELECT * FROM slices_cmp WHERE dur >= 0)', '(SELECT * FROM sched_cmp WHERE dur >= 0)'], + ['s', 'c'], + per := ['_trace_uuid', 'utid'] + ); + """) + self.assertEqual(status, 200, duck_res) + actual_cnt = int(duck_res["rows"][0]["values"][0]) + actual_dur = int(float(duck_res["rows"][0]["values"][1])) + + self.assertEqual(actual_cnt, expected_cnt) + self.assertEqual(actual_dur, expected_dur) + + def test_multi_query_duplicate_table_names_disambiguated(self): + status, res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "queries": [ + { + "perfetto_sql": "SELECT 1 AS col1;", + "table_name": "dup_tbl" + }, + { + "perfetto_sql": "SELECT 2 AS col2;", + "table_name": "dup_tbl" + }, + ] + }) + self.assertEqual(status, 200, res) + names = [e["tableName"] for e in res["executions"]] + self.assertEqual(names, ["dup_tbl", "dup_tbl_2"]) + + tables = { + t["name"] for t in self._request("GET", "/duckdb/tables")[1]["tables"] + } + self.assertIn("dup_tbl", tables) + self.assertIn("dup_tbl_2", tables) + + s1, r1, _ = self._duckdb_query("SELECT col1 FROM dup_tbl") + self.assertEqual(s1, 200, r1) + self.assertEqual(r1["rows"][0]["values"][0], "1") + + s2, r2, _ = self._duckdb_query("SELECT col2 FROM dup_tbl_2") + self.assertEqual(s2, 200, r2) + self.assertEqual(r2["rows"][0]["values"][0], "2") + + def test_multi_query_one_fails_while_other_succeeds_sync_and_async(self): + # Sync: partial success returns 200 with successful query data + status, res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "queries": [ + { + "perfetto_sql": "SELECT * FROM nonexistent_table_xyz;", + "table_name": "bad_sync" + }, + { + "perfetto_sql": "SELECT 42 AS answer;", + "table_name": "good_sync" + }, + ] + }) + self.assertEqual(status, 200, res) + self.assertEqual(len(res["executions"]), 2) + self.assertEqual(res["rows"][0]["values"][0], "42") + bad_uid = res["executions"][0]["queryUuid"] + good_uid = res["executions"][1]["queryUuid"] + self.assertEqual( + self._request("GET", + f"/query_executions/{bad_uid}:status")[1]["status"], + "FAILED") + self.assertEqual( + self._request("GET", + f"/query_executions/{good_uid}:status")[1]["status"], + "SUCCESS") + + # Async: both tracked, one FAILED and one SUCCESS + status, ares, _ = self._request( + "POST", "/execute_bigtrace_query_async", { + "queries": [ + { + "perfetto_sql": "SELECT * FROM nonexistent_table_xyz;", + "table_name": "bad_async" + }, + { + "perfetto_sql": "SELECT 100 AS answer;", + "table_name": "good_async" + }, + ] + }) + self.assertEqual(status, 200, ares) + bad_async_uid = ares["executions"][0]["queryUuid"] + good_async_uid = ares["executions"][1]["queryUuid"] + wait_for(lambda: self._request( + "GET", f"/query_executions/{good_async_uid}:status")[1]["status"] != + "IN_PROGRESS") + self.assertEqual( + self._request("GET", + f"/query_executions/{bad_async_uid}:status")[1]["status"], + "FAILED") + self.assertEqual( + self._request( + "GET", f"/query_executions/{good_async_uid}:status")[1]["status"], + "SUCCESS") + + def test_column_name_collisions_like_id(self): + status, res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "perfetto_sql": + ("SELECT s.id, tt.id FROM slice s " + "JOIN thread_track tt ON s.track_id = tt.id LIMIT 5;"), + "table_name": "colliding_ids" + }) + self.assertEqual(status, 200, res) + row0 = res["rows"][0]["values"] + self.assertNotEqual(row0[0], row0[1]) + + status, dres, _ = self._duckdb_query( + "SELECT id, id_1 FROM colliding_ids LIMIT 5") + self.assertEqual(status, 200, dres) + drow0 = dres["rows"][0]["values"] + self.assertEqual(drow0[0], row0[0]) + self.assertEqual(drow0[1], row0[1]) + + def test_operators_work_on_result_tables_and_inline_sql(self): + if not self.server.executor.duckdb.extension_loaded: + self.skipTest("perfetto.duckdb_extension not available") + status, batch_res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "queries": [ + { + "perfetto_sql": ( + "SELECT s.id, s.parent_id, s.ts, s.dur, s.depth, " + "s.name, s.track_id, tt.utid " + "FROM slice s JOIN thread_track tt ON s.track_id = tt.id " + "WHERE s.dur >= 0 LIMIT 200"), + "table_name": "slices_clean", + }, + { + "perfetto_sql": ("SELECT id, ts, dur, cpu, utid FROM sched " + "WHERE utid != 0 AND dur >= 0 LIMIT 200"), + "table_name": "sched_clean", + }, + ], + "limit": 10000, + "trace_filters": _COLD_ONLY, + }) + self.assertEqual(status, 200, batch_res) + + # 1. Interval flatten: direct table name vs inline SQL (no parens) vs inline SQL (parens), + # and string agg vs list agg. + s_tbl, r_tbl, _ = self._duckdb_query( + "SELECT count(*) FROM pf_interval_flatten(" + "'slices_clean', 'count(*) AS n', per := ['_trace_uuid', 'track_id'])") + s_lst, r_lst, _ = self._duckdb_query( + "SELECT count(*) FROM pf_interval_flatten(" + "'slices_clean', ['count(*) AS n'], per := ['_trace_uuid', 'track_id'])" + ) + s_inl, r_inl, _ = self._duckdb_query( + "SELECT count(*) FROM pf_interval_flatten(" + "'SELECT * FROM slices_clean WHERE dur >= 0;', ['count(*) AS n'], " + "per := ['_trace_uuid', 'track_id'])") + self.assertEqual(s_tbl, 200, r_tbl) + self.assertEqual(s_lst, 200, r_lst) + self.assertEqual(s_inl, 200, r_inl) + self.assertGreater(int(r_tbl["rows"][0]["values"][0]), 0) + self.assertEqual(r_tbl["rows"], r_lst["rows"]) + self.assertEqual(r_tbl["rows"], r_inl["rows"]) + + # 2. Interval intersection: direct table names vs inline SQL without parens. + s_int_tbl, r_int_tbl, _ = self._duckdb_query( + "SELECT count(*) FROM pf_interval_intersection(" + "['slices_clean', 'sched_clean'], ['s', 'c'], " + "per := ['_trace_uuid', 'utid'])") + s_int_inl, r_int_inl, _ = self._duckdb_query( + "SELECT count(*) FROM pf_interval_intersection(" + "['SELECT * FROM slices_clean', 'SELECT * FROM sched_clean'], " + "['s', 'c'], per := ['_trace_uuid', 'utid'])") + self.assertEqual(s_int_tbl, 200, r_int_tbl) + self.assertEqual(s_int_inl, 200, r_int_inl) + self.assertEqual(r_int_tbl["rows"], r_int_inl["rows"]) + + # 3. Tree operator: direct table name vs inline SQL, list agg vs string agg. + # Null out parent_id when the parent row wasn't inside the LIMIT 200 sample. + tree_sql = ( + "SELECT s._trace_uuid, s.id, " + "CASE WHEN p.id IS NOT NULL THEN s.parent_id ELSE NULL END AS parent_id, " + "s.dur FROM slices_clean s " + "LEFT JOIN slices_clean p ON s._trace_uuid = p._trace_uuid AND s.parent_id = p.id" + ) + s_tree, r_tree, _ = self._duckdb_query( + f"SELECT count(*), sum(total) FROM pf_tree_accumulate(" + f"'{tree_sql}', 'up', 'sum(dur) AS total', per := '_trace_uuid')") + self.assertEqual(s_tree, 200, r_tree) + self.assertGreater(int(r_tree["rows"][0]["values"][0]), 0) + + # 4. Graph operator: inline SQL on both nodes, edges, and seeds. + s_graph, r_graph, _ = self._duckdb_query( + "SELECT count(*) FROM pf_graph_dfs_tree(" + "'SELECT _trace_uuid, id, name FROM slices_clean', " + "'SELECT _trace_uuid, parent_id AS src, id AS dst FROM slices_clean WHERE parent_id IS NOT NULL', " + "'SELECT _trace_uuid, id FROM slices_clean WHERE parent_id IS NULL', " + "per := '_trace_uuid')") + self.assertEqual(s_graph, 200, r_graph) + self.assertGreater(int(r_graph["rows"][0]["values"][0]), 0) + + def test_failed_query_does_not_claim_table_and_single_delete_preserves_siblings( + self): + # A failed query ("select * sched" syntax error) must not create or claim a table. + s_fail, r_fail, _ = self._request("POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT * sched", + "table_name": "sched_never_written", + }) + self.assertEqual(s_fail, 400, r_fail) + + s_chk, r_chk, _ = self._request("POST", "/check_table_exists", { + "table_name": "sched_never_written", + }) + self.assertEqual(s_chk, 200, r_chk) + self.assertFalse(r_chk["exists"]) + + # Create two real queries, delete one, and verify only that one disappears. + _, q1, _ = self._request("POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT 1 AS a", + "table_name": "hist_keep", + }) + _, q2, _ = self._request("POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT 2 AS b", + "table_name": "hist_drop", + }) + s_del, _, _ = self._request( + "DELETE", f"/query_executions/{q2['queryUuid']}?drop_table=true") + self.assertEqual(s_del, 200) + + _, hist, _ = self._request("GET", "/query_executions") + uuids = {e["queryUuid"] for e in hist["queryExecutions"]} + self.assertNotIn(q2["queryUuid"], uuids) + self.assertIn(q1["queryUuid"], uuids) + + def test_3_way_interval_intersection_slices_sched_thread_state_against_tp( + self): + if not self.server.executor.duckdb.extension_loaded: + self.skipTest("perfetto.duckdb_extension not available") + + # 1. Ground truth from TraceProcessor (3-way _interval_intersect!) + tp = TraceProcessor( + trace=trace_path(COLD_TRACE), + config=TraceProcessorConfig(bin_path=resolve_trace_processor_bin())) + try: + tp.query("INCLUDE PERFETTO MODULE intervals.intersect;") + tp.query(""" + CREATE PERFETTO TABLE cmp_slices AS + SELECT s.id, s.ts, s.dur, tt.utid + FROM slice s JOIN thread_track tt ON s.track_id = tt.id + WHERE s.dur >= 0; + """) + tp.query(""" + CREATE PERFETTO TABLE cmp_sched AS + SELECT id, ts, dur, utid + FROM sched + WHERE utid != 0 AND dur >= 0; + """) + tp.query(""" + CREATE PERFETTO TABLE cmp_ts AS + SELECT id, ts, dur, utid + FROM thread_state + WHERE dur >= 0; + """) + tp_res = list( + tp.query(""" + SELECT count(*) AS cnt, sum(dur) AS total_dur + FROM _interval_intersect!((cmp_slices, cmp_sched, cmp_ts), (utid)); + """)) + expected_cnt = int(tp_res[0].cnt) + expected_dur = int(tp_res[0].total_dur) + finally: + tp.close() + + # 2. Single-pass 3-query extraction via Bigtrace Server + status, batch_res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "queries": [ + { + "perfetto_sql": ( + "SELECT s.id, s.ts, s.dur, s.name, tt.utid " + "FROM slice s JOIN thread_track tt ON s.track_id = tt.id" + ), + "table_name": "slices_3w", + }, + { + "perfetto_sql": ( + "SELECT id, ts, dur, cpu, utid FROM sched WHERE utid != 0" + ), + "table_name": "sched_3w", + }, + { + "perfetto_sql": + ("SELECT id, ts, dur, state, utid FROM thread_state"), + "table_name": "thread_state_3w", + }, + ], + "limit": + 100000, + "trace_filters": [{ + "field": "file_name", + "op": "=", + "value": COLD_TRACE + }], + }) + self.assertEqual(status, 200, batch_res) + self.assertEqual(len(batch_res["executions"]), 3) + + # 3. 3-way pf_interval_intersection in DuckDB mode + status, duck_res, _ = self._duckdb_query(""" + SELECT count(*) AS cnt, sum(dur) AS total_dur + FROM pf_interval_intersection( + [ + '(SELECT * FROM slices_3w WHERE dur >= 0)', + '(SELECT * FROM sched_3w WHERE dur >= 0)', + '(SELECT * FROM thread_state_3w WHERE dur >= 0)' + ], + ['s', 'c', 'st'], + per := ['_trace_uuid', 'utid'], + full_row := true + ); + """) + self.assertEqual(status, 200, duck_res) + actual_cnt = int(duck_res["rows"][0]["values"][0]) + actual_dur = int(float(duck_res["rows"][0]["values"][1])) + self.assertEqual(actual_cnt, expected_cnt) + self.assertEqual(actual_dur, expected_dur) + + def test_slice_tree_accumulate_and_keep_if_against_trace_processor(self): + if not self.server.executor.duckdb.extension_loaded: + self.skipTest("perfetto.duckdb_extension not available") + + # 1. Ground truth from TraceProcessor + tp = TraceProcessor( + trace=trace_path(COLD_TRACE), + config=TraceProcessorConfig(bin_path=resolve_trace_processor_bin())) + try: + tp_roots = { + int(r.id): int(r.dur) for r in tp.query( + "SELECT id, dur FROM slice WHERE dur >= 0 AND parent_id IS NULL") + } + tp_desc = list( + tp.query(""" + SELECT count(*) AS cnt, coalesce(sum(dur), 0) AS total_dur + FROM slice + WHERE dur >= 0 AND parent_id IS NOT NULL + """))[0] + expected_desc_cnt = int(tp_desc.cnt) + expected_desc_dur = int(tp_desc.total_dur) + + # Exact subtree sum via recursive CTE in SQLite + tp_subtree = list( + tp.query(""" + WITH RECURSIVE sub(root_id, id, dur) AS ( + SELECT id, id, dur FROM slice WHERE dur >= 0 + UNION ALL + SELECT sub.root_id, c.id, c.dur + FROM sub JOIN slice c ON c.parent_id = sub.id + WHERE c.dur >= 0 + ), + acc AS ( + SELECT root_id AS id, sum(dur) AS total FROM sub GROUP BY root_id + ) + SELECT count(*) AS cnt, sum(total) AS grand_total FROM acc + """))[0] + expected_acc_cnt = int(tp_subtree.cnt) + expected_acc_sum = int(tp_subtree.grand_total) + finally: + tp.close() + + # 2. Extract slice table via Bigtrace Server + status, res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "perfetto_sql": ("SELECT id, parent_id, ts, dur, depth, name " + "FROM slice WHERE dur >= 0"), + "table_name": + "slice_tree_tbl", + "limit": + 100000, + "trace_filters": [{ + "field": "file_name", + "op": "=", + "value": COLD_TRACE + }], + }) + self.assertEqual(status, 200, res) + self.assertGreater(len(tp_roots), 0) + + # 3. Verify pf_tree_accumulate_up against TraceProcessor recursive sum + status, acc_res, _ = self._duckdb_query(""" + SELECT count(*) AS cnt, sum(total) AS grand_total + FROM pf_tree_accumulate_up('slice_tree_tbl', 'dur', per := ['_trace_uuid']) + """) + self.assertEqual(status, 200, acc_res) + self.assertEqual(int(acc_res["rows"][0]["values"][0]), expected_acc_cnt) + self.assertEqual( + int(float(acc_res["rows"][0]["values"][1])), expected_acc_sum) + + # 4. Verify pf_tree_keep_if('descendant' of root slices) == all non-root slices + status, keep_res, _ = self._duckdb_query(""" + SELECT count(*) AS cnt, coalesce(sum(dur), 0) AS total_dur + FROM pf_tree_keep_if( + 'slice_tree_tbl', + 'descendant', + '(SELECT _trace_uuid, id FROM slice_tree_tbl WHERE parent_id IS NULL)', + per := ['_trace_uuid'] + ) + """) + self.assertEqual(status, 200, keep_res) + self.assertEqual(int(keep_res["rows"][0]["values"][0]), expected_desc_cnt) + self.assertEqual( + int(float(keep_res["rows"][0]["values"][1])), expected_desc_dur) + + def test_e2e_heap_graph_dominator_retained_size_and_bfs_against_trace_processor( + self): + if not self.server.executor.duckdb.extension_loaded: + self.skipTest("perfetto.duckdb_extension not available") + + # 1. Ground truth from TraceProcessor's android.memory.heap_graph.dominator_tree + tp = TraceProcessor( + trace=trace_path(_HEAP_TRACE), + config=TraceProcessorConfig(bin_path=resolve_trace_processor_bin())) + try: + tp_summary = list( + tp.query(""" + INCLUDE PERFETTO MODULE android.memory.heap_graph.dominator_tree; + SELECT + count(*) AS cnt, + sum(dominated_size_bytes) AS sum_retained, + max(dominated_size_bytes) AS max_retained + FROM heap_graph_dominator_tree; + """))[0] + expected_cnt = int(tp_summary.cnt) + expected_sum = int(tp_summary.sum_retained) + expected_max = int(tp_summary.max_retained) + tp_top10 = [(int(r.id), int(r.dominated_size_bytes)) for r in tp.query(""" + SELECT id, dominated_size_bytes + FROM heap_graph_dominator_tree + ORDER BY dominated_size_bytes DESC, id ASC + LIMIT 10 + """)] + finally: + tp.close() + + # 2. Single-pass 3-query extraction over a temporary server with the heap trace + with tempfile.TemporaryDirectory() as tmp: + copy_trace(_HEAP_TRACE, tmp, _HEAP_TRACE) + with running_server(trace_dir=tmp, max_workers=2) as srv: + url = base_url(srv) + s_ext, ext_res, _ = request( + url, "POST", "/execute_bigtrace_query_async", { + "queries": [ + { + "perfetto_sql": ( + "SELECT o.id, o.upid, o.graph_sample_ts, o.self_size, " + "c.name AS class_name " + "FROM heap_graph_object o " + "JOIN heap_graph_class c ON o.type_id = c.id " + "WHERE o.reachable != 0"), + "table_name": "heap_objects", + "limit": 0, + }, + { + "perfetto_sql": ( + "INCLUDE PERFETTO MODULE android.memory.heap_graph.excluded_refs; " + "SELECT o.upid, o.graph_sample_ts, " + "r.owner_id AS src, r.owned_id AS dst " + "FROM heap_graph_reference r " + "JOIN heap_graph_object o ON r.owner_id = o.id " + "WHERE o.reachable != 0 AND r.owned_id IS NOT NULL " + "AND r.id NOT IN _excluded_refs"), + "table_name": "heap_refs", + "limit": 0, + }, + { + "perfetto_sql": ( + "SELECT id, upid, graph_sample_ts " + "FROM heap_graph_object WHERE root_type IS NOT NULL" + ), + "table_name": "gc_roots", + "limit": 0, + }, + ], + "limit": 0, + }) + self.assertEqual(s_ext, 200, ext_res) + self.assertEqual(len(ext_res["executions"]), 3) + uids = [e["queryUuid"] for e in ext_res["executions"]] + wait_for( + lambda: all( + request(url, "GET", f"/query_executions/{u}:status")[1][ + "status"] != "IN_PROGRESS" for u in uids), + timeout=30.0) + for u in uids: + st = request(url, "GET", f"/query_executions/{u}:status")[1] + self.assertEqual(st["status"], "SUCCESS", st) + + # 3. Run pf_graph_dominator_tree + pf_tree_accumulate_up in DuckDB mode + s_dom, dom_res, _ = request( + url, "POST", "/execute_bigtrace_query", { + "engine": + "duckdb", + "perfetto_sql": + """ + WITH dom AS ( + SELECT * + FROM pf_graph_dominator_tree( + 'heap_objects', + 'heap_refs', + 'gc_roots', + per := ['_trace_uuid', 'upid', 'graph_sample_ts'], + full_row := true + ) + ), + ret AS ( + SELECT id, total AS dominated_size_bytes + FROM pf_tree_accumulate_up( + 'dom', + 'self_size', + per := ['_trace_uuid', 'upid', 'graph_sample_ts'] + ) + ) + SELECT + count(*) AS cnt, + sum(dominated_size_bytes) AS sum_retained, + max(dominated_size_bytes) AS max_retained + FROM ret + """, + "limit": + 10, + }) + self.assertEqual(s_dom, 200, dom_res) + vals = dom_res["rows"][0]["values"] + self.assertEqual(int(vals[0]), expected_cnt) + self.assertEqual(int(float(vals[1])), expected_sum) + self.assertEqual(int(float(vals[2])), expected_max) + + # Verify top-10 retained objects match TraceProcessor row-for-row + s_top, top_res, _ = request( + url, "POST", "/execute_bigtrace_query", { + "engine": + "duckdb", + "perfetto_sql": + """ + SELECT id, total AS dominated_size_bytes + FROM pf_tree_accumulate_up( + '(SELECT * FROM pf_graph_dominator_tree(''heap_objects'', ''heap_refs'', ''gc_roots'', per := [''_trace_uuid'', ''upid'', ''graph_sample_ts''], full_row := true))', + 'self_size', + per := ['_trace_uuid', 'upid', 'graph_sample_ts'] + ) + ORDER BY dominated_size_bytes DESC, id ASC + LIMIT 10 + """, + "limit": + 10, + }) + self.assertEqual(s_top, 200, top_res) + actual_top10 = [(int(r["values"][0]), int(float(r["values"][1]))) + for r in top_res["rows"]] + self.assertEqual(actual_top10, tp_top10) + + # 4. Run pf_graph_bfs_tree + pf_tree_flatten_recursion + s_bfs, bfs_res, _ = request( + url, "POST", "/execute_bigtrace_query", { + "engine": + "duckdb", + "perfetto_sql": + """ + CREATE OR REPLACE TEMP VIEW bfs_tree AS + SELECT * + FROM pf_graph_bfs_tree( + 'heap_objects', + 'heap_refs', + 'gc_roots', + per := ['_trace_uuid', 'upid', 'graph_sample_ts'], + full_row := true + ); + SELECT count(*) AS cnt, max(depth) AS max_depth + FROM pf_tree_flatten_recursion( + 'bfs_tree', + ['class_name'], + per := ['_trace_uuid', 'upid', 'graph_sample_ts'] + ); + """, + "limit": + 10, + }) + self.assertEqual(s_bfs, 200, bfs_res) + bfs_cnt = int(bfs_res["rows"][0]["values"][0]) + bfs_max_depth = int(bfs_res["rows"][0]["values"][1]) + self.assertEqual(bfs_cnt, expected_cnt) + self.assertGreater(bfs_max_depth, 0) + + def test_adversarial_multi_query_zero_rows_and_errors(self): + # Query #1 returns 0 rows, Query #2 returns valid rows + status, res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "queries": [ + { + "perfetto_sql": "SELECT id, name FROM slice WHERE 1 = 0;", + "table_name": "zero_rows_tbl", + }, + { + "perfetto_sql": "SELECT 77 AS lucky;", + "table_name": "non_empty_tbl", + }, + ], + "trace_filters": _COLD_ONLY, + }) + self.assertEqual(status, 200, res) + self.assertEqual(len(res["executions"]), 2) + u0 = res["executions"][0]["queryUuid"] + u1 = res["executions"][1]["queryUuid"] + st0 = self._request("GET", f"/query_executions/{u0}:status")[1] + st1 = self._request("GET", f"/query_executions/{u1}:status")[1] + self.assertEqual(st0["status"], "SUCCESS") + self.assertEqual(st0["processedRows"], 0) + self.assertEqual(st1["status"], "SUCCESS") + self.assertGreater(st1["processedRows"], 0) + + # Both tables are queryable in DuckDB + s0, r0, _ = self._duckdb_query("SELECT count(*) FROM zero_rows_tbl") + self.assertEqual(s0, 200, r0) + self.assertEqual(r0["rows"][0]["values"], ["0"]) + + s1, r1, _ = self._duckdb_query("SELECT DISTINCT lucky FROM non_empty_tbl") + self.assertEqual(s1, 200, r1) + self.assertEqual(r1["rows"][0]["values"], ["77"]) + + # DuckDB syntax error fails cleanly with 400 and does not create table + s_err, r_err, _ = self._request( + "POST", "/execute_bigtrace_query", { + "engine": "duckdb", + "perfetto_sql": "SELECT FROM WHERE SYNTAX_ERR", + "table_name": "bad_duck_syntax_tbl", + }) + self.assertEqual(s_err, 400, r_err) + s_chk, r_chk, _ = self._request("POST", "/check_table_exists", { + "table_name": "bad_duck_syntax_tbl", + }) + self.assertEqual(s_chk, 200) + self.assertFalse(r_chk["exists"]) + + def test_paging_all_filter_operators_and_order_by_edge_cases(self): + status, res, _ = self._duckdb_query( + """ + SELECT * FROM (VALUES + (1, 'alpha', 10.5), + (2, 'beta', 20.0), + (3, 'gamma', NULL), + (4, NULL, 5.0), + (5, 'delta', 100.0) + ) AS t(id, label, score) + """, + limit=100) + self.assertEqual(status, 200, res) + uid = res["queryUuid"] + + def page_ids(filters=None, order_by="id asc"): + s, p, _ = self._request( + "POST", f"/query_executions/{uid}:fetch_results", { + "offset": 0, + "limit": 50, + "filters": filters or [], + "order_by": order_by, + "columns": ["id"], + }) + self.assertEqual(s, 200, p) + return [int(r["values"][0]) for r in p["rows"]] + + self.assertEqual( + page_ids([{ + "field": "label", + "op": "=", + "value": "ALPHA" + }]), [1]) + self.assertEqual( + page_ids([{ + "field": "label", + "op": "!=", + "value": "alpha" + }]), [2, 3, 5]) + self.assertEqual( + page_ids([{ + "field": "score", + "op": "<", + "value": 20 + }]), [1, 4]) + self.assertEqual( + page_ids([{ + "field": "score", + "op": "<=", + "value": 20 + }]), [1, 2, 4]) + self.assertEqual( + page_ids([{ + "field": "score", + "op": ">", + "value": 10.5 + }]), [2, 5]) + self.assertEqual( + page_ids([{ + "field": "score", + "op": ">=", + "value": 10.5 + }]), [1, 2, 5]) + self.assertEqual( + page_ids([{ + "field": "label", + "op": "glob", + "value": "*a" + }]), [1, 2, 3, 5]) + self.assertEqual( + page_ids([{ + "field": "label", + "op": "not glob", + "value": "a*" + }]), [2, 3, 5]) + self.assertEqual( + page_ids([{ + "field": "id", + "op": "in", + "value": [2, 4, 5] + }]), [2, 4, 5]) + self.assertEqual( + page_ids([{ + "field": "id", + "op": "not in", + "value": [2, 4] + }]), [1, 3, 5]) + self.assertEqual(page_ids([{"field": "label", "op": "is null"}]), [4]) + self.assertEqual( + page_ids([{ + "field": "label", + "op": "is not null" + }]), [1, 2, 3, 5]) + self.assertEqual(page_ids([{"field": "score", "op": "is null"}]), [3]) + self.assertEqual( + page_ids([{ + "field": "score", + "op": "is not null" + }]), [1, 2, 4, 5]) + + # Order by with NULLs: ASC puts numbers in ascending order then NULLs last; + # DESC puts numbers in descending order then NULLs first (or consistent bucket order). + asc_scores = page_ids(order_by="score asc") + self.assertEqual(asc_scores, [4, 1, 2, 5, 3]) + desc_scores = page_ids(order_by="score desc") + self.assertEqual(desc_scores, [3, 5, 2, 1, 4]) + asc_labels = page_ids(order_by="label asc") + self.assertEqual(asc_labels, [1, 2, 5, 3, 4]) + + def test_perfetto_sql_next_pipe_syntax_e2e(self): + if not self.server.executor.duckdb.extension_loaded: + self.skipTest("perfetto.duckdb_extension not available") + + # 1. Extract slice and sched via single-pass multi-query + status, batch_res, _ = self._request( + "POST", "/execute_bigtrace_query", { + "queries": [ + { + "perfetto_sql": ( + "SELECT s.id, s.parent_id, s.ts, s.dur, s.name, " + "s.depth, tt.utid " + "FROM slice s JOIN thread_track tt ON s.track_id = tt.id " + "WHERE s.dur >= 0"), + "table_name": "pipe_slices", + }, + { + "perfetto_sql": ("SELECT id, ts, dur, cpu, utid " + "FROM sched WHERE utid != 0 AND dur >= 0"), + "table_name": "pipe_sched", + }, + ], + "limit": + 100000, + "trace_filters": [{ + "field": "file_name", + "op": "=", + "value": COLD_TRACE + }], + }) + self.assertEqual(status, 200, batch_res) + + # 2. Run a PerfettoSQL Next `|>` INTERVAL INTERSECTION OF pipeline directly in DuckDB mode + s_pipe, pipe_res, _ = self._duckdb_query(""" + INTERVAL INTERSECTION OF ( + pipe_slices AS s, + pipe_sched AS c + ) PER (_trace_uuid, utid) + |> AGGREGATE COUNT(*) AS cnt, SUM(dur) AS total_dur + |> SELECT cnt, total_dur + """) + self.assertEqual(s_pipe, 200, pipe_res) + + # Compare against pf_interval_intersection table macro + s_macro, macro_res, _ = self._duckdb_query(""" + SELECT count(*) AS cnt, sum(dur) AS total_dur + FROM pf_interval_intersection( + ['pipe_slices', 'pipe_sched'], + ['s', 'c'], + per := ['_trace_uuid', 'utid'], + full_row := true + ) + """) + self.assertEqual(s_macro, 200, macro_res) + self.assertEqual(pipe_res["rows"], macro_res["rows"]) + self.assertGreater(int(pipe_res["rows"][0]["values"][0]), 0) + + # 3. Run a PerfettoSQL Next `|>` TREE ACCUMULATE UP + TREE KEEP IF pipeline + s_tree_pipe, tree_pipe_res, _ = self._duckdb_query(""" + FROM pipe_slices AS s + |> TREE ACCUMULATE UP SUM(dur) AS subtree_dur PER (_trace_uuid) + |> TREE KEEP IF DESCENDANT OF ( + FROM pipe_slices |> WHERE parent_id IS NULL + ) PER (_trace_uuid) + |> AGGREGATE COUNT(*) AS desc_cnt, SUM(subtree_dur) AS sum_sub + |> SELECT desc_cnt, sum_sub + """) + self.assertEqual(s_tree_pipe, 200, tree_pipe_res) + self.assertGreater(int(tree_pipe_res["rows"][0]["values"][0]), 0) + + # 4. Run a PerfettoSQL Next `|>` pipeline with FORK AS, EXTEND, SET, RENAME, DROP, duration literal + s_fork, fork_res, _ = self._duckdb_query(""" + FROM pipe_slices AS s + |> WHERE dur >= 1us + |> EXTEND ts + dur AS end_ts + |> FORK AS long_slices + |> AGGREGATE COUNT(*) AS n, MAX(dur) AS max_dur + |> JOIN (FROM long_slices |> AGGREGATE MIN(ts) AS min_ts) ON true + |> SELECT n, max_dur, min_ts + """) + self.assertEqual(s_fork, 200, fork_res) + self.assertGreater(int(fork_res["rows"][0]["values"][0]), 0) + + +if __name__ == "__main__": + import unittest + unittest.main()
diff --git a/python/test/bigtrace_server_execution_unittest.py b/python/test/bigtrace_server_execution_unittest.py new file mode 100644 index 0000000..fc7b714 --- /dev/null +++ b/python/test/bigtrace_server_execution_unittest.py
@@ -0,0 +1,434 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Query execution, DuckDB persistence, and pool tests for the Bigtrace server.""" + +import concurrent.futures +import os +import tempfile +import threading +import unittest + +from perfetto.bigtrace.server import ( + DuckDBEngine, + TraceProcessorPool, + resolve_trace_processor_bin, +) +from perfetto.bigtrace.server.filters import sort_records +from perfetto.trace_processor.api import ( + TraceProcessorConfig, + TraceProcessorException, +) +from test.bigtrace_server_fixture import ( + BigtraceServerTestCase, + COLD_TRACE, + ENDLESS_SQL, + MANIFEST_JSON, + TEST_DATA_DIR, + base_url, + copy_trace, + request, + running_server, + trace_path, +) + + +class BigtraceServerExecutionTest(BigtraceServerTestCase): + + def test_concurrent_mixed_requests_all_succeed(self): + requests = [ + ("GET", "/", None), + ("HEAD", "/", None), + ("POST", "/trace_metadata", { + "limit": 5, + "offset": 0 + }), + ("POST", "/trace_metadata_schema", {}), + ("GET", "/trace_presets", None), + ("GET", "/query_executions", None), + ("POST", "/check_table_exists", { + "table_name": "test_table" + }), + ("POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT 1 AS n;", + "limit": 5 + }), + ] + with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool: + futures = [ + pool.submit(self._request, *requests[i % len(requests)]) + for i in range(25) + ] + for future in concurrent.futures.as_completed(futures): + self.assertEqual(future.result()[0], 200) + + def test_negative_paging_is_clamped(self): + status, data, _ = self._request("POST", "/trace_metadata", { + "limit": -10, + "offset": -5 + }) + self.assertEqual(status, 200) + self.assertIn("rows", data) + + def test_offset_past_end_returns_no_rows(self): + status, data, _ = self._request("POST", "/trace_metadata", { + "limit": 10, + "offset": 999999 + }) + self.assertEqual(status, 200) + self.assertEqual(data.get("rows", []), []) + + def test_unknown_order_by_columns_are_ignored(self): + status, data, _ = self._request("POST", "/trace_metadata", { + "order_by": "definitely_not_a_real_col desc, 12345, empty_val", + "limit": 10 + }) + self.assertEqual(status, 200) + self.assertIn("rows", data) + + def test_unknown_filter_operator_is_rejected(self): + status, data, _ = self._request( + "POST", "/trace_metadata", { + "filters": [{ + "field": "fake_column", + "op": "unknown_operator", + "value": "val" + }] + }) + self.assertEqual(status, 400) + self.assertIn("detail", data) + + def test_fetch_results_of_unknown_query_returns_404(self): + status, data, _ = self._request( + "POST", "/query_executions/non-existent-uuid:fetch_results", {}) + self.assertEqual(status, 404) + self.assertIn("detail", data) + + def test_redundant_slashes_and_v1_prefix_are_normalized(self): + status, _, _ = self._request("POST", "///v1///trace_metadata///", + {"limit": 2}) + self.assertEqual(status, 200) + + def test_cancel_and_delete_of_unknown_query_succeed(self): + status, _, _ = self._request("POST", "/query_executions/ghost-uuid:cancel", + {}) + self.assertEqual(status, 200) + status, _, _ = self._request("DELETE", "/query_executions/ghost-uuid") + self.assertEqual(status, 200) + + def test_cancel_after_finish_keeps_success_status(self): + _, data, _ = self._request("POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT 1 AS val;", + "limit": 1 + }) + uid = data["queryUuid"] + self._request("POST", f"/query_executions/{uid}:cancel", {}) + _, data, _ = self._request("GET", f"/query_executions/{uid}:status") + self.assertEqual(data["status"], "SUCCESS") + self.assertEqual(self.server.duckdb.get_status(uid)["status"], "SUCCESS") + + def test_cancel_running_query_marks_cancelled(self): + status, data, _ = self._request("POST", "/execute_bigtrace_query_async", { + "perfetto_sql": ENDLESS_SQL, + "limit": 1000000, + }) + self.assertEqual(status, 200) + uid = data["queryUuid"] + status, _, _ = self._request("POST", f"/query_executions/{uid}:cancel", {}) + self.assertEqual(status, 200) + status, data, _ = self._request("GET", f"/query_executions/{uid}:status") + self.assertEqual(status, 200) + self.assertEqual(data["status"], "CANCELLED") + + def test_sort_records_orders_numbers_then_strings_then_nulls(self): + records = [{"val": v} for v in ("hello", 10, -5, None, 0, "apple")] + result = sort_records(records, lambda r, f: r.get(f), [("val", False)]) + self.assertEqual([r["val"] for r in result], + [-5, 0, 10, "apple", "hello", None]) + + def test_duckdb_materialization_and_multi_table_joins(self): + with tempfile.TemporaryDirectory() as parquet_dir: + with running_server( + trace_dir=TEST_DATA_DIR, + manifest_path=MANIFEST_JSON, + max_workers=2, + parquet_dir=parquet_dir) as server: + url = base_url(server) + # Run 1 -> automatically materialized into result_1 with _trace_uuid. + status, q1, _ = request( + url, "POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT count(1) AS slice_cnt FROM slice;", + "limit": 100, + "trace_filters": [{ + "field": "file_name", + "op": "glob", + "value": "*api24*" + }], + "trace_metadata_columns": ["file_name"], + }) + self.assertEqual(status, 200) + self.assertEqual(len(q1["rows"]), 3) + self.assertEqual(q1["tableName"], "result_1") + schema_names = [s["name"] for s in q1["schema"]] + self.assertIn("_trace_uuid", schema_names) + self.assertIn("slice_cnt", schema_names) + + # Run 2 -> custom alternate table name "max_durs". + status, q2, _ = request( + url, "POST", "/execute_bigtrace_query", { + "perfetto_sql": + "SELECT max(dur) AS max_dur FROM slice;", + "limit": + 100, + "table_name": + "max_durs", + "trace_filters": [{ + "field": "file_name", + "op": "glob", + "value": "*api24*" + }], + }) + self.assertEqual(status, 200) + self.assertEqual(q2["tableName"], "max_durs") + + # Run 3 -> Single-pass multi-query execution amortizing trace load. + status, batch_res, _ = request( + url, "POST", "/execute_bigtrace_query", { + "queries": [ + { + "perfetto_sql": + "SELECT min(dur) AS min_dur FROM slice;", + "table_name": + "min_durs" + }, + { + "perfetto_sql": + "SELECT sum(dur) AS sum_dur FROM slice;", + "table_name": + "sum_durs" + }, + ], + "limit": + 100, + "trace_filters": [{ + "field": "file_name", + "op": "glob", + "value": "*api24*" + }], + }) + self.assertEqual(status, 200, batch_res) + self.assertEqual(len(batch_res["executions"]), 2) + self.assertEqual( + [e["tableName"] for e in batch_res["executions"]], + ["min_durs", "sum_durs"], + ) + + # List DuckDB tables. + status, tables_resp, _ = request(url, "GET", "/duckdb/tables") + self.assertEqual(status, 200) + table_names = {t["name"] for t in tables_resp["tables"]} + self.assertLessEqual( + {"result_1", "max_durs", "min_durs", "sum_durs", "trace_metadata"}, + table_names, + ) + + # Join across result_1, max_durs, min_durs, sum_durs, and trace_metadata. + status, joined, _ = request( + url, "POST", "/execute_bigtrace_query", { + "engine": + "duckdb", + "perfetto_sql": + """ + SELECT + r._trace_uuid, + r.slice_cnt, + m.max_dur, + s.sum_dur, + t.device_name + FROM result_1 r + JOIN max_durs m USING (_trace_uuid) + JOIN sum_durs s USING (_trace_uuid) + JOIN trace_metadata t USING (_trace_uuid) + ORDER BY r.slice_cnt DESC + """, + "limit": + 100, + }) + self.assertEqual(status, 200, joined) + self.assertEqual(len(joined["rows"]), 3) + self.assertEqual( + [s["name"] for s in joined["schema"]], + ["_trace_uuid", "slice_cnt", "max_dur", "sum_dur", "device_name"], + ) + + # PerfettoSQL Next DuckDB extension operators work in DuckDB mode. + status, ext_res, _ = request( + url, "POST", "/execute_bigtrace_query", { + "engine": + "duckdb", + "perfetto_sql": + """ + SELECT ts, dur, count FROM pf_interval_flatten_sum( + '(SELECT 1::BIGINT AS id, 0::BIGINT AS ts, 10::BIGINT AS dur UNION ALL SELECT 2::BIGINT AS id, 5::BIGINT AS ts, 10::BIGINT AS dur)' + ) ORDER BY ts + """, + "limit": + 100, + }) + self.assertEqual(status, 200, ext_res) + self.assertEqual(len(ext_res["rows"]), 3) + + # Parquet files were exported and are reloaded on a fresh server start. + self.assertTrue( + os.path.isfile(os.path.join(parquet_dir, "result_1.parquet"))) + self.assertTrue( + os.path.isfile(os.path.join(parquet_dir, "max_durs.parquet"))) + self.assertTrue( + os.path.isfile(os.path.join(parquet_dir, "sum_durs.parquet"))) + with running_server( + trace_dir=TEST_DATA_DIR, + manifest_path=MANIFEST_JSON, + parquet_dir=parquet_dir) as server2: + url2 = base_url(server2) + status, reloaded, _ = request( + url2, "POST", "/execute_bigtrace_query", { + "engine": + "duckdb", + "perfetto_sql": + "SELECT count(*) AS n FROM result_1 JOIN max_durs USING (_trace_uuid) JOIN sum_durs USING (_trace_uuid)", + "limit": + 10, + }) + self.assertEqual(status, 200, reloaded) + self.assertEqual(reloaded["rows"][0]["values"], ["3"]) + + +class DuckdbHistoryPersistenceTest(unittest.TestCase): + + def setUp(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.tmp_dir = tmp.name + self.duckdb_file = os.path.join(self.tmp_dir, "bigtrace.duckdb") + + def _open_engine(self) -> DuckDBEngine: + eng = DuckDBEngine(db_path=self.duckdb_file) + self.addCleanup(eng.close) + return eng + + def test_duckdb_save_load_and_cancel_in_progress_on_reopen(self): + eng = self._open_engine() + self.assertEqual(eng.list_executions(), []) + eng.save_execution({ + "queryUuid": "uuid-1", + "status": "SUCCESS", + "startTime": "2026-09-28T12:00:00Z", + "perfettoSql": "SELECT slice_id FROM slice", + "tableName": "bigtrace_test_table", + "_columns": ["slice_id", "file_name"], + }) + eng.save_execution({ + "queryUuid": "uuid-2", + "status": "IN_PROGRESS", + "startTime": "2026-09-28T12:10:00Z", + "perfettoSql": "SELECT count(*) FROM slice", + }) + self.assertEqual([e["queryUuid"] for e in eng.list_executions()], + ["uuid-2", "uuid-1"]) + eng.close() + + eng2 = self._open_engine() + status = eng2.get_status("uuid-2") + self.assertEqual(status["status"], "CANCELLED") + self.assertEqual(status["errorMessage"], "Interrupted by server restart") + self.assertTrue(eng2.delete_execution_record("uuid-1")) + self.assertIsNone(eng2.get_execution("uuid-1")) + + def test_server_restart_with_duckdb_path_keeps_history_and_results(self): + copy_trace(COLD_TRACE, self.tmp_dir, "trace.perfetto-trace") + config = dict(trace_dir=self.tmp_dir, duckdb_path=self.duckdb_file) + + with running_server(**config) as server: + url = base_url(server) + status, data, _ = request(url, "POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT count(*) AS slice_count FROM slice;", + "limit": 10 + }) + self.assertEqual(status, 200) + uid = data["queryUuid"] + self.assertIsNotNone(uid) + self.assertGreater(len(data["rows"]), 0) + self._assert_history(url, [(uid, "SUCCESS")]) + + with running_server(**config) as server: + url = base_url(server) + self._assert_history(url, [(uid, "SUCCESS")]) + + status, data, _ = request(url, "GET", f"/query_executions/{uid}") + self.assertEqual(status, 200) + self.assertEqual(data["queryUuid"], uid) + self.assertIn("slice_count", data["perfettoSql"]) + + status, data, _ = request(url, "POST", + f"/query_executions/{uid}:fetch_results", { + "limit": 50, + "offset": 0 + }) + self.assertEqual(status, 200) + self.assertEqual(data["queryUuid"], uid) + self.assertGreater(len(data["rows"]), 0) + + status, _, _ = request(url, "DELETE", f"/query_executions/{uid}") + self.assertEqual(status, 200) + self._assert_history(url, []) + + def _assert_history(self, url, expected): + status, data, _ = request(url, "GET", "/query_executions") + self.assertEqual(status, 200) + self.assertEqual([ + (e["queryUuid"], e["status"]) for e in data.get("queryExecutions", []) + ], expected) + + +class TraceProcessorPoolTest(unittest.TestCase): + + @classmethod + def setUpClass(cls): + cls.tp_config = TraceProcessorConfig(bin_path=resolve_trace_processor_bin()) + cls.cold = trace_path(COLD_TRACE) + + def test_release_closes_trace_processor(self): + pool = TraceProcessorPool(tp_config=self.tp_config) + self.addCleanup(pool.close) + tp = pool.acquire(self.cold) + self.assertIsNotNone(tp.subprocess) + pool.release(tp) + self.assertIsNone(tp.subprocess) + + def test_load_failure_reports_shell_error(self): + pool = TraceProcessorPool(tp_config=self.tp_config) + self.addCleanup(pool.close) + with self.assertRaisesRegex(TraceProcessorException, "Unknown trace type"): + pool.acquire(os.path.join(TEST_DATA_DIR, "..", "..", "OWNERS")) + + def test_load_trace_with_set_cancel_event_aborts(self): + cancel_event = threading.Event() + cancel_event.set() + pool = TraceProcessorPool(tp_config=self.tp_config) + self.addCleanup(pool.close) + with self.assertRaises(TraceProcessorException): + pool.acquire(self.cold, cancel_event=cancel_event) + + +if __name__ == "__main__": + unittest.main()
diff --git a/python/test/bigtrace_server_fixture.py b/python/test/bigtrace_server_fixture.py new file mode 100644 index 0000000..0220565 --- /dev/null +++ b/python/test/bigtrace_server_fixture.py
@@ -0,0 +1,140 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Shared helpers and fixture for the Bigtrace server tests.""" + +import contextlib +import json +import os +import shutil +import tempfile +import time +import unittest +import urllib.error +import urllib.request +from typing import Any, Callable, Iterator, Optional, Tuple + +from perfetto.bigtrace.server import ( + BigtraceServer, + BigtraceServerConfig, + resolve_trace_processor_bin, +) + +REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) +TEST_DATA_DIR = os.path.join(REPO_ROOT, "test", "data") +EXAMPLES_DIR = os.path.join(REPO_ROOT, "examples", "bigtrace") +MANIFEST_JSON = os.path.join(EXAMPLES_DIR, "bigtrace_manifest.json") +MANIFEST_CSV = os.path.join(EXAMPLES_DIR, "bigtrace_manifest.csv") + +COLD_TRACE = "api24_startup_cold.perfetto-trace" +HOT_TRACE = "api24_startup_hot.perfetto-trace" +WARM_TRACE = "api24_startup_warm.perfetto-trace" + +# Runs until killed. +ENDLESS_SQL = ("WITH RECURSIVE n(i) AS " + "(SELECT 0 UNION ALL SELECT i + 1 FROM n) " + "SELECT count(1) FROM n") + + +def trace_path(name: str) -> str: + return os.path.join(TEST_DATA_DIR, name) + + +def copy_trace(name: str, dst_dir: str, dst_name: str) -> str: + """Copies test/data/<name> to <dst_dir>/<dst_name> and returns the path.""" + dst = os.path.join(dst_dir, dst_name) + shutil.copyfile(trace_path(name), dst) + return dst + + +def wait_for(condition: Callable[[], Any], timeout: float = 10.0) -> None: + deadline = time.monotonic() + timeout + while not condition(): + if time.monotonic() > deadline: + raise AssertionError("Timed out waiting for condition") + time.sleep(0.02) + + +def request_raw(base_url: str, + method: str, + path: str, + body: Optional[dict] = None, + headers: Optional[dict] = None) -> Tuple[int, bytes, dict]: + """Sends a request and returns (status, raw body, headers), even on errors.""" + data = json.dumps(body).encode("utf-8") if body is not None else None + all_headers = {"Content-Type": "application/json", **(headers or {})} + req = urllib.request.Request( + f"{base_url}{path}", data=data, headers=all_headers, method=method) + try: + with urllib.request.urlopen(req) as resp: + return resp.status, resp.read(), resp.headers + except urllib.error.HTTPError as e: + with e: + return e.code, e.read(), e.headers + + +def request(base_url: str, + method: str, + path: str, + body: Optional[dict] = None) -> Tuple[int, dict, dict]: + """Like request_raw() but decodes the body as JSON ({} if empty).""" + status, raw, headers = request_raw(base_url, method, path, body) + return status, json.loads(raw) if raw else {}, headers + + +@contextlib.contextmanager +def running_server(**config_overrides) -> Iterator[BigtraceServer]: + """Runs a BigtraceServer on an ephemeral port in the background.""" + config = BigtraceServerConfig( + **{ + "host": "127.0.0.1", + "port": 0, + "bin_path": resolve_trace_processor_bin(), + **config_overrides + }) + server = BigtraceServer(config) + try: + server.start(background=True) + yield server + finally: + server.stop() + + +def base_url(server: BigtraceServer) -> str: + return f"http://127.0.0.1:{server.server_port}" + + +class BigtraceServerTestCase(unittest.TestCase): + """Runs one server per test class over test/data and the example manifest.""" + + @classmethod + def setUpClass(cls): + cls.bin_path = resolve_trace_processor_bin() + cls._server_cm = running_server( + trace_dir=TEST_DATA_DIR, + manifest_path=MANIFEST_JSON, + max_workers=4, + ) + cls.server = cls._server_cm.__enter__() + cls.base_url = base_url(cls.server) + + @classmethod + def tearDownClass(cls): + cls._server_cm.__exit__(None, None, None) + + def setUp(self): + self.server.catalog.reload( + trace_dir=TEST_DATA_DIR, manifest_path=MANIFEST_JSON) + + def _request(self, method: str, path: str, body: Optional[dict] = None): + return request(self.base_url, method, path, body)
diff --git a/python/test/bigtrace_server_manifest_unittest.py b/python/test/bigtrace_server_manifest_unittest.py new file mode 100644 index 0000000..b56d13c --- /dev/null +++ b/python/test/bigtrace_server_manifest_unittest.py
@@ -0,0 +1,279 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Manifest generation and trace catalog tests for the Bigtrace server.""" + +import os +import sqlite3 +import tempfile +import unittest + +from perfetto.bigtrace.server import ( + BigtraceManifestGenerator, + BigtraceQueryExecutor, + TraceCatalog, +) +from perfetto.bigtrace.server.cli import parse_args +from perfetto.trace_processor import TraceProcessorConfig +from test.bigtrace_server_fixture import ( + BigtraceServerTestCase, + COLD_TRACE, + HOT_TRACE, + MANIFEST_CSV, + MANIFEST_JSON, + TEST_DATA_DIR, + WARM_TRACE, + base_url, + copy_trace, + request, + running_server, +) + + +def write_file(path: str, content: bytes) -> None: + with open(path, "wb") as f: + f.write(content) + + +class BigtraceServerManifestTest(BigtraceServerTestCase): + + def setUp(self): + super().setUp() + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.tmp_dir = tmp.name + + def _generator(self, **kwargs) -> BigtraceManifestGenerator: + kwargs.setdefault("max_workers", 2) + return BigtraceManifestGenerator( + trace_dir=self.tmp_dir, bin_path=self.bin_path, **kwargs) + + def test_generator_keeps_traces_and_discards_non_traces(self): + d = self.tmp_dir + copy_trace(COLD_TRACE, d, "trace1.perfetto-trace") + copy_trace("android_job_scheduler.perfetto-trace", d, + "trace2.perfetto-trace") + copy_trace(HOT_TRACE, d, "trace3.customext") + copy_trace(WARM_TRACE, d, "trace4_no_ext") + os.makedirs(os.path.join(d, "nested")) + copy_trace("track_event_with_description.perfetto-trace", + os.path.join(d, "nested"), "trace5.perfetto-trace") + write_file(os.path.join(d, "notes.txt"), b"Plain text, not a trace.") + write_file(os.path.join(d, "empty.bin"), b"") + write_file( + os.path.join(d, "corrupt.perfetto-trace"), b"\xde\xad\xbe\xef" * 16) + + manifest_path = os.path.join(d, "bigtrace_manifest.json") + manifest = self._generator( + recursive=True, max_workers=4).generate(output_path=manifest_path) + + self.assertTrue(os.path.isfile(manifest_path)) + self.assertEqual(len(manifest["traces"]), 5) + schema = {c["name"]: c for c in manifest["schema"]} + self.assertLessEqual( + { + "file_name", "duration_ms", "slice_count", "process_count", + "thread_count" + }, set(schema)) + self.assertEqual(schema["duration_ms"].get("displayName"), "Duration (ms)") + self.assertEqual(schema["duration_ms"].get("type"), "FLOAT64") + self.assertEqual(schema["slice_count"].get("displayName"), "Slice Count") + self.assertEqual(schema["slice_count"].get("type"), "INT64") + + with running_server( + trace_dir=d, manifest_path=manifest_path, max_workers=2) as server: + url = base_url(server) + status, data, _ = request(url, "POST", "/trace_metadata_schema", {}) + self.assertEqual(status, 200) + self.assertLessEqual({"duration_ms", "slice_count"}, + {c["name"] for c in data["columns"]}) + + status, data, _ = request( + url, "POST", "/trace_metadata", + {"columns": ["file_name", "duration_ms", "slice_count"]}) + self.assertEqual(status, 200) + self.assertEqual(data["totalFilteredRows"], 5) + self.assertEqual(len(data["rows"]), 5) + + status, data, _ = request( + url, "POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT count(1) AS total_slices FROM slice;", + "limit": 10 + }) + self.assertEqual(status, 200) + self.assertGreater(len(data["rows"]), 0) + + def test_generator_custom_metadata_query(self): + copy_trace(COLD_TRACE, self.tmp_dir, "cold.perfetto-trace") + manifest = self._generator( + recursive=False, + metadata_query=("SELECT count(1) AS ui_slices_count FROM slice " + "WHERE name LIKE '%Choreographer%';"), + ).generate(output_path=os.path.join(self.tmp_dir, "manifest.json")) + self.assertEqual(len(manifest["traces"]), 1) + self.assertGreater(manifest["traces"][0]["ui_slices_count"], 0) + self.assertIn("ui_slices_count", [c["name"] for c in manifest["schema"]]) + + def test_generator_incremental_refresh_reuses_and_prunes(self): + t1 = copy_trace(COLD_TRACE, self.tmp_dir, "t1.perfetto-trace") + copy_trace(WARM_TRACE, self.tmp_dir, "t2.perfetto-trace") + manifest_path = os.path.join(self.tmp_dir, "manifest.json") + gen = self._generator() + + def uuids_by_name(manifest): + return {t["file_name"]: t["trace_uuid"] for t in manifest["traces"]} + + first = uuids_by_name(gen.generate(output_path=manifest_path)) + self.assertEqual(len(first), 2) + + copy_trace("track_event_with_description.perfetto-trace", self.tmp_dir, + "t3.perfetto-trace") + second = uuids_by_name(gen.generate(output_path=manifest_path)) + self.assertEqual(len(second), 3) + self.assertEqual(second["t1.perfetto-trace"], first["t1.perfetto-trace"]) + + os.remove(t1) + third = uuids_by_name(gen.generate(output_path=manifest_path)) + self.assertEqual(set(third), {"t2.perfetto-trace", "t3.perfetto-trace"}) + + def test_refresh_endpoint_and_restart_pick_up_regenerated_manifest(self): + copy_trace(COLD_TRACE, self.tmp_dir, "trace_a.perfetto-trace") + manifest_path = os.path.join(self.tmp_dir, "bigtrace_manifest.json") + gen = self._generator() + gen.generate(output_path=manifest_path) + config = dict(trace_dir=self.tmp_dir, manifest_path=manifest_path) + + with running_server(**config) as server: + self.assertEqual(len(server.catalog.get_traces()), 1) + copy_trace(WARM_TRACE, self.tmp_dir, "trace_b.perfetto-trace") + gen.generate(output_path=manifest_path) + + status, data, _ = request(base_url(server), "POST", "/refresh", {}) + self.assertEqual(status, 200) + self.assertEqual(data["status"], "ok") + self.assertEqual(data["traceCount"], 2) + self.assertEqual(len(server.catalog.get_traces()), 2) + + with running_server(**config) as server: + self.assertEqual(len(server.catalog.get_traces()), 2) + + def test_catalog_is_configured_with(self): + catalog = TraceCatalog(manifest_path=MANIFEST_JSON) + self.assertTrue(catalog.is_configured_with(manifest_path=MANIFEST_JSON)) + self.assertFalse(catalog.is_configured_with(trace_dir="/tmp/different_dir")) + + def test_catalog_recursive_scanning(self): + d = self.tmp_dir + os.makedirs(os.path.join(d, "a")) + os.makedirs(os.path.join(d, "b")) + copy_trace(COLD_TRACE, d, "root.perfetto-trace") + copy_trace(HOT_TRACE, os.path.join(d, "a"), "a.perfetto-trace") + copy_trace(WARM_TRACE, os.path.join(d, "b"), "b.perfetto-trace") + + flat = TraceCatalog(trace_dir=d, recursive=False).get_traces() + self.assertEqual([t.file_name for t in flat], ["root.perfetto-trace"]) + self.assertEqual( + len(TraceCatalog(trace_dir=d, recursive=True).get_traces()), 3) + + status, data, _ = self._request( + "POST", "/trace_metadata", { + "settings": [ + { + "setting_id": "trace_directory", + "values": [d], + "category": "TRACE_ADDRESS", + }, + { + "setting_id": "recursive", + "values": ["true"], + "category": "TRACE_ADDRESS", + }, + ], + "limit": 10, + "columns": ["file_name"], + }) + self.assertEqual(status, 200) + self.assertEqual(data["totalFilteredRows"], 3) + + def test_catalog_loads_csv_manifest(self): + traces = TraceCatalog( + trace_dir=TEST_DATA_DIR, manifest_path=MANIFEST_CSV).get_traces() + self.assertEqual(len(traces), 7) + first = traces[0] + self.assertEqual(first.get_value("device_name"), "Pixel 6") + self.assertEqual(first.get_value("build_id"), "NRD90M.220624.014") + self.assertEqual(first.get_value("tag"), "cold") + + def test_query_skips_non_trace_files_in_directory(self): + d = self.tmp_dir + copy_trace(COLD_TRACE, d, "trace_no_extension") + copy_trace(HOT_TRACE, d, "trace_custom.xyz123") + write_file(os.path.join(d, "notes.txt"), b"Plain text, not a trace.") + write_file(os.path.join(d, "random.bin"), b"\x00\x01\x02\x03\x04\x05") + + # Directory scans list every file regardless of extension. + catalog = TraceCatalog(trace_dir=d, recursive=False) + self.assertEqual({t.file_name for t in catalog.get_traces()}, { + "trace_no_extension", "trace_custom.xyz123", "notes.txt", "random.bin" + }) + + executor = BigtraceQueryExecutor( + catalog=catalog, + tp_config=TraceProcessorConfig(bin_path=self.bin_path), + max_workers=2, + ) + self.addCleanup(executor.close) + result = executor.execute({ + "perfetto_sql": "SELECT count(1) AS slice_count FROM slice;", + "limit": 100, + "trace_metadata_columns": ["file_name"], + }) + self.assertEqual({r["values"][1] for r in result["rows"]}, + {"trace_no_extension", "trace_custom.xyz123"}) + self.assertEqual(len(result["rows"]), 2) + + def test_empty_setting_overrides_keep_manifest(self): + empty = [{ + "setting_id": s, + "values": [""] + } for s in ("trace_directory", "trace_manifest", "trace_uuids")] + status, data, _ = self._request("POST", "/trace_metadata_schema", + {"settings": empty}) + self.assertEqual(status, 200) + self.assertIn("device_name", [c["name"] for c in data["columns"]]) + self.assertEqual(self.server.catalog.manifest_path, MANIFEST_JSON) + + def test_sqlite_metadata_db_is_the_manifest(self): + with tempfile.TemporaryDirectory() as tmp: + trace = copy_trace(COLD_TRACE, tmp, "cold.perfetto-trace") + with sqlite3.connect(os.path.join(tmp, "metadata.db")) as conn: + conn.execute("CREATE TABLE traces (trace_uuid TEXT, " + "absolute_local_file_path TEXT, device_name TEXT)") + conn.execute("INSERT INTO traces VALUES ('u1', ?, 'pixel')", (trace,)) + conn.close() + with running_server(trace_dir=tmp) as server: + traces = server.catalog.get_traces() + self.assertEqual([t.trace_uuid for t in traces], ["u1"]) + self.assertEqual(traces[0].get_value("device_name"), "pixel") + + def test_defaults_and_explicit_flags(self): + args = parse_args([]) + self.assertIsNone(args.parquet_dir) + self.assertIsNone(args.duckdb_path) + args = parse_args(["--parquet-dir", "/tmp/pq", "--duckdb-path", "/tmp/bt"]) + self.assertEqual(args.parquet_dir, "/tmp/pq") + self.assertEqual(args.duckdb_path, "/tmp/bt") + + +if __name__ == "__main__": + unittest.main()
diff --git a/python/test/bigtrace_server_robustness_unittest.py b/python/test/bigtrace_server_robustness_unittest.py new file mode 100644 index 0000000..f8d7d4a --- /dev/null +++ b/python/test/bigtrace_server_robustness_unittest.py
@@ -0,0 +1,280 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Failure handling tests: crashing shells, bad files, full disks, shutdown.""" + +import concurrent.futures +import contextlib +import io +import os +import signal +import sqlite3 +import stat +import sys +import tempfile +import time +import unittest +from unittest import mock + +from perfetto.bigtrace.server import ( + BigtraceQueryExecutor, + TraceCatalog, + resolve_trace_processor_bin, +) +from perfetto.bigtrace.server.cli import parse_args +from perfetto.trace_processor.api import TraceProcessorConfig +from test.bigtrace_server_fixture import ( + COLD_TRACE, + ENDLESS_SQL, + HOT_TRACE, + WARM_TRACE, + base_url, + copy_trace, + request, + running_server, + wait_for, +) + +TRACES = {"cold.pftrace": COLD_TRACE, "hot.pftrace": HOT_TRACE} + + +def _write_script(path: str, body: str) -> str: + with open(path, "w", encoding="utf-8") as f: + f.write("#!/bin/sh\n" + body) + os.chmod(path, stat.S_IRWXU) + return path + + +def _child_shells(): + """Returns pids of this process's trace_processor_shell children.""" + pids = [] + for pid in filter(str.isdigit, os.listdir("/proc")): + try: + with open(f"/proc/{pid}/stat", encoding="utf-8") as f: + comm, rest = f.read().rsplit(")", 1) + except OSError: + continue + if int(rest.split()[1]) == os.getpid() and "trace_processor" in comm: + pids.append(int(pid)) + return pids + + +class ExecutorRobustnessTest(unittest.TestCase): + """Runs the executor over a temporary copy of three traces.""" + + def setUp(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.dir = tmp.name + self.paths = { + name: copy_trace(trace, self.dir, name) + for name, trace in (("cold.pftrace", COLD_TRACE), + ("hot.pftrace", HOT_TRACE), ("warm.pftrace", + WARM_TRACE)) + } + self.executor = BigtraceQueryExecutor( + catalog=TraceCatalog(trace_dir=self.dir), + tp_config=TraceProcessorConfig(bin_path=resolve_trace_processor_bin()), + max_workers=3) + self.addCleanup(self.executor.close) + + def _run(self, sql, **params): + return self.executor.execute({"perfetto_sql": sql, **params}) + + def _rows_by_file(self, result): + names = [c["name"] for c in result["schema"]] + i, f = names.index("n"), names.index("file_name") + return {r["values"][f]: r["values"][i] for r in result["rows"]} + + def _wait_until_done(self, uid): + wait_for(lambda: self.executor.get_status(uid)["status"] != "IN_PROGRESS") + return self.executor.get_execution(uid) + + def test_shell_killed_mid_query_fails_only_that_trace(self): + counts = self._rows_by_file( + self._run( + "SELECT count(*) AS n FROM slice", + trace_metadata_columns=["file_name"])) + self.assertEqual(len(set(counts.values())), 3) + # Endless on cold only. + sql = ("WITH RECURSIVE n(i) AS (SELECT 0 UNION ALL SELECT i + 1 FROM n " + "WHERE i < IIF((SELECT count(*) FROM slice) = " + f"{counts['cold.pftrace']}, 1e18, 0)) SELECT count(*) AS n FROM n") + uid = self.executor.execute({"perfetto_sql": sql}, + is_async=True)["queryUuid"] + ex = self.executor._executions[uid] + wait_for(lambda: ex["processedTraces"] == 2 and len(ex["_active"]) == 1) + (tp,) = list(ex["_active"]) + os.kill(tp.subprocess.pid, signal.SIGKILL) + + ex = self._wait_until_done(uid) + self.assertEqual(ex["status"], "SUCCESS") + self.assertEqual(ex["processedRows"], 2) + self.assertRegex(ex["errorMessage"], r"^1 trace\(s\) failed: cold.pftrace") + self.assertEqual(self._run("SELECT 1 AS n")["totalFilteredRows"], 3) + + def test_bad_files_are_reported_per_trace(self): + open(os.path.join(self.dir, "empty.pftrace"), "wb").close() + with open(self.paths["cold.pftrace"], "rb") as src: + head = src.read(20000) + with open(os.path.join(self.dir, "truncated.pftrace"), "wb") as f: + f.write(head) + with open(os.path.join(self.dir, "notes.txt"), "w", encoding="utf-8") as f: + f.write("not a trace\n") + self.executor.catalog.reload() + + result = self._run( + "SELECT count(*) AS n FROM slice", trace_metadata_columns=["file_name"]) + counts = self._rows_by_file(result) + # Empty and truncated files load as (nearly) empty traces. + self.assertEqual(counts["empty.pftrace"], "0") + self.assertIn("truncated.pftrace", counts) + self.assertNotIn("notes.txt", counts) + ex = self.executor.get_execution(result["queryUuid"]) + self.assertEqual(ex["status"], "SUCCESS") + self.assertRegex(ex["errorMessage"], + r"^1 trace\(s\) failed: notes.txt: .*Unknown trace type") + + def test_trace_deleted_after_listing_is_reported(self): + os.remove(self.paths["hot.pftrace"]) + result = self._run("SELECT 1 AS n") + self.assertEqual(result["totalFilteredRows"], 2) + ex = self.executor.get_execution(result["queryUuid"]) + self.assertEqual(ex["errorMessage"], + "1 trace(s) failed: hot.pftrace: trace file not found") + + def test_query_failing_everywhere_raises(self): + with self.assertRaisesRegex(Exception, "3 trace\\(s\\) failed"): + self._run("SELECT * FROM no_such_table") + + def test_history_write_failure_still_serves_results(self): + disk_full = OSError("database or disk is full") + with mock.patch.object( + self.executor.duckdb, "save_execution", side_effect=disk_full), \ + self.assertLogs("perfetto.bigtrace.server.executor", "WARNING") as logs: + result = self._run("SELECT 1 AS n") + self.assertIn("database or disk is full", logs.output[0]) + self.assertEqual(result["totalFilteredRows"], 3) + page = self.executor.fetch_results(result["queryUuid"], limit=2) + self.assertEqual(len(page["rows"]), 2) + self.assertEqual(self._run("SELECT 2 AS n")["totalFilteredRows"], 3) + + def test_deleting_a_running_query_does_not_bring_it_back(self): + uid = self.executor.execute({"perfetto_sql": ENDLESS_SQL}, + is_async=True)["queryUuid"] + ex = self.executor._executions[uid] + wait_for(lambda: len(ex["_active"]) == 3) + self.assertTrue(self.executor.delete_execution(uid)) + for run in self.executor._runs: + run.join(timeout=10) + self.assertIsNone(self.executor.get_execution(uid)) + self.assertNotIn(uid, + [e["queryUuid"] for e in self.executor.list_executions()]) + + def test_concurrent_queries_get_their_own_results(self): + with concurrent.futures.ThreadPoolExecutor(max_workers=6) as threads: + results = list( + threads.map(lambda i: self._run(f"SELECT {i} AS n"), range(6))) + for i, result in enumerate(results): + self.assertEqual([r["values"][0] for r in result["rows"]], [str(i)] * 3) + + def test_paging_sorting_and_filtering_results(self): + result = self._run( + "SELECT id AS n FROM slice ORDER BY id LIMIT 3", + trace_metadata_columns=["file_name"]) + uid = result["queryUuid"] + fetch = lambda **kw: self.executor.fetch_results(uid, **kw) + + page = fetch(order_by="n desc, file_name", limit=4) + values = [int(r["values"][0]) for r in page["rows"]] + self.assertEqual(values, sorted(values, reverse=True)) + self.assertEqual(page["totalFilteredRows"], 9) + + hot = [{"field": "file_name", "op": "=", "value": "hot.pftrace"}] + page = fetch(filters=hot, order_by="n", offset=2, limit=5, columns=["n"]) + self.assertEqual(page["totalFilteredRows"], 3) + self.assertEqual([s["name"] for s in page["schema"]], ["n"]) + self.assertEqual(len(page["rows"]), 1) + + # Rows appended to the DuckDB table are immediately reflected in paging. + # Result tables store only `_trace_uuid` + query columns; trace metadata + # is joined in when paging. + tname = self.executor._executions[uid]["tableName"] + duck = self.executor.duckdb + cols = [c["name"] for c in duck._table_meta[tname]["columns"]] + self.assertEqual(sorted(cols), ["_trace_uuid", "n"]) + uuid = duck._conn.execute( + f"SELECT CAST(_trace_uuid AS VARCHAR) FROM {tname} LIMIT 1").fetchone( + )[0] + duck._conn.executemany( + f"INSERT INTO {tname} (_trace_uuid, n) VALUES (?, ?)", [(uuid, 10), + (uuid, 11), + (uuid, 12)]) + self.assertEqual(fetch(order_by="n desc")["totalFilteredRows"], 12) + + +class ServerFailureTest(unittest.TestCase): + + def setUp(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.dir = tmp.name + for name, trace in TRACES.items(): + copy_trace(trace, self.dir, name) + + def test_unusable_shell_fails_queries_but_server_keeps_running(self): + bin_dir = tempfile.TemporaryDirectory() + self.addCleanup(bin_dir.cleanup) + broken = _write_script( + os.path.join(bin_dir.name, "broken_shell"), + "echo shell is broken >&2\nexit 1\n") + for bin_path, error in ((broken, "shell is broken"), + (os.path.join(bin_dir.name, "missing"), + "Path to binary is not valid")): + with self.subTest(bin_path), running_server( + trace_dir=self.dir, bin_path=bin_path) as server: + url = base_url(server) + status, data, _ = request(url, "POST", "/execute_bigtrace_query", + {"perfetto_sql": "SELECT 1"}) + self.assertEqual(status, 400) + self.assertIn("2 trace(s) failed", data["detail"]) + self.assertIn(error, data["detail"]) + self.assertEqual(request(url, "GET", "/")[0], 200) + + def test_missing_shell_path_flag_is_rejected(self): + with self.assertRaises(SystemExit), contextlib.redirect_stderr( + io.StringIO()): + parse_args(["--shell-path", os.path.join(self.dir, "missing")]) + + @unittest.skipUnless(sys.platform.startswith("linux"), "uses /proc") + def test_stop_is_prompt_and_leaves_no_shells(self): + server_cm = running_server(trace_dir=self.dir, max_workers=2) + server = server_cm.__enter__() + try: + url = base_url(server) + request(url, "POST", "/execute_bigtrace_query", + {"perfetto_sql": "SELECT 1"}) + uid = request(url, "POST", "/execute_bigtrace_query_async", + {"perfetto_sql": ENDLESS_SQL})[1]["queryUuid"] + ex = server.executor._executions[uid] + wait_for(lambda: len(ex["_active"]) == 2) + self.assertTrue(_child_shells()) + finally: + start = time.monotonic() + server_cm.__exit__(None, None, None) + self.assertLess(time.monotonic() - start, 5) + self.assertEqual(_child_shells(), []) + + +if __name__ == "__main__": + unittest.main()
diff --git a/python/test/bigtrace_server_stream_unittest.py b/python/test/bigtrace_server_stream_unittest.py new file mode 100644 index 0000000..191f231 --- /dev/null +++ b/python/test/bigtrace_server_stream_unittest.py
@@ -0,0 +1,428 @@ +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Tests streaming ingest of trace processor results into DuckDB.""" + +import os +import socket +import tempfile +import threading +import time +import unittest + +from perfetto.bigtrace.server import stream_ingest +from perfetto.bigtrace.server.duckdb_engine import DuckDBEngine +from perfetto.bigtrace.server.stream_ingest import StreamIngest +from perfetto.trace_processor import TraceProcessor, TraceProcessorConfig +from test.bigtrace_server_fixture import ( + BigtraceServerTestCase, + COLD_TRACE, + resolve_trace_processor_bin, + trace_path, + wait_for, +) + +_COLD_ONLY = [{"field": "file_name", "op": "glob", "value": "*startup_cold*"}] +_SLICE_SQL = "SELECT id, ts, dur, name, depth FROM slice" + + +def _varint(v): + out = bytearray() + while v >= 0x80: + out.append((v & 0x7F) | 0x80) + v >>= 7 + out.append(v) + return bytes(out) + + +def _field(num, payload): + return _varint((num << 3) | 2) + _varint(len(payload)) + payload + + +def _result(columns, rows): + """Encodes a QueryResult; rows hold ints, floats, strings or None.""" + cells, varints, floats, strings = bytearray(), bytearray(), bytearray(), b"" + for row in rows: + for v in row: + if v is None: + cells.append(1) + elif isinstance(v, int): + cells.append(2) + varints += _varint(v & ((1 << 64) - 1)) + elif isinstance(v, float): + cells.append(3) + floats += __import__("struct").pack("<d", v) + else: + cells.append(4) + strings += v.encode() + b"\0" + batch = _field(1, bytes(cells)) + _field(2, bytes(varints)) + if floats: + batch += _field(3, bytes(floats)) + if strings: + batch += _field(5, strings) + out = b"".join(_field(1, c.encode()) for c in columns) + return out + _field(3, batch) + + +class _FakeTp: + """Serves one canned HTTP reply per connection on a local port.""" + + def __init__(self, reply: bytes, delay_after: int = -1): + self.reply = reply + self.delay_after = delay_after + self.release = threading.Event() + self.sock = socket.socket() + self.sock.bind(("127.0.0.1", 0)) + self.sock.listen(8) + self.port = self.sock.getsockname()[1] + self.thread = threading.Thread(target=self._serve, daemon=True) + self.thread.start() + + def _serve(self): + while True: + try: + conn, _ = self.sock.accept() + except OSError: + return + with conn: + data = b"" + while b"\r\n\r\n" not in data: + chunk = conn.recv(65536) + if not chunk: + break + data += chunk + try: + if self.delay_after >= 0: + conn.sendall(self.reply[:self.delay_after]) + self.release.wait(30) + conn.sendall(self.reply[self.delay_after:]) + else: + conn.sendall(self.reply) + except OSError: + pass + + def close(self): + self.release.set() + self.sock.close() + + +def _chunked(body: bytes, pieces: int = 3) -> bytes: + out = b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n" + step = max(1, len(body) // pieces) + for i in range(0, len(body), step): + part = body[i:i + step] + out += b"%x\r\n" % len(part) + part + b"\r\n" + return out + b"0\r\n\r\n" + + +class StreamIngestTest(unittest.TestCase): + """Drives the extension's stream API directly.""" + + @classmethod + def setUpClass(cls): + cls.tmp = tempfile.TemporaryDirectory() + cls.engine = DuckDBEngine(os.path.join(cls.tmp.name, "t.duckdb")) + if cls.engine.stream_api is None: + raise unittest.SkipTest("perfetto DuckDB extension unavailable") + cls.api = cls.engine.stream_api + + @classmethod + def tearDownClass(cls): + cls.engine.close() + cls.tmp.cleanup() + + def _ingest(self, table, columns, types, sql="SELECT 1", **kw): + + def on_schema(cols, typs): + schema = dict(zip(cols, typs)) + schema["_trace_uuid"] = "STRING" + self.engine.create_streaming_table( + table, list(cols) + ["_trace_uuid"], schema, uuid_enum=False) + return lambda handle: self.engine.insert_from_stream( + table, handle, cols, typs, ["t0", "t1", "t2", "t3"], False, None, 0) + + return StreamIngest(self.api, sql, on_schema, columns=columns, **kw) + + def _rows(self, sql): + return self.engine._conn.execute(sql).fetchall() + + def test_streams_fake_trace_processor_reply(self): + body = _result(["a", "b", "c"], [(1, "x", 1.5), (None, "y", None), + (-3, None, 2.0)]) + tp = _FakeTp(_chunked(body, pieces=7)) + try: + ingest = self._ingest("fake1", ["a", "b", "c"], + ["INT64", "STRING", "DOUBLE"]) + res = ingest.feed(f"tcp:127.0.0.1:{tp.port}", 2) + self.assertEqual(res.status, stream_ingest.FEED_OK, res.message) + self.assertEqual(res.rows, 3) + self.assertEqual(ingest.finish(), []) + finally: + tp.close() + self.assertEqual( + sorted(self._rows("SELECT a, b, c, _trace_uuid FROM fake1"), key=str), + [(-3, None, 2.0, "t2"), (1, "x", 1.5, "t2"), (None, "y", None, "t2")]) + + def test_reports_sql_errors_and_column_mismatches(self): + err = _field(1, b"a") + _field(2, b"no such table: foo") + other = _result(["z"], [(1,)]) + tp_err, tp_cols = _FakeTp(_chunked(err)), _FakeTp(_chunked(other)) + try: + ingest = self._ingest("fake2", ["a"], ["INT64"]) + res = ingest.feed(f"tcp:127.0.0.1:{tp_err.port}", 0) + self.assertEqual(res.status, stream_ingest.FEED_SQL_ERROR) + self.assertIn("no such table", res.message) + res = ingest.feed(f"tcp:127.0.0.1:{tp_cols.port}", 0) + self.assertEqual(res.status, stream_ingest.FEED_COLUMN_MISMATCH) + ingest.finish() + finally: + tp_err.close() + tp_cols.close() + self.assertEqual(self._rows("SELECT count(*) FROM fake2"), [(0,)]) + + def test_truncated_reply_and_dead_trace_processor_are_io_errors(self): + body = _chunked(_result(["a"], [(i,) for i in range(1000)])) + tp = _FakeTp(body[:len(body) // 2]) # Closes mid-result. + dead = socket.socket() + dead.bind(("127.0.0.1", 0)) + dead_port = dead.getsockname()[1] + dead.close() + try: + ingest = self._ingest("fake3", ["a"], ["INT64"]) + res = ingest.feed(f"tcp:127.0.0.1:{tp.port}", 0) + self.assertEqual(res.status, stream_ingest.FEED_IO_ERROR) + res = ingest.feed(f"tcp:127.0.0.1:{dead_port}", 0) + self.assertEqual(res.status, stream_ingest.FEED_IO_ERROR) + self.assertIn("connect", res.message) + ingest.finish() + finally: + tp.close() + + def test_cancel_unblocks_stalled_feeds_promptly(self): + body = _chunked(_result(["a"], [(i,) for i in range(10)])) + tp = _FakeTp(body, delay_after=len(body) // 2) # Stalls mid-result. + try: + ingest = self._ingest("fake4", ["a"], ["INT64"]) + out = [] + t = threading.Thread( + target=lambda: out.append(ingest.feed(f"tcp:127.0.0.1:{tp.port}", 0))) + t.start() + time.sleep(0.3) + start = time.monotonic() + ingest.cancel() + t.join(5) + self.assertFalse(t.is_alive()) + self.assertLess(time.monotonic() - start, 2) + self.assertEqual(out[0].status, stream_ingest.FEED_CANCELLED) + ingest.finish() + finally: + tp.close() + + def test_full_queue_backpressure_and_cancel(self): + # One chunk of queue and no consumer: a big result must block, not grow. + handle = self.api.open("SELECT 1", ["a"], queue_chunks=1) + rows = [(i,) for i in range(50000)] + tp = _FakeTp(_chunked(_result(["a"], rows), pieces=50)) + out = [] + try: + t = threading.Thread(target=lambda: out.append( + self.api.feed(handle, f"tcp:127.0.0.1:{tp.port}", 0))) + t.start() + time.sleep(0.5) + self.assertTrue(t.is_alive()) + self.api.cancel(handle) + t.join(5) + self.assertFalse(t.is_alive()) + self.assertEqual(out[0].status, stream_ingest.FEED_CANCELLED) + finally: + self.api.free(handle) + tp.close() + + def test_segments_rotate_and_spool_files_are_consumed(self): + ingest = self._ingest("fake5", ["a"], ["INT64"], segment_traces=2) + tps = [] + try: + for i in range(5): + tp = _FakeTp(_chunked(_result(["a"], [(i,), (i * 10,)]))) + tps.append(tp) + res = ingest.feed(f"tcp:127.0.0.1:{tp.port}", i % 4) + self.assertEqual(res.status, stream_ingest.FEED_OK, res.message) + self.assertEqual(res.rows, 2) + if i == 0: + wait_for(lambda: ingest.started, 2.0) + self.assertEqual(ingest.finish(), []) + self.assertGreaterEqual(len(ingest._segments), 3) + self.assertEqual( + self._rows("SELECT count(*), sum(a) FROM fake5"), [(10, 110)]) + finally: + for tp in tps: + tp.close() + + def test_parallel_inserts_drain_one_stream(self): + ingest = self._ingest("fake7", ["a"], ["INT64"]) + tps = [] + try: + for i in range(20): + tp = _FakeTp(_chunked(_result(["a"], [(j,) for j in range(5000)]))) + tps.append(tp) + self.assertTrue(ingest.feed(f"tcp:127.0.0.1:{tp.port}", i % 4).ok) + self.assertEqual(ingest.finish(), []) + self.assertEqual(self._rows("SELECT count(*) FROM fake7"), [(100000,)]) + self.assertEqual( + self._rows("SELECT DISTINCT _trace_uuid FROM fake7 ORDER BY 1"), + [("t0",), ("t1",), ("t2",), ("t3",)]) + finally: + for tp in tps: + tp.close() + + def test_cell_types_type_unknown_columns(self): + body = _result(["a", "b", "c"], [(1, 1.5, "s"), (2, 3, None)]) + tp = _FakeTp(_chunked(body)) + try: + ingest = self._ingest("fake6", ["a", "b", "c"], + ["STRING", "STRING", "STRING"]) + self.assertTrue(ingest.feed(f"tcp:127.0.0.1:{tp.port}", 0).ok) + ingest.finish() + finally: + tp.close() + types = [stream_ingest.type_from_cell_types(m) for m in ingest.cell_types] + self.assertEqual(types, ["INT64", "DOUBLE", "STRING"]) + self.assertEqual( + sorted(self._rows("SELECT a, b FROM fake6")), [(1, 1.5), (2, 3.0)]) + + def test_matches_real_trace_processor(self): + tp = TraceProcessor( + trace=trace_path(COLD_TRACE), + config=TraceProcessorConfig(bin_path=resolve_trace_processor_bin())) + try: + exp_rows = [ + (r.id, r.ts, r.dur, r.name, r.depth) for r in tp.query(_SLICE_SQL) + ] + ingest = self._ingest( + "real1", ["id", "ts", "dur", "name", "depth"], + ["INT64", "INT64", "INT64", "STRING", "INT64"], + sql=_SLICE_SQL) + conn = tp.http.conn + res = ingest.feed(f"tcp:{conn.host}:{conn.port}", 1) + self.assertEqual(res.status, stream_ingest.FEED_OK, res.message) + self.assertEqual(ingest.finish(), []) + finally: + tp.close() + self.assertGreater(len(exp_rows), 1000) + self.assertEqual(res.rows, len(exp_rows)) + got = self._rows("SELECT id, ts, dur, name, depth FROM real1 ORDER BY id") + self.assertEqual(got, sorted(exp_rows)) + + +class StreamingQueryTest(BigtraceServerTestCase): + """Unlimited (limit=0) trace queries go through streaming ingest.""" + + def _run(self, sql, limit=0, filters=None): + status, data, _ = self._request( + "POST", "/execute_bigtrace_query", { + "perfetto_sql": sql, + "limit": limit, + **({ + "trace_filters": filters + } if filters else {}) + }) + self.assertEqual(status, 200, data) + return data + + def _status(self, uid): + return self._request("GET", f"/query_executions/{uid}")[1] + + def _count(self, table): + status, res, _ = self._request("POST", "/execute_bigtrace_query", { + "engine": "duckdb", + "perfetto_sql": f"SELECT count(*) AS n FROM {table}", + }) + self.assertEqual(status, 200, res) + return int(res["rows"][0]["values"][0]) + + def test_unlimited_matches_limited_rows(self): + if self.server.duckdb.stream_api is None: + self.skipTest("streaming unavailable") + sql = "SELECT ts, dur, name FROM slice" + unlimited = self._run(sql) + limited = self._run(sql, limit=10**9) + ex_u = self._status(unlimited["queryUuid"]) + ex_l = self._status(limited["queryUuid"]) + self.assertEqual(ex_u["status"], "SUCCESS") + self.assertGreater(ex_u["processedRows"], 0) + self.assertEqual(ex_u["processedRows"], ex_l["processedRows"]) + self.assertEqual(self._count(ex_u["tableName"]), ex_u["processedRows"]) + types = {c["name"]: c["type"] for c in ex_u["schema"]} + self.assertEqual(types["ts"], "INT64") + self.assertEqual(types["name"], "STRING") + + def test_query_failing_everywhere_fails(self): + status, data, _ = self._request("POST", "/execute_bigtrace_query", { + "perfetto_sql": "SELECT * FROM no_such_table", + "limit": 0 + }) + self.assertEqual(status, 400, data) + execs = self._request("GET", "/query_executions")[1]["queryExecutions"] + last = next(e for e in execs if e["perfettoSql"].endswith("no_such_table")) + self.assertEqual(last["status"], "FAILED") + self.assertIn("no such table", last["errorMessage"]) + + def test_per_trace_errors_keep_other_traces(self): + # abs() overflows, failing the query, on traces with an even slice count. + sql = ("SELECT CASE WHEN (SELECT count(*) FROM slice) % 2 = 0 " + "THEN abs(-9223372036854775807 - 1) ELSE 1 END AS v") + status, data, _ = self._request("POST", "/execute_bigtrace_query", { + "perfetto_sql": sql, + "limit": 0 + }) + execs = self._request("GET", "/query_executions")[1]["queryExecutions"] + ex = next(e for e in execs if e["perfettoSql"] == sql) + failed = 0 + if ex.get("errorMessage"): + failed = int(ex["errorMessage"].split(" ")[0]) + self.assertEqual(ex.get("processedRows", 0) + failed, ex["totalTraces"]) + self.assertEqual(ex["status"], + "SUCCESS" if failed < ex["totalTraces"] else "FAILED", + (status, data)) + + def test_untyped_columns_get_types_from_later_traces(self): + # NULL on the cold trace, numbers elsewhere: the probe cannot type `v`. + sql = ("SELECT CASE WHEN (SELECT count(*) FROM slice) = " + "(SELECT count(*) FROM slice WHERE name GLOB '*cold*') " + "THEN NULL ELSE 7 END AS v") + data = self._run(sql) + ex = self._status(data["queryUuid"]) + self.assertEqual(ex["status"], "SUCCESS") + types = {c["name"]: c["type"] for c in ex["schema"]} + self.assertEqual(types["v"], "INT64") + + def test_cancel_stops_streaming_query(self): + if self.server.duckdb.stream_api is None: + self.skipTest("streaming unavailable") + status, data, _ = self._request("POST", "/execute_bigtrace_query_async", { + "perfetto_sql": "SELECT s1.ts FROM slice s1, slice s2", + "limit": 0, + }) + self.assertEqual(status, 200, data) + uid = data["queryUuid"] + time.sleep(1.0) + start = time.monotonic() + self._request("POST", f"/query_executions/{uid}:cancel", {}) + executor = self.server.executor + wait_for(lambda: not any(r.is_alive() for r in executor._runs), 30) + self.assertLess(time.monotonic() - start, 15) + self.assertEqual(self._status(uid)["status"], "CANCELLED") + + +if __name__ == "__main__": + unittest.main()
diff --git a/python/tools/code_format_clang.py b/python/tools/code_format_clang.py index 26e1e30..a46fe64 100755 --- a/python/tools/code_format_clang.py +++ b/python/tools/code_format_clang.py
@@ -19,6 +19,8 @@ from code_format_utils import CodeFormatterBase, run_code_formatters SKIP_PATHS = ( + 'python/perfetto/bigtrace/duckdb_ext/src/duckdb.h', + 'python/perfetto/bigtrace/duckdb_ext/src/duckdb_extension.h', 'src/perfetto_sql/syntaqlite/syntaqlite_perfetto.c', 'src/perfetto_sql/syntaqlite/syntaqlite_perfetto.h', )
diff --git a/python/tools/code_format_sql.py b/python/tools/code_format_sql.py index 9e2ba99..eb9e950 100755 --- a/python/tools/code_format_sql.py +++ b/python/tools/code_format_sql.py
@@ -21,6 +21,7 @@ from code_format_utils import CodeFormatterBase, run_code_formatters IGNORE_DIRS = [ + 'python/perfetto/bigtrace/duckdb_ext/sql/', 'test/trace_processor/diff_tests/', 'src/trace_processor/metrics/sql/', # The source of truth for the chrome stdlib is in chromium. Skip formatting
diff --git a/tools/bigtrace_server.py b/tools/bigtrace_server.py new file mode 100755 index 0000000..8c93df5 --- /dev/null +++ b/tools/bigtrace_server.py
@@ -0,0 +1,28 @@ +#!/usr/bin/env python3 +# Copyright (C) 2026 The Android Open Source Project +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""CLI entrypoint for Perfetto Bigtrace open source Python server.""" + +import os +import sys + +ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +PYTHON_DIR = os.path.join(ROOT_DIR, "python") +if PYTHON_DIR not in sys.path: + sys.path.insert(0, PYTHON_DIR) + +from perfetto.bigtrace.server import main + +if __name__ == "__main__": + main()
diff --git a/ui/build.mjs b/ui/build.mjs index 667f205..9cfcb82 100644 --- a/ui/build.mjs +++ b/ui/build.mjs
@@ -960,6 +960,36 @@ next(); }); + // Bigtrace dev builds proxy /v1/* and /trace/* to the local Bigtrace server + // (tools/bigtrace_server.py), so a browser that can only reach this port + // (e.g. over SSH forwarding) still reaches the backend. + const bigtracePort = Number(process.env.BIGTRACE_SERVER_PORT || 5001); + server.middlewares.use((req, res, next) => { + const isApi = req.url.startsWith('/v1/') || req.url.startsWith('/trace/'); + if (!cfg.bigtrace || !isApi) return next(); + const proxyReq = http.request( + { + hostname: '127.0.0.1', + port: bigtracePort, + path: req.url, + method: req.method, + headers: {...req.headers, host: `127.0.0.1:${bigtracePort}`}, + }, + (proxyRes) => { + res.writeHead(proxyRes.statusCode || 502, proxyRes.headers); + proxyRes.pipe(res); + }, + ); + proxyReq.on('error', (err) => { + res.statusCode = 502; + res.setHeader('Content-Type', 'application/json'); + res.end( + JSON.stringify({detail: `Bigtrace backend unavailable: ${err.message}`}), + ); + }); + req.pipe(proxyReq); + }); + // Fall back to serving files from outDistRootDir / outDistDir for anything // Vite hasn't claimed (wasm modules, fonts under /assets/, manifest.json, // etc.). In prod, frontend.css lives inside the versioned dir, so the @@ -968,10 +998,15 @@ // browser asks for /assets/Roboto.woff2. Try outDistDir as a second root // so those font requests succeed. server.middlewares.use((req, res, next) => { - const url = req.url.split('?', 1)[0]; + let url = req.url.split('?', 1)[0]; + if (url === '/bigtrace' || url === '/bigtrace/') { + url = '/bigtrace.html'; + } + url = url.replace('/assets/assets/', '/assets/'); // Vite handles JS/TS/CSS modules and HMR endpoints itself. if (url === '/' || url === '/index.html') return next(); const roots = [cfg.outDistRootDir, cfg.outDistDir]; + const tryRoot = (i) => { if (i >= roots.length) return next(); const root = roots[i];
diff --git a/ui/src/assets/bigtrace.scss b/ui/src/assets/bigtrace.scss index 5f90c89..bf3c647 100644 --- a/ui/src/assets/bigtrace.scss +++ b/ui/src/assets/bigtrace.scss
@@ -280,7 +280,6 @@ } .pf-bt-status-bar--running { - .pf-bt-status-bar-stat--traces, .pf-bt-status-bar-stat-label, .pf-bt-status-bar-stat-value { display: none; @@ -342,6 +341,66 @@ background-color: var(--pf-color-accent); transition: width 0.3s ease-out; will-change: width; + + &--indeterminate { + width: 40%; + animation: pf-bt-indeterminate-progress 1.2s ease-in-out infinite; + } +} + +@keyframes pf-bt-indeterminate-progress { + 0% { + transform: translateX(-100%); + } + 100% { + transform: translateX(350%); + } +} + +.pf-bt-subquery-bar { + display: flex; + align-items: center; + flex-wrap: wrap; + gap: 6px; + padding: 4px 12px; + background: var(--pf-color-background-secondary); + border-bottom: 1px solid var(--pf-color-border-secondary); + flex: 0 0 auto; +} + +.pf-bt-subquery-pill { + display: inline-flex; + align-items: center; + gap: 2px; + background: var(--pf-color-background); + border: 1px solid var(--pf-color-border-secondary); + border-radius: 4px; + padding: 1px 2px; + + &--active { + border-color: var(--pf-color-primary); + background: var(--pf-color-background-raised); + } + + &__num { + font-size: var(--pf-font-size-s); + color: var(--pf-color-text-muted); + padding: 0 4px 0 6px; + font-variant-numeric: tabular-nums; + } +} + +.pf-bt-subquery-rename-input { + border: none; + border-bottom: 1px solid var(--pf-color-primary); + outline: none; + background: transparent; + font-family: inherit; + font-size: var(--pf-font-size-s); + color: inherit; + padding: 1px 4px; + min-width: 72px; + max-width: 160px; } // ---------- Query history ---------- @@ -438,10 +497,11 @@ .pf-query-history__item-buttons { position: absolute; - top: 4px; + top: 2px; right: 4px; float: none; z-index: 1; + align-items: center; } // Status-colored left bar, meta card only (clashes with the pre's border). @@ -473,12 +533,12 @@ border-left-color: var(--pf-color-text-muted); } -// padding-right reserves space for the abs-positioned Open/Delete buttons. +// padding-right reserves space for the abs-positioned DuckDB/Open/Delete buttons. .pf-bt-history-item-meta { position: relative; background: var(--pf-color-background-tertiary); padding: 6px 8px; - padding-right: 56px; + padding-right: 76px; // Top + right only; the border-left status rule owns the left edge // (shorthand would shrink it to 1px). border-top: 1px solid var(--pf-color-border-secondary); @@ -517,7 +577,7 @@ display: flex; align-items: baseline; justify-content: space-between; - gap: 12px; + gap: 6px; width: 100%; > span {
diff --git a/ui/src/bigtrace/index.ts b/ui/src/bigtrace/index.ts index 12f5fdf..ecbd3c5 100644 --- a/ui/src/bigtrace/index.ts +++ b/ui/src/bigtrace/index.ts
@@ -56,6 +56,8 @@ 'object-src': [`'none'`], 'connect-src': [ `'self'`, + 'http://127.0.0.1:*', + 'http://localhost:*', 'https://autopush-brush-googleapis.corp.google.com', 'https://brush-googleapis.corp.google.com', ],
diff --git a/ui/src/bigtrace/layout/connection_button.ts b/ui/src/bigtrace/layout/connection_button.ts index 27d59f7..4b32248 100644 --- a/ui/src/bigtrace/layout/connection_button.ts +++ b/ui/src/bigtrace/layout/connection_button.ts
@@ -13,11 +13,12 @@ // limitations under the License. import m from 'mithril'; -import {Button, ButtonVariant} from '../../widgets/button'; +import {Button, ButtonGroup, ButtonVariant} from '../../widgets/button'; import {Callout} from '../../widgets/callout'; import {Intent} from '../../widgets/common'; import {Popup, PopupPosition} from '../../widgets/popup'; import {TextInput} from '../../widgets/text_input'; +import {getActiveQueryTabsState} from '../pages/query_page'; import {bigTraceSettingsStorage} from '../settings/bigtrace_settings_storage'; import { endpointStorage, @@ -45,6 +46,8 @@ view(): m.Children { const endpoint = getBigtraceEndpoint(); + const tabsState = getActiveQueryTabsState(); + const isDuckdb = tabsState?.activeEngine === 'duckdb'; // The endpoint ships with a default, so "set" doesn't mean "reachable" — // a backend that failed to answer counts as not connected, otherwise this // button claims a connection the app doesn't have. @@ -59,19 +62,33 @@ this.autoOpened = true; this.open = true; } + const toggleOpen = () => { + this.open = !this.open; + }; + const trigger = m(ButtonGroup, [ + m(Button, { + icon: connected ? 'cloud_done' : 'cloud_off', + label: endpointLabel(endpoint), + variant: ButtonVariant.Outlined, + title: connected + ? endpoint + : `Not connected to ${endpoint === '' ? 'a backend' : endpoint}`, + onclick: toggleOpen, + }), + m(Button, { + icon: isDuckdb ? 'database' : 'code', + label: isDuckdb ? 'DuckDB SQL' : 'PerfettoSQL', + rightIcon: 'arrow_drop_down', + variant: ButtonVariant.Outlined, + intent: isDuckdb ? Intent.Warning : Intent.Primary, + title: 'Switch between PerfettoSQL and DuckDB SQL', + onclick: toggleOpen, + }), + ]); return m( Popup, { - trigger: m(Button, { - icon: connected ? 'cloud_done' : 'cloud_off', - label: endpointLabel(endpoint), - title: connected - ? endpoint - : `Not connected to ${endpoint === '' ? 'a backend' : endpoint}`, - onclick: () => { - this.open = !this.open; - }, - }), + trigger, isOpen: this.open, onChange: (isOpen: boolean) => { this.open = isOpen; @@ -87,7 +104,40 @@ const setting = endpointStorage.get('bigtraceEndpoint'); if (setting === undefined) return null; const loadError = bigTraceSettingsStorage.execConfigLoadError; + const tabsState = getActiveQueryTabsState(); + const isDuckdb = tabsState.activeEngine === 'duckdb'; return m('.pf-bt-connection-panel', [ + m('.pf-bt-connection-panel__title', 'Query engine'), + m( + '.pf-bt-connection-panel__desc', + isDuckdb + ? 'Query materialized result tables in DuckDB.' + : 'Run PerfettoSQL across traces in the corpus.', + ), + m(ButtonGroup, [ + m(Button, { + label: 'PerfettoSQL', + icon: 'code', + variant: ButtonVariant.Outlined, + active: !isDuckdb, + intent: !isDuckdb ? Intent.Primary : Intent.None, + onclick: () => { + tabsState.setEngine('traces'); + this.open = false; + }, + }), + m(Button, { + label: 'DuckDB SQL', + icon: 'database', + variant: ButtonVariant.Outlined, + active: isDuckdb, + intent: isDuckdb ? Intent.Warning : Intent.None, + onclick: () => { + tabsState.setEngine('duckdb'); + this.open = false; + }, + }), + ]), m('.pf-bt-connection-panel__title', 'BigTrace backend'), m( '.pf-bt-connection-panel__desc',
diff --git a/ui/src/bigtrace/pages/editor_tab_view.ts b/ui/src/bigtrace/pages/editor_tab_view.ts index dcc504c..4a04b8f 100644 --- a/ui/src/bigtrace/pages/editor_tab_view.ts +++ b/ui/src/bigtrace/pages/editor_tab_view.ts
@@ -16,6 +16,7 @@ import {Box} from '../../widgets/box'; import {Button, ButtonVariant} from '../../widgets/button'; import {Callout} from '../../widgets/callout'; +import {Chip} from '../../widgets/chip'; import {Intent} from '../../widgets/common'; import {Editor} from '../../widgets/editor'; import {HotkeyGlyphs} from '../../widgets/hotkey_glyphs'; @@ -23,6 +24,7 @@ import {Stack, StackAuto} from '../../widgets/stack'; import {Switch} from '../../widgets/switch'; import {TextInput} from '../../widgets/text_input'; +import {showModal} from '../../widgets/modal'; import {InMemoryDataSource} from '../../components/widgets/datagrid/in_memory_data_source'; import {getBigtraceEndpoint} from '../settings/endpoint_storage'; import {BigtraceAsyncDataSource} from '../query/bigtrace_async_data_source'; @@ -30,12 +32,20 @@ import {BigtraceQueryClient} from '../query/bigtrace_query_client'; import type {QueryRunner} from '../query/query_runner'; import { + addSubQuery, type BigTraceEditorTab, type QueryTabsState, applyModeDefaults, - hasQueryText, + canRenameSubQuery, effectiveTabSettings, + getSubQueryTableName, + hasQueryText, + removeSubQuery, + renameSubQuery, + selectSubQuery, + syncActiveSubQueryToSlot, } from './query_tabs_state'; +import {formatCompact} from '../query/query_store'; import {openQuerySettingsModal} from './query_settings_modal'; import {renderResultsPanel} from './results_panel'; import {QueryLauncher} from './query_launcher'; @@ -68,7 +78,10 @@ // A tab with no configuration yet shows the launcher instead of the editor: // every query starts from a preset or a deliberate custom setup. A // configured tab shows it too while its Settings are open. - if (!tab.configured || tab.settingsSession !== undefined) { + if ( + tab.queryEngine !== 'duckdb' && + (!tab.configured || tab.settingsSession !== undefined) + ) { return m( '.pf-bt-editor-tab', m(QueryLauncher, { @@ -80,11 +93,12 @@ } return m('.pf-bt-editor-tab', [ - m(BigtraceSettingsBar, { - tab, - tabsState, - bindings: buildTabBindings(tab, tabsState), - }), + tab.queryEngine !== 'duckdb' && + m(BigtraceSettingsBar, { + tab, + tabsState, + bindings: buildTabBindings(tab, tabsState), + }), m(SplitPanel, { direction: 'vertical', initialSplit: {percent: 22}, @@ -157,14 +171,14 @@ getRowLimit: () => tab.limit, setRowLimit: (limit) => { tab.limit = limit; - tabsState.markDirty(); + tabsState.saveNow(); }, getMaterialize: () => tab.materialize, setMaterialize: (materialize) => { // Same path as the toolbar switch: caps the user never touched follow // the mode. applyModeDefaults(tab, materialize); - tabsState.markDirty(); + tabsState.saveNow(); }, getExperimentFilter: () => tab.experimentFilter, setExperimentFilter: (filter) => { @@ -175,7 +189,7 @@ setTraceLimit: (limit) => { if (limit >= 0) { tab.traceLimit = limit; - tabsState.markDirty(); + tabsState.saveNow(); } }, getTableTtlDays: () => tab.tableTtlDays, @@ -202,8 +216,107 @@ m.redraw(); } +// Checks if running `query` on `tab` would overwrite an existing result table +// (or this tab's previous run) and shows a Perfetto modal offering to either +// overwrite in place or create and run in a new tab with `<name>_1` (editable). +async function runTabWithOverwriteCheck( + tab: BigTraceEditorTab, + tabsState: QueryTabsState, + runner: QueryRunner, + query: string, +): Promise<void> { + const isMulti = tab.queryEngine !== 'duckdb' && tab.subQueries.length > 1; + const activeTables: Array<{name: string; slotIndex?: number}> = isMulti + ? tab.subQueries + .map((q, i) => ({ + hasSql: hasQueryText(q.sql), + name: getSubQueryTableName(q, i), + slotIndex: i, + })) + .filter((x) => x.hasSql) + : [{name: tab.tableName.trim() || tab.title.trim()}]; + + const endpoint = getBigtraceEndpoint().trim(); + let collidingTable = ''; + let suggested = ''; + let collidingSlotIndex: number | undefined; + + if (endpoint !== '') { + const client = new BigtraceQueryClient(endpoint); + for (const entry of activeTables) { + const name = entry.name.replace(/[^A-Za-z0-9_]/g, '_'); + if (!name) continue; + try { + const check = await client.checkTableExists(name); + if (check.exists) { + collidingTable = check.resolvedTableName || name; + suggested = check.suggestedTableName ?? `${collidingTable}_1`; + collidingSlotIndex = entry.slotIndex; + break; + } + } catch { + // Fall back to runner if check fails + } + } + } + + if (!collidingTable) { + await runner.run(tab, query); + return; + } + + let newTableName = suggested; + let choice = 'cancel' as 'cancel' | 'overwrite' | 'new_tab' | string; + + await showModal({ + title: `Table "${collidingTable}" already exists`, + content: () => + m( + Stack, + {orientation: 'vertical', gap: 'normal'}, + m( + 'div', + 'Running in this tab will overwrite it, or you can create a new tab with the suggested table name below:', + ), + m(TextInput, { + value: newTableName, + placeholder: suggested, + autofocus: true, + onInput: (val: string) => { + newTableName = val; + }, + }), + ), + buttons: [ + {text: 'Cancel', action: () => (choice = 'cancel')}, + {text: 'Overwrite', action: () => (choice = 'overwrite')}, + { + text: 'Create New Tab', + primary: true, + action: () => (choice = 'new_tab'), + }, + ], + }); + + if (choice === 'overwrite') { + await runner.run(tab, query); + } else if (choice === 'new_tab') { + const chosen = newTableName.trim() || suggested; + const newTab = tabsState.cloneTab( + tab.id, + chosen, + query, + collidingSlotIndex, + ); + m.redraw(); + if (newTab) { + await runner.run(newTab, query); + } + } +} + // --------------------------------------------------------------------------- -// Editor panel: toolbar (Run/Cancel + limit + Persistent) and the editor. +// Editor panel: toolbar (Run/Cancel + limit) // --------------------------------------------------------------------------- function renderEditorPanel( @@ -212,6 +325,14 @@ runner: QueryRunner, useBigtraceBackend: boolean, ): m.Children { + const isDuckdb = tab.queryEngine === 'duckdb'; + const isMultiQuery = !isDuckdb && tab.subQueries.length > 1; + const canRun = isMultiQuery + ? tab.subQueries.some((q, i) => + hasQueryText(i === tab.activeSubQueryIndex ? tab.editorText : q.sql), + ) + : hasQueryText(tab.editorText); + return m('.pf-bt-query-page__editor-panel', [ m(Box, {className: 'pf-bt-query-page__toolbar'}, [ m(Stack, {orientation: 'horizontal'}, [ @@ -224,12 +345,24 @@ onclick: () => runner.cancel(tab), }) : m(Button, { - label: 'Run Query', + label: isMultiQuery + ? `Run ${tab.subQueries.length} Queries` + : 'Run', icon: 'play_arrow', intent: Intent.Primary, variant: ButtonVariant.Filled, - disabled: !hasQueryText(tab.editorText), - onclick: () => runner.run(tab, tab.editorText), + disabled: !canRun, + onclick: () => { + if (tab.subQueries.length > 0) { + syncActiveSubQueryToSlot(tab); + } + void runTabWithOverwriteCheck( + tab, + tabsState, + runner, + tab.editorText, + ); + }, }), m( Stack, @@ -238,18 +371,20 @@ m(HotkeyGlyphs, {hotkey: 'Mod+Enter'}), ), m('span.pf-bt-divider', {'aria-hidden': 'true'}), - // Icon-only to keep the toolbar lean; the editor binds the same chord. m(Button, { icon: 'format_align_left', title: 'Format query (Alt+Shift+F)', disabled: !hasQueryText(tab.editorText), onclick: () => void formatTabQuery(tab, tabsState, tab.editorText), }), + isDuckdb && m(Chip, {label: 'DuckDB Mode', intent: Intent.Warning}), m(StackAuto), - useBigtraceBackend && renderRunControls(tab, tabsState), + !isDuckdb && useBigtraceBackend && renderRunControls(tab, tabsState), ]), ]), - tab.editorText.includes('"') && + isMultiQuery && renderSubQueryBar(tab, tabsState), + !isDuckdb && + tab.editorText.includes('"') && m( Callout, {icon: 'warning', intent: Intent.None}, @@ -262,26 +397,158 @@ language: 'perfetto-sql', autofocus: true, onSave: () => {}, - // Alt+Shift+F (option+shift+F on Mac); listed in the help modal. onFormat: (text: string) => formatTabQuery(tab, tabsState, text), onUpdate: (text: string) => { tab.editorText = text; + if ( + tab.subQueries.length > 0 && + tab.subQueries[tab.activeSubQueryIndex] !== undefined + ) { + tab.subQueries[tab.activeSubQueryIndex].sql = text; + } tabsState.markDirty(); }, - onExecute: (query: string) => runner.run(tab, query), + onExecute: (query: string) => { + if (tab.subQueries.length > 0) { + syncActiveSubQueryToSlot(tab); + } + void runTabWithOverwriteCheck(tab, tabsState, runner, query); + }, }), ]); } -// Mode switch, the row cap, and the gear opening the query-settings modal -// (the trace cap and every non-trace-selection setting). The gear is always -// there: the trace cap is a universal request field, whatever the backend -// declares. +let editingSubQueryKey: string | undefined; +let editingSubQueryValue = ''; + +function commitSubQueryRename( + tab: BigTraceEditorTab, + idx: number, + tabsState: QueryTabsState, +): void { + if (editingSubQueryKey === undefined) return; + editingSubQueryKey = undefined; + renameSubQuery(tab, idx, editingSubQueryValue); + tabsState.markDirty(); +} + +function renderSubQueryBar( + tab: BigTraceEditorTab, + tabsState: QueryTabsState, +): m.Children { + return m( + '.pf-bt-subquery-bar', + tab.subQueries.map((slot, idx) => { + const isActive = idx === tab.activeSubQueryIndex; + const resolvedTbl = getSubQueryTableName(slot, idx); + const exec = slot.execution; + const rows = exec?.processedRows ?? slot.queryResult?.totalRowCount; + const rowsText = + rows !== undefined && rows > 0 ? ` (${formatCompact(rows)})` : ''; + const renamable = canRenameSubQuery(tab, slot); + const editKey = `${tab.id}:${slot.id}`; + const isEditing = renamable && editingSubQueryKey === editKey; + return m( + 'span.pf-bt-subquery-pill', + { + key: slot.id, + className: isActive ? 'pf-bt-subquery-pill--active' : undefined, + }, + isEditing + ? [ + m('span.pf-bt-subquery-pill__num', `#${idx + 1}`), + m('input.pf-bt-subquery-rename-input', { + type: 'text', + value: editingSubQueryValue, + oncreate: (vnode: m.VnodeDOM) => { + const el = vnode.dom as HTMLInputElement; + el.focus(); + el.select(); + }, + onclick: (e: Event) => e.stopPropagation(), + oninput: (e: InputEvent) => { + editingSubQueryValue = (e.target as HTMLInputElement).value; + }, + onblur: () => { + commitSubQueryRename(tab, idx, tabsState); + }, + onkeydown: (e: KeyboardEvent) => { + if (e.key === 'Enter') { + e.preventDefault(); + e.stopPropagation(); + commitSubQueryRename(tab, idx, tabsState); + } else if (e.key === 'Escape') { + e.preventDefault(); + e.stopPropagation(); + editingSubQueryKey = undefined; + } + }, + }), + ] + : m(Button, { + compact: true, + active: isActive, + variant: isActive ? ButtonVariant.Filled : ButtonVariant.Minimal, + label: `#${idx + 1} ${resolvedTbl}${rowsText}`, + title: renamable + ? 'Double-click to rename result table' + : 'Result table name cannot be changed after running', + onclick: () => { + selectSubQuery(tab, idx); + tabsState.markDirty(); + }, + ondblclick: renamable + ? () => { + selectSubQuery(tab, idx); + editingSubQueryKey = editKey; + editingSubQueryValue = resolvedTbl; + tabsState.markDirty(); + } + : undefined, + }), + m(Button, { + compact: true, + icon: 'close', + title: `Remove query #${idx + 1}`, + disabled: tab.isLoading, + onclick: () => { + removeSubQuery(tab, idx); + tabsState.markDirty(); + }, + }), + ); + }), + m(Button, { + compact: true, + icon: 'add', + label: 'Add Query', + disabled: tab.isLoading, + onclick: () => { + addSubQuery(tab); + tabsState.markDirty(); + }, + }), + ); +} + +// Persistent switch, row cap, and settings. function renderRunControls( tab: BigTraceEditorTab, tabsState: QueryTabsState, ): m.Children { return [ + tab.queryEngine !== 'duckdb' && + tab.subQueries.length <= 1 && + m(Button, { + icon: 'add', + label: 'Add Query', + compact: true, + disabled: tab.isLoading, + onclick: () => { + addSubQuery(tab); + tabsState.markDirty(); + }, + }), m(Switch, { label: 'Persistent', title: @@ -300,13 +567,18 @@ type: 'number', className: 'pf-bt-limit-input', value: String(tab.limit), - title: 'Maximum rows this query returns.', + title: 'Maximum rows this query returns (0 = unlimited).', disabled: tab.isLoading, onInput: (value: string) => { + if (value.trim() === '') { + tab.limit = 0; + tabsState.saveNow(); + return; + } const newLimit = parseInt(value, 10); if (!isNaN(newLimit) && newLimit >= 0) { tab.limit = newLimit; - tabsState.markDirty(); + tabsState.saveNow(); } }, }),
diff --git a/ui/src/bigtrace/pages/query_page.ts b/ui/src/bigtrace/pages/query_page.ts index 47c4a5d..cdbf572 100644 --- a/ui/src/bigtrace/pages/query_page.ts +++ b/ui/src/bigtrace/pages/query_page.ts
@@ -22,20 +22,32 @@ import {bigTraceSettingsStorage} from '../settings/bigtrace_settings_storage'; import {showModal} from '../../widgets/modal'; import {EditorTabView} from './editor_tab_view'; -import {QueryTabsState} from './query_tabs_state'; +import { + canRenameTab, + QueryTabsState, + stageDuckdbQuery, +} from './query_tabs_state'; interface QueryPageAttrs { useBigtraceBackend?: boolean; } -// Lets the global keyboard command reach the active QueryPage instance +// Lets the global keyboard command and Topbar reach the active QueryPage instance // (same pattern as sidebarToggleFn in index.ts). export let queryRightSidebarToggleFn: (() => void) | undefined; +let activeQueryTabsState: QueryTabsState | undefined; + +export function getActiveQueryTabsState(): QueryTabsState { + if (activeQueryTabsState === undefined) { + activeQueryTabsState = new QueryTabsState(); + } + return activeQueryTabsState; +} export class QueryPage implements m.ClassComponent<QueryPageAttrs> { private useBigtraceBackend = false; private sidebarVisible = true; - private readonly tabsState = new QueryTabsState(); + private readonly tabsState = getActiveQueryTabsState(); private historyRefreshSignal = 0; private readonly runner = new QueryRunner({ onHistoryChanged: () => { @@ -46,6 +58,7 @@ oninit({attrs}: m.Vnode<QueryPageAttrs>) { this.useBigtraceBackend = attrs.useBigtraceBackend || false; + activeQueryTabsState = this.tabsState; queryRightSidebarToggleFn = () => { this.sidebarVisible = !this.sidebarVisible; m.redraw(); @@ -56,12 +69,24 @@ } view() { - const editorTabs: TabsTab[] = this.tabsState.tabs.map((tab) => ({ + activeQueryTabsState = this.tabsState; + const visibleTabs = this.tabsState.getVisibleTabs(); + const editorTabs: TabsTab[] = visibleTabs.map((tab) => ({ key: tab.id, - title: tab.title, + title: canRenameTab(tab) + ? tab.title + : m( + 'span', + {title: 'Result table name cannot be changed after running'}, + tab.title, + ), // Spinner on in-flight tabs so switching away doesn't hide the run. - leftIcon: tab.isLoading ? m(Spinner) : 'code', - closeButton: this.tabsState.tabs.length > 1, + leftIcon: tab.isLoading + ? m(Spinner) + : tab.queryEngine === 'duckdb' + ? 'database' + : 'code', + closeButton: visibleTabs.length > 1, content: m(EditorTabView, { tab, tabsState: this.tabsState, @@ -81,9 +106,9 @@ }, onTabRename: (key, newTitle) => this.tabsState.renameTab(key, newTitle), onTabClose: async (key) => { - // closeTab is a no-op with one tab left; bail before the confirm - // so middle-click doesn't dead-end. - if (this.tabsState.tabs.length <= 1) return; + // closeTab is a no-op with one tab left in the current engine; bail + // before the confirm so middle-click doesn't dead-end. + if (this.tabsState.getVisibleTabs().length <= 1) return; // Confirm only for ephemeral queries — closing loses the results. // Persistent queries keep running and can be reopened from History. const tab = this.tabsState.tabs.find((t) => t.id === key); @@ -141,7 +166,19 @@ forceNew?: boolean, limit?: number, startTime?: number, + queryEngine?: 'traces' | 'duckdb', ) => { + if (!uuid) { + // A table opened in DuckDB: switch mode with the query staged, but + // don't run it (running would also add a History entry). + stageDuckdbQuery(this.tabsState, query, limit); + return; + } + const alreadyOpen = + !forceNew && this.tabsState.tabs.some((t) => t.queryUuid === uuid); + if (queryEngine) { + this.tabsState.activeEngine = queryEngine; + } const tab = this.tabsState.addNewTab( undefined, query, @@ -150,8 +187,15 @@ materialize, forceNew, ); + if (queryEngine) { + tab.queryEngine = queryEngine; + this.tabsState.activeEngine = queryEngine; + } this.tabsState.activeTabId = tab.id; this.tabsState.markDirty(); + if (alreadyOpen && tab.dataSource !== undefined) { + return; + } if (startTime !== undefined && tab.execution) { tab.execution.startTime = startTime; }
diff --git a/ui/src/bigtrace/pages/query_tabs_state.ts b/ui/src/bigtrace/pages/query_tabs_state.ts index 8d60830..d2c7400 100644 --- a/ui/src/bigtrace/pages/query_tabs_state.ts +++ b/ui/src/bigtrace/pages/query_tabs_state.ts
@@ -17,6 +17,7 @@ import type {Row as DataGridRow} from '../../trace_processor/query_result'; import {debounce} from '../../base/rate_limiters'; import {shortUuid} from '../../base/uuid'; +import {BigtraceAsyncDataSource} from '../query/bigtrace_async_data_source'; import type { BigtraceQueryClient, ExperimentFilterSpec, @@ -27,6 +28,37 @@ import type {SettingCategory, SettingFilter} from '../settings/settings_types'; import {bigTraceSettingsStorage} from '../settings/bigtrace_settings_storage'; +const RESERVED_SQL_WORDS = new Set([ + 'select', + 'where', + 'group', + 'order', + 'limit', + 'join', + 'left', + 'right', + 'inner', + 'outer', + 'cross', + 'natural', + 'using', + 'on', + 'with', + 'union', + 'from', + 'table', +]); + +export function deriveTableNameFromSql(sql: string): string | undefined { + if (!sql) return undefined; + const match = sql.match( + /\bFROM\s+([a-zA-Z_][a-zA-Z0-9_]*)(?:\.([a-zA-Z_][a-zA-Z0-9_]*))?\b/i, + ); + if (!match) return undefined; + const name = (match[2] ?? match[1]).toLowerCase(); + return RESERVED_SQL_WORDS.has(name) ? undefined : name; +} + const QUERY_TABS_STORAGE_KEY = 'bigtraceQueryTabs'; const DEFAULT_SQL = ''; @@ -79,10 +111,16 @@ } export function traceUuidsActive(tab: BigTraceEditorTab): boolean { + const entry = tab.querySettings.find( + (s) => s.settingId === TRACE_UUIDS_SETTING_ID, + ); + const inLauncher = !tab.configured || tab.settingsSession !== undefined; + const hasEntry = + entry !== undefined && (inLauncher || entry.values.length > 0); return traceUuidsState( traceUuidsDeclared(), tab.disabledSettings.includes(TRACE_UUIDS_SETTING_ID), - tab.querySettings.some((s) => s.settingId === TRACE_UUIDS_SETTING_ID), + hasEntry, ); } @@ -388,15 +426,261 @@ if (!keep && tab.settingsSession !== undefined) { restoreTabConfig(tab, tab.settingsSession.before); } + const uuidEntry = tab.querySettings.find( + (s) => s.settingId === TRACE_UUIDS_SETTING_ID, + ); + if (uuidEntry !== undefined && uuidEntry.values.length === 0) { + setTraceUuidsActive(tab, false); + } tab.settingsSession = undefined; tab.configured = true; } +// Puts `query` in a DuckDB-mode tab without running it. Reuses the active tab +// when it is idle and empty, a tab already holding exactly `query`, or else +// opens a new one, so the user's SQL is never overwritten. +export function stageDuckdbQuery( + tabsState: QueryTabsState, + query: string, + limit?: number, +): BigTraceEditorTab { + const active = tabsState.getActiveTab(); + const reuseActive = + active !== undefined && + !active.isLoading && + !hasQueryText(active.editorText); + tabsState.activeEngine = 'duckdb'; + const tab = reuseActive + ? active + : tabsState.addNewTab(undefined, query, limit); + tab.editorText = query; + if (limit !== undefined) tab.limit = limit; + tab.queryEngine = 'duckdb'; + tab.settingsSession = undefined; + tab.configured = true; + tabsState.activeTabId = tab.id; + tabsState.markDirty(); + return tab; +} + +export function defaultTableName(): string { + const suffix = shortUuid() + .slice(1) + .replace(/[^a-zA-Z0-9]/g, '') + .slice(0, 6) + .toLowerCase(); + return `tbl_${suffix || '1'}`; +} + +export interface SubQuerySlot { + readonly id: string; + sql: string; + tableName: string; + queryUuid?: string; + queryResult?: QueryResponse; + dataSource?: DataSource; + execution?: QueryExecution; +} + +export interface StoredSubQuerySlot { + readonly id: string; + readonly sql: string; + readonly tableName: string; + readonly queryUuid?: string; +} + +export function loadSlotIntoTab(tab: BigTraceEditorTab, index: number): void { + if (index < 0 || index >= tab.subQueries.length) return; + tab.activeSubQueryIndex = index; + const slot = tab.subQueries[index]; + tab.editorText = slot.sql; + tab.tableName = slot.tableName; + tab.queryUuid = slot.queryUuid; + tab.queryResult = slot.queryResult; + tab.dataSource = slot.dataSource; + tab.execution = + slot.execution ?? + (slot.queryUuid + ? queryStore.getOrCreate(slot.queryUuid, { + materialized: tab.materialize, + queryEngine: tab.queryEngine, + perfettoSql: slot.sql, + tableName: slot.tableName, + }) + : undefined); + if (slot.queryUuid && !slot.dataSource && tab.queryClient) { + slot.dataSource = new BigtraceAsyncDataSource( + slot.queryUuid, + tab.queryClient, + () => slot.execution?.processedRows ?? 0, + tab.lifecycle.signal, + () => slot.execution?.schema, + ); + tab.dataSource = slot.dataSource; + if (!slot.queryResult) { + slot.queryResult = makeQueryResponse(slot.sql, { + totalRowCount: slot.execution?.processedRows ?? 0, + }); + tab.queryResult = slot.queryResult; + } + } +} + +export function syncActiveSubQueryToSlot(tab: BigTraceEditorTab): void { + if (tab.subQueries.length === 0) return; + const idx = Math.max( + 0, + Math.min(tab.activeSubQueryIndex, tab.subQueries.length - 1), + ); + const slot = tab.subQueries[idx]; + slot.sql = tab.editorText; + slot.tableName = tab.tableName; + slot.queryUuid = tab.queryUuid; + slot.queryResult = tab.queryResult; + slot.dataSource = tab.dataSource; + slot.execution = tab.execution; +} + +export function selectSubQuery(tab: BigTraceEditorTab, index: number): void { + if (index < 0 || index >= tab.subQueries.length) return; + syncActiveSubQueryToSlot(tab); + loadSlotIntoTab(tab, index); +} + +export function addSubQuery(tab: BigTraceEditorTab): void { + if (tab.subQueries.length === 0) { + const firstTable = tab.tableName.trim() || defaultTableName(); + tab.tableName = firstTable; + tab.subQueries.push({ + id: shortUuid(), + sql: tab.editorText, + tableName: firstTable, + queryUuid: tab.queryUuid, + queryResult: tab.queryResult, + dataSource: tab.dataSource, + execution: tab.execution, + }); + tab.activeSubQueryIndex = 0; + } else { + syncActiveSubQueryToSlot(tab); + } + tab.subQueries.push({ + id: shortUuid(), + sql: '', + tableName: defaultTableName(), + }); + loadSlotIntoTab(tab, tab.subQueries.length - 1); +} + +export function removeSubQuery(tab: BigTraceEditorTab, index: number): void { + if (index < 0 || index >= tab.subQueries.length) return; + syncActiveSubQueryToSlot(tab); + tab.subQueries.splice(index, 1); + if (tab.subQueries.length <= 1) { + const remaining: SubQuerySlot | undefined = tab.subQueries[0]; + tab.subQueries = []; + tab.activeSubQueryIndex = 0; + if (remaining !== undefined) { + tab.editorText = remaining.sql; + tab.tableName = remaining.tableName; + tab.queryUuid = remaining.queryUuid; + tab.queryResult = remaining.queryResult; + tab.dataSource = remaining.dataSource; + tab.execution = remaining.execution; + } + return; + } + const nextIdx = Math.min(tab.activeSubQueryIndex, tab.subQueries.length - 1); + loadSlotIntoTab(tab, nextIdx); +} + +export function getSubQueryTableName( + slot: { + sql?: string; + editorText?: string; + tableName?: string; + execution?: QueryExecution; + }, + index: number, +): string { + const custom = slot.tableName?.trim(); + if (custom) return custom; + if (slot.execution?.tableName) return slot.execution.tableName; + const sql = slot.sql ?? slot.editorText ?? ''; + const derived = deriveTableNameFromSql(sql); + if (derived) return derived; + return `result_${index + 1}`; +} + +export function buildMultiQuerySpecs( + subQueries: ReadonlyArray<SubQuerySlot>, +): Array<{slotId: string; perfettoSql: string; tableName: string}> { + const active = subQueries.filter((q) => hasQueryText(q.sql)); + const used = new Set<string>(); + return active.map((q, idx) => { + const base = + q.tableName.trim() || + deriveTableNameFromSql(q.sql) || + `result_${idx + 1}`; + let name = base; + let count = 1; + while (used.has(name.toLowerCase())) { + name = `${base}_${++count}`; + } + used.add(name.toLowerCase()); + return { + slotId: q.id, + perfettoSql: q.sql, + tableName: name, + }; + }); +} + +// A tab's title / result table name can only be renamed before a query runs. +export function canRenameTab(tab: BigTraceEditorTab): boolean { + return ( + !tab.isLoading && + tab.queryUuid === undefined && + tab.queryResult === undefined + ); +} + +// A sub-query slot's result table name can only be renamed before it runs. +export function canRenameSubQuery( + tab: BigTraceEditorTab, + slot: SubQuerySlot, +): boolean { + return ( + !tab.isLoading && + slot.queryUuid === undefined && + slot.queryResult === undefined + ); +} + +export function renameSubQuery( + tab: BigTraceEditorTab, + index: number, + newName: string, +): void { + if (index < 0 || index >= tab.subQueries.length) return; + const slot = tab.subQueries[index]; + const trimmed = newName.trim(); + if (trimmed === '' || !canRenameSubQuery(tab, slot)) return; + slot.tableName = trimmed; + if (index === tab.activeSubQueryIndex) { + tab.tableName = trimmed; + } +} + // Mutated in-place by the runner; only QueryTabsState creates/destroys. export interface BigTraceEditorTab { readonly id: string; title: string; editorText: string; + queryEngine: 'traces' | 'duckdb'; + tableName: string; + subQueries: SubQuerySlot[]; + activeSubQueryIndex: number; limit: number; // Cap on how many traces the run fans out to; a top-level request field // like `limit`, defaulted per mode and moved with it while untouched. @@ -459,6 +743,10 @@ readonly id: string; readonly title: string; readonly editorText: string; + readonly queryEngine?: 'traces' | 'duckdb'; + readonly tableName?: string; + readonly subQueries?: ReadonlyArray<StoredSubQuerySlot>; + readonly activeSubQueryIndex?: number; readonly limit: number; readonly traceLimit?: number; readonly tableTtlDays?: number; @@ -482,14 +770,17 @@ interface StoredState { readonly tabs: ReadonlyArray<StoredTab>; readonly activeTabId?: string; + readonly activeEngine?: 'traces' | 'duckdb'; } // Manages editor tabs + localStorage persistence across page reloads. export class QueryTabsState { tabs: BigTraceEditorTab[] = []; activeTabId = ''; + activeEngine: 'traces' | 'duckdb' = 'traces'; + private lastActiveTabByEngine: Partial<Record<'traces' | 'duckdb', string>> = + {}; - private tabCounter = 0; private readonly debouncedSave = debounce(() => this.saveToStorage(), 1000); constructor() { @@ -502,10 +793,47 @@ this.debouncedSave(); } + saveNow(): void { + this.saveToStorage(); + } + getActiveTab(): BigTraceEditorTab | undefined { return this.tabs.find((t) => t.id === this.activeTabId); } + getVisibleTabs(): BigTraceEditorTab[] { + return this.tabs.filter((t) => t.queryEngine === this.activeEngine); + } + + setEngine(engine: 'traces' | 'duckdb'): void { + const current = this.getActiveTab(); + if (this.activeEngine === engine && current?.queryEngine === engine) { + return; + } + if (current !== undefined) { + this.lastActiveTabByEngine[current.queryEngine] = current.id; + } + this.activeEngine = engine; + const visible = this.getVisibleTabs(); + if (visible.length === 0) { + this.addNewTab( + undefined, + DEFAULT_SQL, + undefined, + undefined, + undefined, + true, + ); + return; + } + const rememberedId = this.lastActiveTabByEngine[engine]; + const remembered = visible.find((t) => t.id === rememberedId); + const target = remembered ?? visible[visible.length - 1]; + this.activeTabId = target.id; + this.lastActiveTabByEngine[engine] = target.id; + this.markDirty(); + } + // Create and activate. Without `forceNew`, reactivates an existing tab // matching by `queryUuid` (preferred) or `initialQuery`. addNewTab( @@ -518,25 +846,34 @@ stored?: Partial<StoredTab>, traceLimit?: number, ): BigTraceEditorTab { + const queryEngine: 'traces' | 'duckdb' = + stored?.queryEngine === 'duckdb' || stored?.queryEngine === 'traces' + ? stored.queryEngine + : this.activeEngine; + if (!forceNew) { const existingTab = this.tabs.find((t) => { if (queryUuid && t.queryUuid === queryUuid) return true; - if (!queryUuid && initialQuery && t.editorText === initialQuery) { + if ( + !queryUuid && + initialQuery && + t.editorText === initialQuery && + t.queryEngine === queryEngine + ) { return true; } return false; }); if (existingTab) { + this.activeEngine = existingTab.queryEngine; this.activeTabId = existingTab.id; + this.lastActiveTabByEngine[existingTab.queryEngine] = existingTab.id; this.markDirty(); return existingTab; } } - // Restored tabs bring their own title (possibly one the user typed); - // everything else is "Query N". - const derivedTitle = title; // Seed the per-tab settings snapshot. Restored tabs use the persisted // one; history-reopen tabs start empty (the runner rehydrates from // /query_executions/{uuid}); fresh tabs start from the backend defaults. @@ -581,10 +918,62 @@ .getAllSettings() .filter((s) => s.isDisabled()) .map((s) => s.id); + + const subQueries: SubQuerySlot[] = Array.isArray(stored?.subQueries) + ? stored.subQueries.map((s: StoredSubQuerySlot) => ({ + id: typeof s.id === 'string' && s.id !== '' ? s.id : shortUuid(), + sql: typeof s.sql === 'string' ? s.sql : '', + tableName: + typeof s.tableName === 'string' && s.tableName.trim() !== '' + ? s.tableName + : defaultTableName(), + queryUuid: typeof s.queryUuid === 'string' ? s.queryUuid : undefined, + })) + : []; + const activeSubQueryIndex = + typeof stored?.activeSubQueryIndex === 'number' && + stored.activeSubQueryIndex >= 0 && + stored.activeSubQueryIndex < subQueries.length + ? stored.activeSubQueryIndex + : 0; + + let initialEditorText = initialQuery ?? ''; + let initialTableName = + (typeof stored?.tableName === 'string' && stored.tableName.trim() !== '' + ? stored.tableName.trim() + : undefined) ?? + (typeof title === 'string' && title.trim() !== '' + ? title.trim() + : undefined) ?? + defaultTableName(); + let initialQueryUuid = queryUuid; + const activeSlot = subQueries[activeSubQueryIndex]; + if (activeSlot !== undefined) { + initialEditorText = activeSlot.sql; + initialTableName = activeSlot.tableName; + if (activeSlot.queryUuid) { + initialQueryUuid = activeSlot.queryUuid; + } + } + for (const slot of subQueries) { + if (slot.queryUuid) { + slot.execution = queryStore.getOrCreate(slot.queryUuid, { + materialized: isPersistent, + queryEngine, + perfettoSql: slot.sql, + tableName: slot.tableName, + }); + } + } + const tab: BigTraceEditorTab = { id: shortUuid(), - title: derivedTitle || this.nextTabName(), - editorText: initialQuery ?? '', + title: initialTableName, + editorText: initialEditorText, + queryEngine, + tableName: initialTableName, + subQueries, + activeSubQueryIndex, // No caller-supplied caps: take the ones that fit the execution mode. limit: limit ?? modeDefaults(isPersistent).rowLimit, traceLimit: @@ -618,18 +1007,28 @@ pollGeneration: 0, // Tabs predating the launcher have no flag; treat them as configured so // a reload never drops the user back into the picker. - configured: isFromStorage - ? (stored?.configured ?? true) - : isFromHistory + configured: + queryEngine === 'duckdb' ? true - : false, + : isFromStorage + ? (stored?.configured ?? true) + : isFromHistory + ? true + : false, lastPresetId: isFromStorage ? stored?.lastPresetId : undefined, }; - tab.execution = queryStore.getOrCreate(queryUuid || tab.id, { + tab.execution = queryStore.getOrCreate(initialQueryUuid || tab.id, { materialized: tab.materialize, + queryEngine: tab.queryEngine, }); + const restoredActiveSlot = subQueries[activeSubQueryIndex]; + if (restoredActiveSlot !== undefined) { + restoredActiveSlot.execution = tab.execution; + } this.tabs.push(tab); + this.activeEngine = tab.queryEngine; this.activeTabId = tab.id; + this.lastActiveTabByEngine[tab.queryEngine] = tab.id; this.markDirty(); return tab; } @@ -637,17 +1036,44 @@ // Clone a tab: same query and configuration, no results and no queryUuid, // so running it creates its own execution instead of colliding with the // original's in History. - cloneTab(tabId: string): BigTraceEditorTab | undefined { + cloneTab( + tabId: string, + tableName?: string, + overrideQuery?: string, + targetSubQueryIndex?: number, + ): BigTraceEditorTab | undefined { const src = this.tabs.find((t) => t.id === tabId); if (src === undefined) return undefined; + const trimmedName = tableName?.trim() || undefined; + const targetIdx = targetSubQueryIndex ?? src.activeSubQueryIndex; const clone = this.addNewTab( - undefined, // "Query N", like any new tab - src.editorText, + src.subQueries.length > 1 ? src.title : trimmedName, + overrideQuery ?? src.editorText, src.limit, undefined, src.materialize, true, // forceNew { + tableName: trimmedName, + queryEngine: src.queryEngine, + subQueries: src.subQueries.map((s, i) => { + const isTarget = i === targetIdx; + const isCurrent = i === src.activeSubQueryIndex; + const tbl = + isTarget && trimmedName + ? trimmedName + : s.queryUuid === undefined && s.tableName.trim() !== '' + ? s.tableName.trim() + : defaultTableName(); + const sql = + isCurrent && overrideQuery !== undefined ? overrideQuery : s.sql; + return { + id: shortUuid(), + sql, + tableName: tbl, + }; + }), + activeSubQueryIndex: src.activeSubQueryIndex, querySettings: src.querySettings, disabledSettings: src.disabledSettings, traceFilters: src.traceFilters, @@ -663,16 +1089,21 @@ traceLimit: src.traceLimit, tableTtlDays: src.tableTtlDays, }, + src.traceLimit, ); - this.markDirty(); + this.saveToStorage(); return clone; } closeTab(tabId: string): void { - if (this.tabs.length <= 1) return; const index = this.tabs.findIndex((t) => t.id === tabId); if (index === -1) return; const tabToClose = this.tabs[index]; + const sameEngineTabs = this.tabs.filter( + (t) => t.queryEngine === tabToClose.queryEngine, + ); + if (sameEngineTabs.length <= 1) return; + const engineIndex = sameEngineTabs.findIndex((t) => t.id === tabId); if (tabToClose.pollInterval !== undefined) { window.clearTimeout(tabToClose.pollInterval); tabToClose.pollInterval = undefined; @@ -681,19 +1112,30 @@ tabToClose.activeRequest?.abort(); tabToClose.lifecycle.abort(); this.tabs.splice(index, 1); - if (this.activeTabId === tabId) { - const newIndex = Math.min(index, this.tabs.length - 1); - this.activeTabId = this.tabs[newIndex].id; + const remainingEngineTabs = sameEngineTabs.filter((t) => t.id !== tabId); + const nextTab = + remainingEngineTabs[ + Math.min(engineIndex, remainingEngineTabs.length - 1) + ]; + if (this.activeTabId === tabId && nextTab !== undefined) { + this.activeTabId = nextTab.id; + } + if ( + this.lastActiveTabByEngine[tabToClose.queryEngine] === tabId && + nextTab !== undefined + ) { + this.lastActiveTabByEngine[tabToClose.queryEngine] = nextTab.id; } this.markDirty(); } renameTab(tabId: string, newTitle: string): void { const tab = this.tabs.find((t) => t.id === tabId); - if (tab) { - tab.title = newTitle; - this.markDirty(); - } + const trimmed = newTitle.trim(); + if (!tab || !trimmed || !canRenameTab(tab)) return; + tab.title = trimmed; + tab.tableName = trimmed; + this.markDirty(); } reorderTab(draggedId: string, beforeId: string | undefined): void { @@ -720,6 +1162,19 @@ id: t.id, title: t.title, editorText: t.editorText, + queryEngine: t.queryEngine, + tableName: t.tableName, + subQueries: + t.subQueries.length > 1 + ? t.subQueries.map((s) => ({ + id: s.id, + sql: s.sql, + tableName: s.tableName, + queryUuid: s.queryUuid, + })) + : undefined, + activeSubQueryIndex: + t.subQueries.length > 1 ? t.activeSubQueryIndex : undefined, limit: t.limit, traceLimit: t.traceLimit, tableTtlDays: t.tableTtlDays, @@ -739,6 +1194,7 @@ lastPresetId: t.lastPresetId, })), activeTabId: this.activeTabId, + activeEngine: this.activeEngine, }; try { localStorage.setItem(QUERY_TABS_STORAGE_KEY, JSON.stringify(state)); @@ -784,15 +1240,15 @@ } } } - return true; - } - - private nextTabName(): string { - const existingNames = new Set(this.tabs.map((t) => t.title)); - let count = ++this.tabCounter; - while (existingNames.has(`Query ${count}`)) { - count = ++this.tabCounter; + const activeTab = this.getActiveTab(); + if (parsed.activeEngine === 'traces' || parsed.activeEngine === 'duckdb') { + this.activeEngine = parsed.activeEngine; + } else if (activeTab !== undefined) { + this.activeEngine = activeTab.queryEngine; } - return `Query ${count}`; + if (activeTab !== undefined) { + this.lastActiveTabByEngine[activeTab.queryEngine] = activeTab.id; + } + return true; } }
diff --git a/ui/src/bigtrace/pages/query_tabs_state_unittest.ts b/ui/src/bigtrace/pages/query_tabs_state_unittest.ts index e4ba9b5..93f7f6a 100644 --- a/ui/src/bigtrace/pages/query_tabs_state_unittest.ts +++ b/ui/src/bigtrace/pages/query_tabs_state_unittest.ts
@@ -15,12 +15,22 @@ import {beforeEach, describe, expect, test} from 'vitest'; import {z} from 'zod'; import { + addSubQuery, applyModeDefaults, applyPresetSetup, + buildMultiQuerySpecs, + canRenameSubQuery, + canRenameTab, closeSettings, + getSubQueryTableName, isTraceSelectionSetting, + makeQueryResponse, parseTraceUuids, + removeSubQuery, + renameSubQuery, + selectSubQuery, setTraceUuidsActive, + syncActiveSubQueryToSlot, TRACE_UUIDS_SETTING_ID, traceUuidsActive, DEFAULT_TABLE_TTL_DAYS, @@ -31,6 +41,7 @@ QueryTabsState, restoreTabConfig, snapshotTabConfig, + stageDuckdbQuery, type BigTraceEditorTab, } from './query_tabs_state'; import {bigTraceSettingsStorage} from '../settings/bigtrace_settings_storage'; @@ -256,12 +267,16 @@ tab.traceLimit = 0; tab.tableTtlDays = 45; (tabs as unknown as {saveToStorage: () => void}).saveToStorage(); - const restored = new QueryTabsState().tabs.find( + const restoredState = new QueryTabsState(); + const restored = restoredState.tabs.find( (t) => t.editorText === 'select 1', )!; expect(restored.limit).toBe(0); expect(restored.traceLimit).toBe(0); expect(restored.tableTtlDays).toBe(45); + const cloned = restoredState.cloneTab(restored.id)!; + expect(cloned.limit).toBe(0); + expect(cloned.traceLimit).toBe(0); }); }); @@ -284,7 +299,8 @@ function configuredTab(tabs: QueryTabsState) { const tab = tabs.addNewTab(undefined, '', undefined, undefined, true); - tab.title = 'Jank by device'; + tab.title = 'jank_by_device'; + tab.tableName = 'jank_by_device'; tab.editorText = 'select * from slice'; tab.configured = true; tab.queryUuid = 'uuid-1'; @@ -332,10 +348,12 @@ expect(clone.isLoading).toBe(false); }); - test('a clone is a new tab, named like one', () => { + test('a clone is a new tab, named with a fresh default table name', () => { const tabs = new QueryTabsState(); const src = configuredTab(tabs); - expect(tabs.cloneTab(src.id)?.title).toMatch(/^Query \d+$/); + const clone = tabs.cloneTab(src.id)!; + expect(clone.title).toMatch(/^tbl_[a-z0-9]+$/); + expect(clone.tableName).toBe(clone.title); }); test('editing the clone leaves the original alone', () => { @@ -368,7 +386,7 @@ expect(tabs.cloneTab(src.id)?.lastPresetId).toBe('local:sweep'); (tabs as unknown as {saveToStorage: () => void}).saveToStorage(); const restored = new QueryTabsState().tabs.find( - (t) => t.title === 'Jank by device', + (t) => t.title === 'jank_by_device', )!; expect(restored.lastPresetId).toBe('local:sweep'); }); @@ -813,3 +831,369 @@ expect(tab.experimentFilter?.experimentName).toBeUndefined(); }); }); + +describe('stageDuckdbQuery (open a table in DuckDB from History)', () => { + const SQL = 'SELECT *\nFROM result_1\nLIMIT 1000;\n'; + + beforeEach(() => { + localStorage.clear(); + }); + + test('switches an empty active tab to DuckDB mode without running', () => { + const tabs = new QueryTabsState(); + const active = tabs.getActiveTab()!; + const tab = stageDuckdbQuery(tabs, SQL, 1000); + expect(tab).toBe(active); + expect(tabs.tabs.length).toBe(1); + expect(tab.queryEngine).toBe('duckdb'); + expect(tab.configured).toBe(true); + expect(tab.editorText).toBe(SQL); + expect(tab.limit).toBe(1000); + expect(tab.isLoading).toBe(false); + expect(tab.queryUuid).toBeUndefined(); + }); + + test('never overwrites a tab that already holds SQL', () => { + const tabs = new QueryTabsState(); + const active = tabs.getActiveTab()!; + active.editorText = 'SELECT * FROM slice'; + const tab = stageDuckdbQuery(tabs, SQL); + expect(tab).not.toBe(active); + expect(active.editorText).toBe('SELECT * FROM slice'); + expect(active.queryEngine).toBe('traces'); + expect(tabs.activeTabId).toBe(tab.id); + expect(tab.queryEngine).toBe('duckdb'); + }); + + test('leaves a running tab alone', () => { + const tabs = new QueryTabsState(); + const active = tabs.getActiveTab()!; + active.isLoading = true; + const tab = stageDuckdbQuery(tabs, SQL); + expect(tab).not.toBe(active); + expect(active.queryEngine).toBe('traces'); + }); + + test('repeated clicks reuse the staged tab', () => { + const tabs = new QueryTabsState(); + tabs.getActiveTab()!.editorText = 'SELECT 1'; + const first = stageDuckdbQuery(tabs, SQL); + const count = tabs.tabs.length; + tabs.activeTabId = tabs.tabs[0].id; + expect(stageDuckdbQuery(tabs, SQL)).toBe(first); + expect(tabs.tabs.length).toBe(count); + }); + + test('closes an open Settings session', () => { + const tabs = new QueryTabsState(); + const active = tabs.getActiveTab()!; + openSettings(active); + stageDuckdbQuery(tabs, SQL); + expect(active.settingsSession).toBeUndefined(); + }); +}); + +describe('Tab title and table name unification', () => { + beforeEach(() => { + localStorage.clear(); + }); + + test('new tab initializes with a default UUID table name for both title and tableName', () => { + const tabs = new QueryTabsState(); + const tab = tabs.getActiveTab()!; + expect(tab.title).toMatch(/^tbl_[a-z0-9]+$/); + expect(tab.tableName).toBe(tab.title); + expect(canRenameTab(tab)).toBe(true); + }); + + test('renameTab before running a query sets both title and tableName and persists', () => { + const tabs1 = new QueryTabsState(); + const tab = tabs1.getActiveTab()!; + tab.editorText = 'SELECT * FROM slice'; + tabs1.renameTab(tab.id, ' slices '); + expect(tab.title).toBe('slices'); + expect(tab.tableName).toBe('slices'); + + (tabs1 as unknown as {saveToStorage: () => void}).saveToStorage(); + + const tabs2 = new QueryTabsState(); + expect(tabs2.tabs).toHaveLength(1); + expect(tabs2.tabs[0].title).toBe('slices'); + expect(tabs2.tabs[0].tableName).toBe('slices'); + }); + + test('renameTab is blocked while loading or after a query has run', () => { + const tabs = new QueryTabsState(); + const tab = tabs.getActiveTab()!; + tabs.renameTab(tab.id, 'initial_name'); + expect(tab.title).toBe('initial_name'); + + // Blocked while loading + tab.isLoading = true; + expect(canRenameTab(tab)).toBe(false); + tabs.renameTab(tab.id, 'while_loading'); + expect(tab.title).toBe('initial_name'); + expect(tab.tableName).toBe('initial_name'); + + // Blocked once queryUuid is set + tab.isLoading = false; + tab.queryUuid = 'exec-uuid-1'; + expect(canRenameTab(tab)).toBe(false); + tabs.renameTab(tab.id, 'after_uuid'); + expect(tab.title).toBe('initial_name'); + expect(tab.tableName).toBe('initial_name'); + + // Blocked once queryResult is set + tab.queryUuid = undefined; + tab.queryResult = makeQueryResponse('SELECT 1'); + expect(canRenameTab(tab)).toBe(false); + tabs.renameTab(tab.id, 'after_result'); + expect(tab.title).toBe('initial_name'); + expect(tab.tableName).toBe('initial_name'); + }); +}); + +describe('Top-level engine mode (Traces vs Analysis)', () => { + beforeEach(() => { + localStorage.clear(); + }); + + test('setEngine switches visible tabs and creates a tab if none exist in that mode', () => { + const tabs = new QueryTabsState(); + const tracesTab = tabs.getActiveTab()!; + expect(tabs.activeEngine).toBe('traces'); + expect(tabs.getVisibleTabs()).toEqual([tracesTab]); + + // Switch to Analysis (duckdb): creates a new duckdb tab automatically + tabs.setEngine('duckdb'); + expect(tabs.activeEngine).toBe('duckdb'); + const analysisTab = tabs.getActiveTab()!; + expect(analysisTab.id).not.toBe(tracesTab.id); + expect(analysisTab.queryEngine).toBe('duckdb'); + expect(analysisTab.configured).toBe(true); + expect(tabs.getVisibleTabs()).toEqual([analysisTab]); + + // New tab in Analysis mode creates another duckdb tab + const analysisTab2 = tabs.addNewTab( + undefined, + 'SELECT 1', + undefined, + undefined, + undefined, + true, + ); + expect(analysisTab2.queryEngine).toBe('duckdb'); + expect(tabs.getVisibleTabs()).toEqual([analysisTab, analysisTab2]); + + // Switch back to Traces: restores the original tracesTab and shows only traces tabs + tabs.setEngine('traces'); + expect(tabs.activeEngine).toBe('traces'); + expect(tabs.getActiveTab()).toBe(tracesTab); + expect(tabs.getVisibleTabs()).toEqual([tracesTab]); + + // Switch to Analysis again: restores the last active analysis tab (analysisTab2) + tabs.setEngine('duckdb'); + expect(tabs.getActiveTab()).toBe(analysisTab2); + }); + + test('closeTab is scoped to the tab engine and never closes the last tab in that engine', () => { + const tabs = new QueryTabsState(); + const tracesTab = tabs.getActiveTab()!; + tabs.setEngine('duckdb'); + const analysisTab1 = tabs.getActiveTab()!; + const analysisTab2 = tabs.addNewTab( + undefined, + 'SELECT 2', + undefined, + undefined, + undefined, + true, + ); + + // Cannot close the only traces tab even though total tabs.length is 3 + tabs.closeTab(tracesTab.id); + expect(tabs.tabs).toContain(tracesTab); + + // Closing analysisTab2 activates analysisTab1 (not tracesTab) + tabs.closeTab(analysisTab2.id); + expect(tabs.getActiveTab()).toBe(analysisTab1); + expect(tabs.getVisibleTabs()).toEqual([analysisTab1]); + + // Cannot close the last remaining analysis tab + tabs.closeTab(analysisTab1.id); + expect(tabs.getVisibleTabs()).toEqual([analysisTab1]); + }); +}); + +describe('SubQuery operations and persistence', () => { + beforeEach(() => { + localStorage.clear(); + }); + + test('new tab and addSubQuery initialize with default UUID table names', () => { + const tabs = new QueryTabsState(); + const tab = tabs.getActiveTab()!; + expect(tab.tableName).toMatch(/^tbl_[a-z0-9]+$/); + tab.editorText = 'SELECT * FROM slices'; + tab.tableName = 'slices'; + + addSubQuery(tab); + expect(tab.subQueries).toHaveLength(2); + expect(tab.subQueries[0].sql).toBe('SELECT * FROM slices'); + expect(tab.subQueries[0].tableName).toBe('slices'); + expect(tab.subQueries[1].sql).toBe(''); + expect(tab.subQueries[1].tableName).toMatch(/^tbl_[a-z0-9]+$/); + expect(tab.activeSubQueryIndex).toBe(1); + expect(tab.editorText).toBe(''); + expect(tab.tableName).toBe(tab.subQueries[1].tableName); + }); + + test('selectSubQuery switches active subquery and synchronizes in-flight edits', () => { + const tabs = new QueryTabsState(); + const tab = tabs.getActiveTab()!; + tab.editorText = 'SELECT * FROM slices'; + addSubQuery(tab); + + // Edit query #2 + tab.editorText = 'SELECT * FROM sched'; + tab.tableName = 'sched'; + + // Switch back to query #1 + selectSubQuery(tab, 0); + expect(tab.activeSubQueryIndex).toBe(0); + expect(tab.editorText).toBe('SELECT * FROM slices'); + expect(tab.subQueries[1].sql).toBe('SELECT * FROM sched'); + expect(tab.subQueries[1].tableName).toBe('sched'); + + // Switch to query #2 again + selectSubQuery(tab, 1); + expect(tab.activeSubQueryIndex).toBe(1); + expect(tab.editorText).toBe('SELECT * FROM sched'); + expect(tab.tableName).toBe('sched'); + }); + + test('removeSubQuery reverts to single query when only 1 remains', () => { + const tabs = new QueryTabsState(); + const tab = tabs.getActiveTab()!; + tab.editorText = 'SELECT * FROM slices'; + addSubQuery(tab); + tab.editorText = 'SELECT * FROM sched'; + tab.tableName = 'sched'; + + // Remove query #1 (index 0) + removeSubQuery(tab, 0); + expect(tab.subQueries).toHaveLength(0); + expect(tab.activeSubQueryIndex).toBe(0); + expect(tab.editorText).toBe('SELECT * FROM sched'); + expect(tab.tableName).toBe('sched'); + }); + + test('removeSubQuery selects previous valid index when removing from 3 queries', () => { + const tabs = new QueryTabsState(); + const tab = tabs.getActiveTab()!; + tab.editorText = 'Q1'; + addSubQuery(tab); + tab.editorText = 'Q2'; + addSubQuery(tab); + tab.editorText = 'Q3'; + expect(tab.subQueries).toHaveLength(3); + expect(tab.activeSubQueryIndex).toBe(2); + + // Remove active query (index 2) + removeSubQuery(tab, 2); + expect(tab.subQueries).toHaveLength(2); + expect(tab.activeSubQueryIndex).toBe(1); + expect(tab.editorText).toBe('Q2'); + }); + + test('getSubQueryTableName derives names sensibly and respects overrides', () => { + expect( + getSubQueryTableName({sql: 'SELECT * FROM slices WHERE dur >= 0'}, 0), + ).toBe('slices'); + expect(getSubQueryTableName({sql: 'SELECT * FROM sched'}, 1)).toBe('sched'); + expect( + getSubQueryTableName({sql: 'SELECT 1', tableName: 'custom_tbl'}, 2), + ).toBe('custom_tbl'); + expect(getSubQueryTableName({sql: 'SELECT 1'}, 3)).toBe('result_4'); + }); + + test('buildMultiQuerySpecs derives unique table names', () => { + const slots = [ + {id: '1', sql: 'SELECT * FROM slices', tableName: ''}, + {id: '2', sql: 'SELECT * FROM sched', tableName: ''}, + {id: '3', sql: 'SELECT * FROM slices', tableName: ''}, + ]; + const specs = buildMultiQuerySpecs(slots); + expect(specs).toEqual([ + {slotId: '1', perfettoSql: 'SELECT * FROM slices', tableName: 'slices'}, + {slotId: '2', perfettoSql: 'SELECT * FROM sched', tableName: 'sched'}, + {slotId: '3', perfettoSql: 'SELECT * FROM slices', tableName: 'slices_2'}, + ]); + }); + + test('subqueries persist across reload via localStorage', () => { + const tabs1 = new QueryTabsState(); + const tab = tabs1.getActiveTab()!; + tab.editorText = 'SELECT * FROM slices'; + tab.tableName = 'slices'; + addSubQuery(tab); + tab.editorText = 'SELECT * FROM sched'; + tab.tableName = 'sched'; + + // Force save to storage immediately + syncActiveSubQueryToSlot(tab); + (tabs1 as unknown as {saveToStorage: () => void}).saveToStorage(); + + // Reload state + const tabs2 = new QueryTabsState(); + expect(tabs2.tabs).toHaveLength(1); + const restored = tabs2.tabs[0]; + expect(restored.subQueries).toHaveLength(2); + expect(restored.subQueries[0].sql).toBe('SELECT * FROM slices'); + expect(restored.subQueries[0].tableName).toBe('slices'); + expect(restored.subQueries[1].sql).toBe('SELECT * FROM sched'); + expect(restored.subQueries[1].tableName).toBe('sched'); + expect(restored.activeSubQueryIndex).toBe(1); + expect(restored.editorText).toBe('SELECT * FROM sched'); + expect(restored.tableName).toBe('sched'); + }); + + test('renameSubQuery updates active and non-active slot tableName and blocks after run', () => { + const tabs = new QueryTabsState(); + const tab = tabs.getActiveTab()!; + tab.editorText = 'SELECT * FROM slice'; + tab.tableName = 'slice_tbl'; + addSubQuery(tab); + tab.editorText = 'SELECT * FROM sched'; + + // Rename non-active slot 0 + expect(canRenameSubQuery(tab, tab.subQueries[0])).toBe(true); + renameSubQuery(tab, 0, ' custom_slices '); + expect(tab.subQueries[0].tableName).toBe('custom_slices'); + + // Rename active slot 1 -> updates both slot and tab.tableName + expect(canRenameSubQuery(tab, tab.subQueries[1])).toBe(true); + renameSubQuery(tab, 1, ' custom_sched '); + expect(tab.subQueries[1].tableName).toBe('custom_sched'); + expect(tab.tableName).toBe('custom_sched'); + + // Blocked while tab is loading + tab.isLoading = true; + expect(canRenameSubQuery(tab, tab.subQueries[0])).toBe(false); + renameSubQuery(tab, 0, 'ignored_loading'); + expect(tab.subQueries[0].tableName).toBe('custom_slices'); + tab.isLoading = false; + + // Blocked once slot has queryUuid or queryResult + tab.subQueries[0].queryUuid = 'uuid-slot-0'; + expect(canRenameSubQuery(tab, tab.subQueries[0])).toBe(false); + renameSubQuery(tab, 0, 'ignored_after_run'); + expect(tab.subQueries[0].tableName).toBe('custom_slices'); + + // Clone with targetSubQueryIndex replaces the colliding slot's tableName + const cloned = tabs.cloneTab(tab.id, 'custom_slices_1', undefined, 0)!; + expect(cloned.subQueries).toHaveLength(2); + expect(cloned.subQueries[0].tableName).toBe('custom_slices_1'); + expect(cloned.subQueries[1].tableName).toBe('custom_sched'); + }); +});
diff --git a/ui/src/bigtrace/pages/results_panel.ts b/ui/src/bigtrace/pages/results_panel.ts index efa2a3d..fda1041 100644 --- a/ui/src/bigtrace/pages/results_panel.ts +++ b/ui/src/bigtrace/pages/results_panel.ts
@@ -60,14 +60,21 @@ const renderSpinner = () => m(RunningQuerySpinner, { startMs: tab.clientStartTime ?? Date.now(), - showDuration: !tab.materialize, + showDuration: !tab.materialize || tab.queryEngine === 'duckdb', }); + const isRunning = + tab.isLoading || + (Boolean(tab.queryUuid) && + tab.execution !== undefined && + tab.execution.status !== undefined && + !TERMINAL_STATUSES.has(tab.execution.status)); + if (!tab.dataSource || !tab.queryResult) { return m( '.pf-bt-results-panel', status, - tab.isLoading + isRunning ? renderSpinner() : m(EmptyState, { title: 'Run a query to see results', @@ -119,7 +126,7 @@ }); } else if (hasRowsToShow) { tableContent = renderResultsGrid(tab, tabsState); - } else if (tab.isLoading) { + } else if (isRunning) { tableContent = renderSpinner(); } else { tableContent = m(EmptyState, {
diff --git a/ui/src/bigtrace/pages/results_panel_unittest.ts b/ui/src/bigtrace/pages/results_panel_unittest.ts index ddf5c9f..a4d52c6 100644 --- a/ui/src/bigtrace/pages/results_panel_unittest.ts +++ b/ui/src/bigtrace/pages/results_panel_unittest.ts
@@ -134,4 +134,74 @@ const text = vnodeText(renderResultsPanel(tab, tabsState)); expect(text).toContain('Results no longer available'); }); + + test('renders at most one inline progress bar when running and on rerun', async () => { + function countSelector(node: unknown, cls: string): number { + if (node === null || node === undefined || typeof node !== 'object') { + return 0; + } + const vnode = node as Vnode; + let n = 0; + const tagStr = typeof vnode.tag === 'string' ? vnode.tag : ''; + const clsAttr = String( + (vnode.attrs as {className?: string} | undefined)?.className ?? '', + ); + if ( + tagStr.split('.').includes(cls) || + clsAttr.split(/\s+/).includes(cls) + ) { + n++; + } + for (const [k, v] of Object.entries(vnode)) { + if (k === 'dom' || k === 'events' || k === 'tag') continue; + if (Array.isArray(v)) { + for (const child of v) n += countSelector(child, cls); + } else if (v !== null && typeof v === 'object') { + n += countSelector(v, cls); + } + } + return n; + } + + const tabsState = new QueryTabsState(); + const tab = runningTab(tabsState, 'uuid-progress', fakeClient('tbl_prog')); + tab.limit = 10000; + // Before first poll returns (totalTraces === 0), only Traces has a bar. + expect( + countSelector( + renderResultsPanel(tab, tabsState), + 'pf-bt-inline-progress', + ), + ).toBe(1); + + await pollToCompletion(tab); + expect( + countSelector( + renderResultsPanel(tab, tabsState), + 'pf-bt-inline-progress', + ), + ).toBe(0); + + // Rerunning resets execution to undefined while loading: still strictly one bar. + tab.isLoading = true; + tab.execution = undefined; + expect( + countSelector( + renderResultsPanel(tab, tabsState), + 'pf-bt-inline-progress', + ), + ).toBe(1); + expect( + countSelector(renderResultsPanel(tab, tabsState), 'pf-bt-status-bar'), + ).toBe(1); + }); + + test('new unexecuted tab renders empty state instead of running spinner', () => { + const tabsState = new QueryTabsState(); + tabsState.setEngine('duckdb'); + const tab = tabsState.getActiveTab()!; + const text = vnodeText(renderResultsPanel(tab, tabsState)); + expect(text).toContain('<EmptyState>'); + expect(text).not.toContain('<RunningQuerySpinner>'); + }); });
diff --git a/ui/src/bigtrace/pages/status_box.ts b/ui/src/bigtrace/pages/status_box.ts index e3823ab..2f084cb 100644 --- a/ui/src/bigtrace/pages/status_box.ts +++ b/ui/src/bigtrace/pages/status_box.ts
@@ -34,9 +34,10 @@ } export function renderStatusBox(tab: BigTraceEditorTab): m.Children { - if (!tab.materialize || !tab.queryUuid) return false; + if (!tab.materialize || (!tab.queryUuid && !tab.isLoading)) return false; const isTerminal = + !tab.isLoading && tab.execution?.status !== undefined && TERMINAL_STATUSES.has(tab.execution.status); const processedRows = tab.execution?.processedRows ?? 0; @@ -59,7 +60,8 @@ } } - const status = tab.execution?.status ?? 'UNKNOWN'; + const status = + tab.execution?.status ?? (tab.isLoading ? 'IN_PROGRESS' : 'UNKNOWN'); const processedTraces = tab.execution?.processedTraces ?? 0; const totalTraces = tab.execution?.totalTraces ?? 0; const durationStr = formatDurationS(durationMs); @@ -98,38 +100,54 @@ ] .filter(Boolean) .join(' '); + const isDuckdb = tab.queryEngine === 'duckdb'; + const duckdbPct = Math.max(0, Math.min(99, processedTraces)); const rightGroupContent = m( '.pf-bt-status-bar-group', - m( - 'span.pf-bt-status-bar-stat.pf-bt-status-bar-stat--traces', - m('span.pf-bt-status-bar-stat-label', 'Traces:'), + !isDuckdb && m( - 'span.pf-bt-status-bar-stat-value', - { - title: - `${processedTraces.toLocaleString()} of ` + - `${totalTraces.toLocaleString()}` + - (!isTerminal - ? ' — numerator lags the poll (≤3s); denominator is exact.' - : ''), - }, - formatCompact(processedTraces), + 'span.pf-bt-status-bar-stat.pf-bt-status-bar-stat--traces', + m('span.pf-bt-status-bar-stat-label', 'Traces:'), + m( + 'span.pf-bt-status-bar-stat-value', + { + title: + `${processedTraces.toLocaleString()} of ` + + `${totalTraces.toLocaleString()}` + + (!isTerminal + ? ' — numerator lags the poll (≤3s); denominator is exact.' + : ''), + }, + formatCompact(processedTraces), + ), + renderInlineProgressBar(processedTraces, totalTraces, !isTerminal), ), - renderInlineProgressBar(processedTraces, totalTraces, !isTerminal), - ), - m( - 'span', - {className: rowsStatClasses}, - m('span.pf-bt-status-bar-stat-label', 'Rows:'), - m( - 'span.pf-bt-status-bar-stat-value', - { - title: `${processedRows.toLocaleString()} of result limit ${tab.limit.toLocaleString()}`, - }, - formatCompact(processedRows), - ), - renderInlineProgressBar(processedRows, tab.limit, !isTerminal), - ), + isDuckdb && !isTerminal + ? m( + 'span.pf-bt-status-bar-stat.pf-bt-status-bar-stat--rows', + m('span.pf-bt-status-bar-stat-label', 'Progress:'), + m( + 'span.pf-bt-status-bar-stat-value', + {title: `DuckDB execution progress: ${duckdbPct}%`}, + `${duckdbPct}%`, + ), + renderInlineProgressBar(duckdbPct, 100, true), + ) + : m( + 'span', + {className: rowsStatClasses}, + m('span.pf-bt-status-bar-stat-label', 'Rows:'), + m( + 'span.pf-bt-status-bar-stat-value', + { + title: + tab.limit > 0 + ? `${processedRows.toLocaleString()} of result limit ${tab.limit.toLocaleString()}` + : `${processedRows.toLocaleString()} rows (no limit)`, + }, + formatCompact(processedRows), + ), + ), ); const rightGroup = !isTerminal @@ -141,8 +159,12 @@ }, m( '.pf-bt-status-bar-progress-tooltip', - m('div', `Traces: ${formatCompact(processedTraces)}`), - m('div', `Rows: ${formatCompact(processedRows)}`), + isDuckdb + ? m('div', `Progress: ${duckdbPct}%`) + : [ + m('div', `Traces: ${formatCompact(processedTraces)}`), + m('div', `Rows: ${formatCompact(processedRows)}`), + ], ), ) : rightGroupContent; @@ -165,7 +187,14 @@ live: boolean, ): m.Children { if (!live) return null; - if (total <= 0) return null; + if (total <= 0 || done <= 0) { + return m( + 'span.pf-bt-inline-progress', + m( + 'span.pf-bt-inline-progress-fill.pf-bt-inline-progress-fill--indeterminate', + ), + ); + } const pct = Math.max(0, Math.min(100, (done / total) * 100)); return m( 'span.pf-bt-inline-progress',
diff --git a/ui/src/bigtrace/query/bigtrace_async_data_source.ts b/ui/src/bigtrace/query/bigtrace_async_data_source.ts index bf8189f..8f1b5d2 100644 --- a/ui/src/bigtrace/query/bigtrace_async_data_source.ts +++ b/ui/src/bigtrace/query/bigtrace_async_data_source.ts
@@ -247,7 +247,13 @@ } catch (e) { // Aborts are routine here — the tab closing, or a newer window // superseding this one — so they are not failures to show. - if (e instanceof QueryCancelledError) return; + if ( + e instanceof QueryCancelledError || + controller.signal.aborted || + !this.scheduler.isCurrent(controller) + ) { + return; + } console.error('[bigtrace] fetch_results failed:', e); if (e instanceof BigtraceHttpError) { // Surface the backend's detail (not the wrapped message) and keep the
diff --git a/ui/src/bigtrace/query/bigtrace_async_data_source_unittest.ts b/ui/src/bigtrace/query/bigtrace_async_data_source_unittest.ts index 6af0f73..c2253aa 100644 --- a/ui/src/bigtrace/query/bigtrace_async_data_source_unittest.ts +++ b/ui/src/bigtrace/query/bigtrace_async_data_source_unittest.ts
@@ -209,4 +209,43 @@ expect(calls).toHaveLength(0); expect(ds.useRows(model(400, 100)).isPending).toBe(true); }); + + test('an aborted or superseded fetch rejecting with AbortError never surfaces an error', async () => { + let rejectFirst: ((err: unknown) => void) | undefined; + let callCount = 0; + const client = { + fetchResults: vi.fn( + ( + _uuid: string, + _limit: number, + _offset: number, + _signal?: AbortSignal, + ): Promise<unknown> => { + callCount++; + if (callCount === 1) { + return new Promise((_resolve, reject) => { + rejectFirst = reject; + }); + } + return Promise.resolve({ + ...PAGE, + schema: [{name: 'a', type: 'STRING'}], + }); + }, + ), + } as unknown as BigtraceQueryClient; + + const ds = new BigtraceAsyncDataSource('uid', client, () => 5000); + ds.useRows(model(0, 100)); + expect(callCount).toBe(1); + + // Trigger refresh() while the initial fetch is still in flight (superseding it). + const refreshPromise = ds.refresh(); + rejectFirst?.( + new DOMException('signal is aborted without reason', 'AbortError'), + ); + await refreshPromise; + + expect(ds.getError()).toBeNull(); + }); });
diff --git a/ui/src/bigtrace/query/bigtrace_query_client.ts b/ui/src/bigtrace/query/bigtrace_query_client.ts index 095f49b..ad755c0 100644 --- a/ui/src/bigtrace/query/bigtrace_query_client.ts +++ b/ui/src/bigtrace/query/bigtrace_query_client.ts
@@ -19,10 +19,17 @@ import {coerceFiltersForWire} from './filter_encoding'; import type {RawQueryExecution} from './query_history_storage'; +export interface BatchExecutionItem { + readonly queryUuid: string; + readonly tableName?: string; +} + // Tabular wire shape for QueryResponse (`/execute_*` and `:fetch_results`). // Values are always strings; JSON null or "NULL" denotes SQL NULL. interface QueryResponsePayload { queryUuid?: string; + tableName?: string; + executions?: BatchExecutionItem[]; schema?: BigtraceColumnSchema[]; rows?: Array<{values: Array<string | null>}>; // Filtered count for scrollbar sizing. @@ -45,6 +52,8 @@ readonly columns: ReadonlyArray<string>; readonly schema: ReadonlyArray<BigtraceColumnSchema>; readonly queryUuid?: string; + readonly tableName?: string; + readonly executions?: ReadonlyArray<BatchExecutionItem>; // Post-filter count from `:fetch_results`; undefined elsewhere. readonly totalFilteredRows?: number; // Every projectable column (result + metadata columns); `:fetch_results`-only. @@ -105,6 +114,11 @@ }; } +export interface SubQuerySpec { + readonly perfettoSql: string; + readonly tableName?: string; +} + // The submit-time trace-selection snapshot shipped as top-level fields on // /execute_*. Each is omitted from the wire when empty / default, so a query // run with no trace selection sends just the base limit/perfetto_sql/settings. @@ -127,6 +141,19 @@ // How many days that table lives for. Independent of the name: the table // exists either way. readonly tableTtlDays?: number; + // Execution engine: 'traces' (PerfettoSQL fan-out) or 'duckdb' (DuckDB over + // materialized history tables). + readonly engine?: 'traces' | 'duckdb'; + // Optional batch of queries to run in a single pass per trace. + readonly queries?: ReadonlyArray<SubQuerySpec>; +} + +export interface DuckdbTableDescriptor { + readonly name: string; + readonly rowCount: number; + readonly columns: ReadonlyArray<BigtraceColumnSchema>; + readonly queryUuid?: string; + readonly sql?: string; } // What `/check_table_exists` says about a name before it is used. @@ -135,6 +162,8 @@ // The name the backend would actually create, which may namespace or // normalise what was asked for. readonly resolvedTableName: string; + // A fresh non-colliding alternative (e.g. `<name>_1`). + readonly suggestedTableName?: string; } // One analysis preset from `GET /trace_presets`: display metadata plus a @@ -376,6 +405,7 @@ return { exists: result.exists === true, resolvedTableName: result.resolvedTableName ?? tableName, + suggestedTableName: result.suggestedTableName, }; } @@ -393,6 +423,22 @@ return result.tracePresets ?? []; } + async listDuckdbTables( + signal?: AbortSignal, + ): Promise<ReadonlyArray<DuckdbTableDescriptor>> { + try { + const result = await this.requestJson<{ + tables?: DuckdbTableDescriptor[]; + }>('/duckdb/tables', { + method: 'GET', + signal, + }); + return result.tables ?? []; + } catch { + return []; + } + } + // Deleting an execution takes its table with it: the results are the // query's, so keeping them under a name whose execution is gone would // leave a table nobody can trace back. `drop_table` rides as a query @@ -516,6 +562,15 @@ if (options?.tableTtlDays !== undefined && options.tableTtlDays > 0) { body.table_ttl_days = options.tableTtlDays; } + if (options?.engine !== undefined && options.engine !== 'traces') { + body.engine = options.engine; + } + if (options?.queries !== undefined && options.queries.length > 0) { + body.queries = options.queries.map((q) => ({ + perfetto_sql: q.perfettoSql, + ...(q.tableName ? {table_name: q.tableName} : {}), + })); + } const result = await this.requestJson<QueryResponsePayload>(path, { method: 'POST', headers: {'Content-Type': 'application/json'}, @@ -545,8 +600,8 @@ ...init, }); } catch (e) { - // AbortSignal → DOMException, surface as our typed error. - if (e instanceof DOMException && e.name === 'AbortError') { + // AbortSignal → DOMException/AbortError, surface as our typed error. + if (isAbortError(e) || init?.signal?.aborted) { throw new QueryCancelledError(); } throw e; @@ -586,10 +641,24 @@ private async requestJson<T>(path: string, init?: RequestInit): Promise<T> { const response = await this.request(path, init); - return (await response.json()) as T; + try { + return (await response.json()) as T; + } catch (e) { + if (isAbortError(e) || init?.signal?.aborted) { + throw new QueryCancelledError(); + } + throw e; + } } } +function isAbortError(e: unknown): boolean { + if (e instanceof QueryCancelledError) return true; + if (e instanceof DOMException && e.name === 'AbortError') return true; + if (e instanceof Error && e.name === 'AbortError') return true; + return false; +} + // Passes wire values through as-is: no numeric coercion (would corrupt 64-bit // ids/timestamps past 2^53), and SQL NULL arrives as JSON null. Do NOT // special-case the literal string "NULL" — that would corrupt a genuine "NULL" @@ -614,6 +683,8 @@ rows: [], columns: [], queryUuid: result.queryUuid, + tableName: result.tableName, + executions: result.executions, totalFilteredRows: result.totalFilteredRows, schema: [], }; @@ -631,6 +702,8 @@ rows, columns, queryUuid: result.queryUuid, + tableName: result.tableName, + executions: result.executions, totalFilteredRows: result.totalFilteredRows, schema, };
diff --git a/ui/src/bigtrace/query/bigtrace_trace_list_data_source.ts b/ui/src/bigtrace/query/bigtrace_trace_list_data_source.ts index 3ec6a70..7671879 100644 --- a/ui/src/bigtrace/query/bigtrace_trace_list_data_source.ts +++ b/ui/src/bigtrace/query/bigtrace_trace_list_data_source.ts
@@ -253,7 +253,13 @@ this.columns = [...result.columns]; } } catch (e) { - if (e instanceof QueryCancelledError) return; + if ( + e instanceof QueryCancelledError || + controller.signal.aborted || + !this.scheduler.isCurrent(controller) + ) { + return; + } console.error('[bigtrace] trace_metadata failed:', e); this.error = e instanceof Error ? e.message : String(e); // A 400 while the trace source is unset/unreadable is the common case
diff --git a/ui/src/bigtrace/query/duckdb_tables_store.ts b/ui/src/bigtrace/query/duckdb_tables_store.ts new file mode 100644 index 0000000..93cc070 --- /dev/null +++ b/ui/src/bigtrace/query/duckdb_tables_store.ts
@@ -0,0 +1,75 @@ +// Copyright (C) 2026 The Android Open Source Project +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import m from 'mithril'; +import { + BigtraceQueryClient, + type DuckdbTableDescriptor, +} from './bigtrace_query_client'; +import {getBigtraceEndpoint} from '../settings/endpoint_storage'; + +export class DuckdbTablesStore { + private tables: ReadonlyArray<DuckdbTableDescriptor> = []; + private lastEndpoint = ''; + private isFetching = false; + private hasFetched = false; + + getTables(): ReadonlyArray<DuckdbTableDescriptor> { + return this.tables; + } + + hasLoaded(): boolean { + return this.hasFetched; + } + + async fetchIfNeeded(): Promise<void> { + const endpoint = getBigtraceEndpoint().trim(); + if (!endpoint) return; + if (endpoint !== this.lastEndpoint || !this.hasFetched) { + await this.refresh(); + } + } + + async refresh(): Promise<void> { + const endpoint = getBigtraceEndpoint().trim(); + if (!endpoint || this.isFetching) return; + this.isFetching = true; + this.lastEndpoint = endpoint; + const client = new BigtraceQueryClient(endpoint); + try { + this.tables = await client.listDuckdbTables(); + this.hasFetched = true; + } catch { + // Retry once after 500ms on failure + await new Promise((r) => setTimeout(r, 500)); + try { + this.tables = await client.listDuckdbTables(); + this.hasFetched = true; + } catch { + // Keep previous tables + } + } finally { + this.isFetching = false; + m.redraw(); + } + } + + invalidate(): void { + this.tables = []; + this.hasFetched = false; + this.lastEndpoint = ''; + } +} + +export const duckdbTablesStore = new DuckdbTablesStore();
diff --git a/ui/src/bigtrace/query/history_store.ts b/ui/src/bigtrace/query/history_store.ts index 0ad4f9e..3736c12 100644 --- a/ui/src/bigtrace/query/history_store.ts +++ b/ui/src/bigtrace/query/history_store.ts
@@ -110,6 +110,7 @@ private lastRefreshSignal = -1; private debounceTimer?: number; private hasEverLoaded = false; + private loadSeq = 0; requestRefresh(refreshSignal: number): void { if (refreshSignal === this.lastRefreshSignal) return; @@ -135,21 +136,31 @@ this.load(); } + remove(uuid: string): void { + this.history = this.history.filter((entry) => entry.uuid !== uuid); + m.redraw(); + } + private async load(): Promise<void> { + const seq = ++this.loadSeq; this.hasEverLoaded = true; this.isLoading = true; this.error = null; m.redraw(); try { const list = await queryHistoryStorage.getAllHistory(); + if (seq !== this.loadSeq) return; this.history = list.map((entry) => queryStore.getOrCreate(entry.uuid, entry), ); } catch (e) { + if (seq !== this.loadSeq) return; this.error = e instanceof Error ? e.message : String(e); } finally { - this.isLoading = false; - m.redraw(); + if (seq === this.loadSeq) { + this.isLoading = false; + m.redraw(); + } } } }
diff --git a/ui/src/bigtrace/query/polling_controller.ts b/ui/src/bigtrace/query/polling_controller.ts index af662dd..bf29382 100644 --- a/ui/src/bigtrace/query/polling_controller.ts +++ b/ui/src/bigtrace/query/polling_controller.ts
@@ -13,62 +13,120 @@ // limitations under the License. import {BigtraceAsyncDataSource} from './bigtrace_async_data_source'; -import {QueryNotFoundError} from './bigtrace_query_client'; +import {QueryCancelledError, QueryNotFoundError} from './bigtrace_query_client'; import {isoToEpochMs, type RawQueryExecution} from './query_history_storage'; import {queryStore, TERMINAL_STATUSES} from './query_store'; -import {makeQueryResponse} from '../pages/query_tabs_state'; import type {BigTraceEditorTab} from '../pages/query_tabs_state'; -const POLL_INTERVAL_MS = 3000; +const POLL_INTERVAL_MS = 500; const POLL_RETRY_MS = 1000; export interface PollingCallbacks { readonly redraw: () => void; readonly onHistoryChanged: () => void; + readonly markDirty?: () => void; +} + +function applyDetails(uuid: string, details: RawQueryExecution): void { + const endMs = isoToEpochMs(details.endTime); + queryStore.update(uuid, { + ...(endMs !== undefined ? {endTime: endMs} : {}), + ...(details.schema !== undefined ? {schema: details.schema} : {}), + ...(details.tableName !== undefined ? {tableName: details.tableName} : {}), + ...(details.processedRows !== undefined + ? {processedRows: details.processedRows} + : {}), + ...(details.processedTraces !== undefined + ? {processedTraces: details.processedTraces} + : {}), + ...(details.totalTraces !== undefined + ? {totalTraces: details.totalTraces} + : {}), + ...(details.status !== undefined ? {status: details.status} : {}), + ...(details.errorMessage !== undefined + ? {error: details.errorMessage} + : {}), + }); } // Owns the poll-until-terminal loop for async (materialized) queries. export class PollingController { constructor(private readonly cb: PollingCallbacks) {} - // Begin polling `:status` for the given tab's queryUuid. - start(tab: BigTraceEditorTab): void { - if (!tab.queryUuid) return; + start(tab: BigTraceEditorTab, uuids?: ReadonlyArray<string>): void { + const pollUuids = ( + uuids && uuids.length > 0 ? uuids : [tab.queryUuid] + ).filter((u): u is string => Boolean(u)); + if (pollUuids.length === 0) return; const generation = ++tab.pollGeneration; const poll = async () => { - if (tab.pollGeneration !== generation) return; - if (!tab.queryUuid || !tab.isLoading) return; + if (tab.pollGeneration !== generation || !tab.isLoading) return; try { - const status = await tab.queryClient?.getStatus( - tab.queryUuid, - tab.lifecycle.signal, + const terminalSet = new Set<string>(); + const notFoundSet = new Set<string>(); + + await Promise.all( + pollUuids.map(async (uuid) => { + try { + const status = await tab.queryClient?.getStatus( + uuid, + tab.lifecycle.signal, + ); + if (status) { + queryStore.update(uuid, { + processedRows: status.processedRows ?? 0, + processedTraces: status.processedTraces ?? 0, + totalTraces: status.totalTraces ?? 0, + status: status.status ?? 'N/A', + }); + if (status.status && TERMINAL_STATUSES.has(status.status)) { + terminalSet.add(uuid); + } + } + } catch (e) { + if (e instanceof QueryNotFoundError) { + terminalSet.add(uuid); + notFoundSet.add(uuid); + } + } + }), ); + if (tab.pollGeneration !== generation || !tab.isLoading) return; - if (status !== undefined && status !== null) { - this.applyStatus(tab, status); + if (notFoundSet.size === pollUuids.length) { + this.dropStaleQueryUuid(tab); + return; } - const isTerminal = - status !== undefined && - status.status !== undefined && - TERMINAL_STATUSES.has(status.status); + const activeUuid = tab.queryUuid ?? pollUuids[0]; + const activeExec = queryStore.get(activeUuid); + if (activeExec !== undefined && tab.queryResult !== undefined) { + tab.queryResult.totalRowCount = activeExec.processedRows; + } + + const isTerminal = pollUuids.every((u) => terminalSet.has(u)); if (isTerminal) { - await this.finalize(tab, status!); + await this.finalize( + tab, + pollUuids.filter((u) => !notFoundSet.has(u)), + ); } else if (tab.pollInterval !== undefined) { tab.isLoading = true; tab.pollInterval = window.setTimeout(poll, POLL_INTERVAL_MS); } this.cb.redraw(); } catch (e) { - if (e instanceof QueryNotFoundError) { - this.dropStaleQueryUuid(tab); + if ( + e instanceof QueryCancelledError || + tab.lifecycle.signal.aborted || + tab.pollGeneration !== generation + ) { return; } - console.error('Poll failed:', e); if (tab.pollInterval !== undefined) { tab.pollInterval = window.setTimeout(poll, POLL_RETRY_MS); } @@ -82,7 +140,6 @@ tab.pollInterval = window.setTimeout(poll, 0); } - // Stop any active poll timer on a tab. stop(tab: BigTraceEditorTab): void { if (tab.pollInterval !== undefined) { window.clearTimeout(tab.pollInterval); @@ -90,100 +147,91 @@ } } - // Strip dead async metadata but keep the saved SQL for re-run. dropStaleQueryUuid(tab: BigTraceEditorTab): void { - this.stop(tab); - tab.pollGeneration++; + tab.pollInterval = undefined; tab.queryUuid = undefined; - tab.execution = undefined; - tab.dataSource = undefined; - tab.queryResult = undefined; tab.isLoading = false; - tab.lastProcessedRows = 0; - this.cb.redraw(); - } - - // ----- Internals ----- - - // Progress only — the poll never pulls result rows. A page fetch costs a - // full count over the result table under the same lock the worker writes - // through, so doing it every 3s would fight the query it is reporting on. - // The status bar's refresh button (lit while processedRows is ahead of - // lastProcessedRows) is how a partial result gets asked for. - private applyStatus(tab: BigTraceEditorTab, status: RawQueryExecution): void { - if (!tab.queryUuid) return; - queryStore.update(tab.queryUuid, { - processedRows: status.processedRows ?? 0, - processedTraces: status.processedTraces ?? 0, - totalTraces: status.totalTraces ?? 0, - status: status.status ?? 'N/A', - }); + tab.execution = undefined; + tab.queryResult = undefined; + tab.dataSource = undefined; + for (const slot of tab.subQueries) { + slot.queryUuid = undefined; + slot.queryResult = undefined; + slot.dataSource = undefined; + } + this.cb.markDirty?.(); this.cb.redraw(); } private async finalize( tab: BigTraceEditorTab, - status: RawQueryExecution, + pollUuids: ReadonlyArray<string>, ): Promise<void> { tab.pollInterval = undefined; - if ( - tab.execution !== undefined && - tab.execution.endTime === undefined && - tab.queryUuid - ) { - queryStore.update(tab.queryUuid, {endTime: Date.now()}); - } + try { + if (tab.queryClient) { + await Promise.all( + pollUuids.map(async (uuid) => { + queryStore.update(uuid, {endTime: Date.now()}); + try { + const details = await tab.queryClient?.getQueryExecution( + uuid, + tab.lifecycle.signal, + ); + if (details !== undefined) { + applyDetails(uuid, details); + } + } catch { + // Ignore details fetch errors + } + }), + ); + } - if (tab.isLoading) { - this.cb.onHistoryChanged(); - } - tab.isLoading = false; + for (const slot of tab.subQueries) { + if (slot.queryUuid === undefined || slot.queryUuid === '') continue; + const exec = queryStore.get(slot.queryUuid); + if (exec !== undefined) { + if (slot.queryResult !== undefined) { + slot.queryResult.totalRowCount = exec.processedRows; + if (exec.status === 'FAILED') { + slot.queryResult.error = exec.error ?? 'Query failed'; + } + } + if (exec.tableName !== undefined && exec.tableName !== '') { + slot.tableName = exec.tableName; + } + } + if (slot.dataSource instanceof BigtraceAsyncDataSource) { + await slot.dataSource.refresh(); + } + } - const isFailed = status.status === 'FAILED'; - const isSuccess = status.status === 'SUCCESS'; - - if (isFailed) { - const startMs = tab.execution?.startTime; - const endMs = tab.execution?.endTime; - tab.queryResult = makeQueryResponse(tab.editorText, { - error: 'Fetching error details...', - durationMs: - startMs !== undefined && endMs !== undefined ? endMs - startMs : 0, - }); + if (tab.queryUuid !== undefined && tab.queryUuid !== '') { + const exec = queryStore.get(tab.queryUuid); + if (exec !== undefined) { + if (tab.queryResult !== undefined) { + tab.queryResult.totalRowCount = exec.processedRows; + if (exec.status === 'FAILED') { + tab.queryResult.error = exec.error ?? 'Query failed'; + } + } + if (exec.tableName !== undefined && exec.tableName !== '') { + tab.tableName = exec.tableName; + } + } + if (tab.dataSource instanceof BigtraceAsyncDataSource) { + await tab.dataSource.refresh(); + tab.lastProcessedRows = exec?.processedRows ?? 0; + } + } + } finally { + if (tab.isLoading) { + this.cb.onHistoryChanged(); + } + tab.isLoading = false; this.cb.redraw(); } - - // Fetch full execution details for timing + error message. - void tab.queryClient - ?.getQueryExecution(tab.queryUuid!, tab.lifecycle.signal) - .then((details: RawQueryExecution) => { - const endMs = isoToEpochMs(details.endTime); - if (tab.queryUuid) { - queryStore.update(tab.queryUuid, { - ...(endMs !== undefined ? {endTime: endMs} : {}), - ...(details.schema !== undefined ? {schema: details.schema} : {}), - ...(details.tableName !== undefined - ? {tableName: details.tableName} - : {}), - }); - } - if (isFailed && tab.queryResult !== undefined) { - tab.queryResult.error = details.errorMessage || 'Query failed'; - } - this.cb.redraw(); - }) - .catch((e: unknown) => { - console.error('Failed to fetch query execution details:', e); - if (isFailed && tab.queryResult !== undefined) { - tab.queryResult.error = `Failed to fetch error details: ${e instanceof Error ? e.message : String(e)}`; - this.cb.redraw(); - } - }); - - if (isSuccess && tab.dataSource instanceof BigtraceAsyncDataSource) { - tab.dataSource.refresh(); - tab.lastProcessedRows = tab.execution?.processedRows ?? 0; - } } }
diff --git a/ui/src/bigtrace/query/query_history.ts b/ui/src/bigtrace/query/query_history.ts index a914530..b2ec9f6 100644 --- a/ui/src/bigtrace/query/query_history.ts +++ b/ui/src/bigtrace/query/query_history.ts
@@ -185,9 +185,7 @@ defaultOpen: this.allExpanded, summary: `${group.label} (${group.entries.length})`, }, - group.entries.map((entry, index) => - renderHistoryItem(entry, index, openQuery), - ), + group.entries.map((entry) => renderHistoryItem(entry, openQuery)), ), ), );
diff --git a/ui/src/bigtrace/query/query_history_item.ts b/ui/src/bigtrace/query/query_history_item.ts index d0ef1e3..96cb913 100644 --- a/ui/src/bigtrace/query/query_history_item.ts +++ b/ui/src/bigtrace/query/query_history_item.ts
@@ -33,7 +33,8 @@ formatCompactTime, } from './history_store'; -// Reopens an existing history entry. +// Reopens an existing history entry (uuid set) or, with an empty uuid and +// queryEngine 'duckdb', stages a DuckDB query over a table without running it. export type OpenQueryFn = ( query: string, uuid: string, @@ -41,6 +42,7 @@ forceNew?: boolean, limit?: number, startTime?: number, + queryEngine?: 'traces' | 'duckdb', ) => void; // SQL block clamped to ~4 lines with a fade-out mask; click to expand. @@ -118,7 +120,6 @@ export function renderHistoryItem( entry: QueryExecution, - index: number, openQuery?: OpenQueryFn, ): m.Children { // One mixed list, so the kind comes from the entry itself. @@ -140,7 +141,29 @@ const openThis = () => { if (openQuery && uuid) { - openQuery(queryText, uuid, isMaterialized, false, entry.limit, startTime); + openQuery( + queryText, + uuid, + isMaterialized, + false, + entry.limit, + startTime, + entry.queryEngine, + ); + } + }; + + const queryInDuckdb = () => { + if (openQuery && entry.tableName) { + openQuery( + `SELECT *\nFROM ${entry.tableName};\n`, + '', + true, + false, + undefined, + undefined, + 'duckdb', + ); } }; @@ -149,11 +172,23 @@ { className: 'pf-query-history__item-buttons', orientation: 'horizontal', + spacing: 'none', }, [ + ...(Boolean(entry.tableName) && rows > 0 + ? [ + m(Button, { + onclick: queryInDuckdb, + icon: 'database', + compact: true, + title: `Query ${entry.tableName} in DuckDB`, + }), + ] + : []), m(Button, { onclick: openThis, icon: Icons.ChangeTab, + compact: true, title: 'Open', }), @@ -189,10 +224,12 @@ ], }); if (!confirmed) return; + historyStore.remove(uuid); await queryHistoryStorage.deleteQuery(uuid); historyStore.refreshNow(); }, icon: Icons.Delete, + compact: true, // Red hover so destructive intent reads before click. intent: Intent.Danger, title: 'Delete query', @@ -200,9 +237,12 @@ ], ); + const hasLocalDuckdbTable = + Boolean(entry.tableName) && rows > 0 && (link === undefined || link === ''); + return m( '.pf-query-history__item', - {key: `${uuid}-${index}`}, + {key: uuid}, m( '.pf-bt-history-item-meta', { @@ -227,11 +267,18 @@ m( 'span.pf-bt-history-item-kind', { - title: isMaterialized - ? 'Results saved to a backend table — reopen to browse them.' - : 'Results were shown inline at run time and not saved.', + title: + entry.queryEngine === 'duckdb' + ? 'Executed in DuckDB over materialized tables.' + : isMaterialized + ? 'Results saved to a backend table — reopen to browse or query in DuckDB.' + : 'Results were shown inline at run time.', }, - isMaterialized ? 'Persistent' : 'Ephemeral', + entry.queryEngine === 'duckdb' + ? 'DuckDB' + : isMaterialized + ? 'Persistent' + : 'Ephemeral', ), m( 'span.pf-bt-history-item-date', @@ -241,8 +288,8 @@ ]), ], ), - // Materialized only: a banded strip between the header and the SQL pre. - isMaterialized && + // Materialized or local DuckDB table: a banded strip between the header and the SQL pre. + (isMaterialized || Boolean(entry.tableName)) && m( 'div.pf-bt-history-item-details', { @@ -253,15 +300,24 @@ 'a.pf-bt-history-item-table-link', { class: - rows === 0 || link === undefined || link === '' + rows === 0 || + ((link === undefined || link === '') && !hasLocalDuckdbTable) ? 'pf-bt-history-item-table-link--disabled' : 'pf-bt-history-item-table-link--active', href: link || '#', - target: '_blank', + target: link ? '_blank' : undefined, + onclick: hasLocalDuckdbTable + ? (e: MouseEvent) => { + e.preventDefault(); + queryInDuckdb(); + } + : undefined, title: rows === 0 ? 'No table created for empty results' - : entry.tableName || 'View Table', + : hasLocalDuckdbTable + ? `Click to query ${entry.tableName} in DuckDB` + : entry.tableName || 'View Table', }, entry.tableName || '—', ),
diff --git a/ui/src/bigtrace/query/query_history_storage.ts b/ui/src/bigtrace/query/query_history_storage.ts index 00ae6e4..b0d9965 100644 --- a/ui/src/bigtrace/query/query_history_storage.ts +++ b/ui/src/bigtrace/query/query_history_storage.ts
@@ -35,6 +35,7 @@ readonly error?: string; readonly errorMessage?: string; readonly perfettoSql?: string; + readonly queryEngine?: 'traces' | 'duckdb'; readonly limit?: number; readonly traceLimit?: number; readonly materialized?: boolean; @@ -146,6 +147,7 @@ totalTraces: raw.totalTraces ?? 0, error: raw.error ?? raw.errorMessage, perfettoSql: raw.perfettoSql, + queryEngine: raw.queryEngine, limit: raw.limit, materialized: raw.materialized, tableName: raw.tableName,
diff --git a/ui/src/bigtrace/query/query_runner.ts b/ui/src/bigtrace/query/query_runner.ts index 1bcc3b2..4b0a938 100644 --- a/ui/src/bigtrace/query/query_runner.ts +++ b/ui/src/bigtrace/query/query_runner.ts
@@ -34,8 +34,11 @@ } from './query_history_storage'; import {queryStore, TERMINAL_STATUSES} from './query_store'; import { + buildMultiQuerySpecs, disabledSettingsFromSnapshot, effectiveTabSettings, + hasQueryText, + loadSlotIntoTab, makeQueryResponse, } from '../pages/query_tabs_state'; import type {BigTraceEditorTab} from '../pages/query_tabs_state'; @@ -55,18 +58,36 @@ constructor(private readonly cb: QueryRunnerCallbacks) { this.poller = new PollingController({ redraw: () => this.redraw(), - onHistoryChanged: cb.onHistoryChanged, + onHistoryChanged: () => { + cb.onHistoryChanged(); + }, + markDirty: () => { + cb.markDirty?.(); + }, }); } // Run `query` on `tab`. Aborts any in-flight query on the tab first. async run(tab: BigTraceEditorTab, query: string): Promise<void> { - if (!query) return; + const multiSubQueries = + tab.queryEngine !== 'duckdb' && tab.subQueries.length > 1 + ? buildMultiQuerySpecs(tab.subQueries) + : undefined; + const actualQuery = + multiSubQueries && multiSubQueries.length > 0 + ? multiSubQueries[0].perfettoSql + : query; + if (!hasQueryText(actualQuery)) return; tab.activeRequest?.abort(); + this.poller.stop(tab); + tab.pollGeneration++; tab.isLoading = true; + tab.queryUuid = undefined; + tab.execution = undefined; tab.queryResult = undefined; + tab.dataSource = undefined; tab.lastProcessedRows = 0; tab.clientStartTime = Date.now(); this.cb.markDirty?.(); @@ -76,7 +97,7 @@ const endpoint = getBigtraceEndpoint(); if (endpoint.trim() === '') { - tab.queryResult = makeQueryResponse(query, { + tab.queryResult = makeQueryResponse(actualQuery, { error: 'Set the BigTrace endpoint from the connection button (top right) ' + 'before running queries.', @@ -89,14 +110,7 @@ await bigTraceSettingsStorage.loadSettings(); - // Global defaults overridden by per-tab values, minus per-tab-disabled - // settings. Same helper the trace grid uses, so /trace_metadata and - // /execute_* stay in sync. const settings = effectiveTabSettings(tab); - - // Per-tab snapshot is the trace-selection source of truth: what the user - // staged here (seeded from globals at tab creation, then edited). Globals - // aren't re-read at run time. const traceFilters = tab.traceFilters; const traceOrderBy = tab.traceOrderBy; @@ -106,8 +120,6 @@ const cancelForward = forwardAbort(tab.lifecycle.signal, requestController); tab.activeRequest = requestController; - // null → use the schema's defaultVisible columns; an explicit list (incl. - // []) ships as-is; a schema-fetch failure attaches nothing. let traceMetadataColumns: readonly string[] = tab.traceMetadataColumns ?? []; if (tab.traceMetadataColumns === null) { @@ -122,21 +134,35 @@ } } const executeOptions: ExecuteOptions = { + engine: tab.queryEngine, traceFilters, traceMetadataColumns, traceOrderBy, traceLimit: tab.traceLimit, experimentFilter: toExperimentFilterSpec(tab.experimentFilter), tableTtlDays: tab.materialize ? tab.tableTtlDays : undefined, + tableName: + multiSubQueries && multiSubQueries.length > 0 + ? multiSubQueries[0].tableName + : tab.tableName.trim() || tab.title.trim() || undefined, + queries: + multiSubQueries && multiSubQueries.length > 1 + ? multiSubQueries + : undefined, }; + const isDuckdb = tab.queryEngine === 'duckdb'; + if (isDuckdb) { + tab.materialize = true; + tab.limit = 0; + } const wallStartMs = performance.now(); try { if (tab.materialize) { await this.runAsync( tab, - query, + actualQuery, queryClient, settings, executeOptions, @@ -146,7 +172,7 @@ } else { await this.runSync( tab, - query, + actualQuery, queryClient, settings, executeOptions, @@ -160,7 +186,7 @@ this.redraw(); return; } - tab.queryResult = makeQueryResponse(query, { + tab.queryResult = makeQueryResponse(actualQuery, { error: e instanceof Error ? e.message : String(e), durationMs: performance.now() - wallStartMs, }); @@ -190,12 +216,28 @@ tab.execution.status = 'CANCELLED'; tab.execution.endTime = Date.now(); } + for (const slot of tab.subQueries) { + if (slot.execution && slot.execution.status === 'IN_PROGRESS') { + slot.execution.status = 'CANCELLED'; + slot.execution.endTime = Date.now(); + } + } - if (tab.materialize && queryUuid && tab.queryClient) { - try { - await tab.queryClient.cancelQuery(queryUuid); - } catch (e) { - console.error(`Failed to cancel query ${queryUuid} on backend:`, e); + if (tab.materialize && tab.queryClient) { + const uuidsToCancel = + tab.subQueries.length > 1 + ? tab.subQueries + .map((s) => s.queryUuid) + .filter((u): u is string => Boolean(u)) + : queryUuid + ? [queryUuid] + : []; + for (const uuid of uuidsToCancel) { + try { + await tab.queryClient.cancelQuery(uuid); + } catch (e) { + console.error(`Failed to cancel query ${uuid} on backend:`, e); + } } } @@ -255,9 +297,12 @@ if (details.schema !== undefined) { exec.schema = details.schema; } - if (details.tableName !== undefined) { + if (details.tableName !== undefined && details.tableName !== '') { exec.tableName = details.tableName; + tab.tableName = details.tableName; + tab.title = details.tableName; } + tab.queryEngine = details.queryEngine === 'duckdb' ? 'duckdb' : 'traces'; exec.status = details.status ?? 'N/A'; exec.processedRows = details.processedRows ?? 0; exec.processedTraces = details.processedTraces ?? 0; @@ -343,6 +388,7 @@ details.errorMessage ?? 'Query failed without a specific error message.'; } + this.redraw(); } @@ -376,39 +422,100 @@ if (data.queryUuid === undefined || data.queryUuid === '') { throw new Error('Backend did not return a queryUuid for async execute'); } - tab.queryUuid = data.queryUuid; - tab.execution = queryStore.getOrCreate(tab.queryUuid, { - perfettoSql: query, - }); - try { - const details = await client.getQueryExecution( - tab.queryUuid, - tab.lifecycle.signal, - ); - const serverStartMs = isoToEpochMs(details?.startTime); - queryStore.update(tab.queryUuid, { - ...(serverStartMs !== undefined ? {startTime: serverStartMs} : {}), - ...(details?.schema !== undefined ? {schema: details.schema} : {}), - ...(details?.tableName !== undefined - ? {tableName: details.tableName} - : {}), + if ( + data.executions && + data.executions.length > 1 && + options.queries && + options.queries.length > 0 + ) { + for ( + let i = 0; + i < data.executions.length && i < options.queries.length; + i++ + ) { + const item = data.executions[i]; + const spec = options.queries[i] as { + slotId?: string; + perfettoSql: string; + tableName?: string; + }; + const slot = tab.subQueries.find((s) => s.id === spec.slotId); + if (!slot) continue; + slot.queryUuid = item.queryUuid; + if (item.tableName) slot.tableName = item.tableName; + slot.execution = queryStore.getOrCreate(item.queryUuid, { + perfettoSql: slot.sql, + tableName: item.tableName, + status: 'IN_PROGRESS', + }); + slot.dataSource = new BigtraceAsyncDataSource( + item.queryUuid, + client, + () => slot.execution?.processedRows ?? 0, + tab.lifecycle.signal, + () => slot.execution?.schema, + ); + slot.queryResult = makeQueryResponse(slot.sql, { + durationMs: performance.now() - wallStartMs, + }); + } + loadSlotIntoTab(tab, tab.activeSubQueryIndex); + const uuids = data.executions.map((e) => e.queryUuid).filter(Boolean); + this.redraw(); + this.poller.start(tab, uuids); + } else { + tab.queryUuid = data.queryUuid; + tab.execution = queryStore.getOrCreate(tab.queryUuid, { + perfettoSql: query, + status: 'IN_PROGRESS', }); - } catch (e) { - console.error('Failed to fetch query details after executeAsync:', e); + tab.dataSource = new BigtraceAsyncDataSource( + tab.queryUuid, + client, + () => tab.execution?.processedRows ?? 0, + tab.lifecycle.signal, + () => tab.execution?.schema, + ); + tab.queryResult = makeQueryResponse(query, { + durationMs: performance.now() - wallStartMs, + }); + this.redraw(); + this.poller.start(tab, [data.queryUuid]); } - this.poller.start(tab); - tab.dataSource = new BigtraceAsyncDataSource( - tab.queryUuid, - client, - () => tab.execution?.processedRows ?? 0, - tab.lifecycle.signal, - () => tab.execution?.schema, - ); - tab.queryResult = makeQueryResponse(query, { - durationMs: performance.now() - wallStartMs, - }); + if (tab.queryUuid) { + try { + const details = await client.getQueryExecution( + tab.queryUuid, + tab.lifecycle.signal, + ); + const serverStartMs = isoToEpochMs(details?.startTime); + queryStore.update(tab.queryUuid, { + ...(serverStartMs !== undefined ? {startTime: serverStartMs} : {}), + ...(details?.schema !== undefined ? {schema: details.schema} : {}), + ...(details?.tableName !== undefined + ? {tableName: details.tableName} + : {}), + ...(details?.processedRows !== undefined + ? {processedRows: details.processedRows} + : {}), + ...(details?.processedTraces !== undefined + ? {processedTraces: details.processedTraces} + : {}), + ...(details?.totalTraces !== undefined + ? {totalTraces: details.totalTraces} + : {}), + }); + if (details?.tableName !== undefined && details.tableName !== '') { + tab.tableName = details.tableName; + this.cb.markDirty?.(); + } + this.redraw(); + } catch (e) { + console.error('Failed to fetch query details after executeAsync:', e); + } + } } private async runSync( @@ -430,21 +537,98 @@ if (result.queryUuid === undefined || result.queryUuid === '') { throw new Error('Backend did not return a queryUuid for sync execute'); } - tab.queryUuid = result.queryUuid; - tab.execution = queryStore.getOrCreate(tab.queryUuid, { - perfettoSql: query, - }); - tab.queryResult = makeQueryResponse(query, { - rows: [...result.rows], - columns: [...result.columns], - schema: result.schema, - totalRowCount: result.rows.length, - durationMs: performance.now() - wallStartMs, - statementWithOutputCount: 1, - }); - queryStore.update(tab.queryUuid, { - processedRows: result.rows.length, - }); + + if ( + result.executions && + result.executions.length > 1 && + options.queries && + options.queries.length > 0 + ) { + for ( + let i = 0; + i < result.executions.length && i < options.queries.length; + i++ + ) { + const item = result.executions[i]; + const spec = options.queries[i] as { + slotId?: string; + perfettoSql: string; + tableName?: string; + }; + const slot = tab.subQueries.find((s) => s.id === spec.slotId); + if (!slot) continue; + slot.queryUuid = item.queryUuid; + if (item.tableName) slot.tableName = item.tableName; + if (item.queryUuid === result.queryUuid) { + slot.execution = queryStore.getOrCreate(item.queryUuid, { + perfettoSql: slot.sql, + tableName: item.tableName, + }); + slot.queryResult = makeQueryResponse(slot.sql, { + rows: [...result.rows], + columns: [...result.columns], + schema: result.schema, + totalRowCount: result.rows.length, + durationMs: performance.now() - wallStartMs, + statementWithOutputCount: 1, + }); + slot.dataSource = new InMemoryDataSource(slot.queryResult.rows); + } else { + try { + const page = await client.fetchResults( + item.queryUuid, + tab.limit, + 0, + signal, + ); + slot.execution = queryStore.getOrCreate(item.queryUuid, { + perfettoSql: slot.sql, + tableName: item.tableName, + }); + slot.queryResult = makeQueryResponse(slot.sql, { + rows: [...page.rows], + columns: [...page.columns], + schema: page.schema, + totalRowCount: page.rows.length, + durationMs: performance.now() - wallStartMs, + statementWithOutputCount: 1, + }); + slot.dataSource = new InMemoryDataSource(slot.queryResult.rows); + } catch (err) { + slot.queryResult = makeQueryResponse(slot.sql, { + error: err instanceof Error ? err.message : String(err), + }); + slot.dataSource = new InMemoryDataSource([]); + } + } + } + loadSlotIntoTab(tab, tab.activeSubQueryIndex); + } else { + tab.queryUuid = result.queryUuid; + const resolvedTable = + result.tableName || result.executions?.[0]?.tableName || undefined; + if (resolvedTable) { + tab.tableName = resolvedTable; + this.cb.markDirty?.(); + } + tab.execution = queryStore.getOrCreate(tab.queryUuid, { + perfettoSql: query, + tableName: resolvedTable, + }); + tab.queryResult = makeQueryResponse(query, { + rows: [...result.rows], + columns: [...result.columns], + schema: result.schema, + totalRowCount: result.rows.length, + durationMs: performance.now() - wallStartMs, + statementWithOutputCount: 1, + }); + if (tab.queryUuid) { + queryStore.update(tab.queryUuid, { + processedRows: result.rows.length, + }); + } + } tab.isLoading = false; } }
diff --git a/ui/src/bigtrace/query/query_store.ts b/ui/src/bigtrace/query/query_store.ts index 351b0df..d2af185 100644 --- a/ui/src/bigtrace/query/query_store.ts +++ b/ui/src/bigtrace/query/query_store.ts
@@ -80,6 +80,7 @@ totalTraces: number; error?: string; perfettoSql?: string; + queryEngine?: 'traces' | 'duckdb'; limit?: number; materialized?: boolean; tableName?: string; @@ -93,6 +94,10 @@ export class QueryStore { private queries = new Map<string, QueryExecution>(); + get(uuid: string): QueryExecution | undefined { + return this.queries.get(uuid); + } + getOrCreate( uuid: string, initialData?: Partial<QueryExecution>,
diff --git a/ui/src/bigtrace/settings/endpoint_storage.ts b/ui/src/bigtrace/settings/endpoint_storage.ts index f8cbe44..3a75e77 100644 --- a/ui/src/bigtrace/settings/endpoint_storage.ts +++ b/ui/src/bigtrace/settings/endpoint_storage.ts
@@ -23,18 +23,57 @@ new LocalStorage(BIGTRACE_SETTINGS_STORAGE_KEY), ); +const ENDPOINT_SETTING_ID = 'bigtraceEndpoint'; + +// `?server=<url>`, as printed by tools/bigtrace_server.py, selects the backend. +// It is saved like a manual edit, before registration so no reload is asked +// for, and dropped from the address bar so reloading doesn't undo a later +// change made in the connection panel. +function adoptEndpointFromUrl(): void { + if (typeof window === 'undefined') return; + const url = new URL(window.location.href); + const endpoint = url.searchParams.get('server'); + if (!endpoint) return; + endpointStorage.setStoredValue(ENDPOINT_SETTING_ID, endpoint); + url.searchParams.delete('server'); + window.history.replaceState(window.history.state, '', url.toString()); +} + +adoptEndpointFromUrl(); + +function defaultBigtraceEndpoint(): string { + if ( + typeof window !== 'undefined' && + window.location?.port === '10000' && + typeof window.location.origin === 'string' + ) { + return `${window.location.origin}/v1`; + } + return 'https://brush-googleapis.corp.google.com/v1'; +} + endpointStorage.register({ - id: 'bigtraceEndpoint', + id: ENDPOINT_SETTING_ID, name: 'BigTrace Endpoint', description: 'The URL of the BigTrace backend service.', schema: z.string(), - defaultValue: 'https://brush-googleapis.corp.google.com/v1', + defaultValue: defaultBigtraceEndpoint(), requiresReload: true, }); // The configured BigTrace endpoint URL, or '' if unset. Centralizes the // null/empty handling call sites would otherwise re-inline. export function getBigtraceEndpoint(): string { - const setting = endpointStorage.get('bigtraceEndpoint'); - return setting ? ((setting.get() as string) ?? '') : ''; + const setting = endpointStorage.get(ENDPOINT_SETTING_ID); + const raw = setting ? ((setting.get() as string) ?? '') : ''; + if ( + typeof window !== 'undefined' && + window.location?.port === '10000' && + (raw === 'https://brush-googleapis.corp.google.com/v1' || + raw === 'http://127.0.0.1:5001' || + raw === 'http://localhost:5001') + ) { + return `${window.location.origin}/v1`; + } + return raw; }
diff --git a/ui/src/test/bigtrace.test.ts b/ui/src/test/bigtrace.test.ts new file mode 100644 index 0000000..e3ba92c --- /dev/null +++ b/ui/src/test/bigtrace.test.ts
@@ -0,0 +1,385 @@ +// Copyright (C) 2026 The Android Open Source Project +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import {test, type Page, expect} from '@playwright/test'; +import {spawn, type ChildProcess} from 'child_process'; +import fs from 'fs'; +import os from 'os'; +import path from 'path'; +import {getFreePort, waitForServer} from './perfetto_ui_test_helper'; + +test.skip( + process.env.BIGTRACE_E2E !== '1' && process.env.BIGTRACE_E2E !== 'true', + 'Set BIGTRACE_E2E=1 to run Bigtrace live server E2E suite', +); + +let page: Page; +let bigtraceServerProc: ChildProcess | undefined; +let bigtracePort: number; +let tracesDir: string; +let targetUrl: string; + +async function killProcess(proc?: ChildProcess): Promise<void> { + if (proc === undefined || proc.exitCode !== null) return; + const exitPromise = new Promise((resolve) => proc.once('exit', resolve)); + proc.kill('SIGTERM'); + const timer = setTimeout(() => proc.kill('SIGKILL'), 3000); + await exitPromise; + clearTimeout(timer); +} + +test.describe('Bigtrace UI E2E suite', () => { + test.describe.configure({mode: 'serial'}); + + test.beforeAll(async ({browser}) => { + test.setTimeout(120_000); + + // 1. Create a clean temp directory with trace symlinks. + tracesDir = fs.mkdtempSync(path.join(os.tmpdir(), 'bt_e2e_traces_')); + const candidateDirs = [ + process.env.BIGTRACE_TRACES, + path.resolve(__dirname, '../../../test/data'), + path.resolve(__dirname, '../../../../dev/perfetto/test/data'), + path.resolve(os.tmpdir(), 'bt_test_traces'), + ].filter((d): d is string => Boolean(d && fs.existsSync(d))); + + for (const d of candidateDirs) { + const traces = fs + .readdirSync(d) + .filter((f) => f.endsWith('.perfetto-trace')); + if (traces.length > 0) { + for (const f of traces) { + try { + fs.symlinkSync(path.join(d, f), path.join(tracesDir, f)); + } catch { + // Ignore + } + } + break; + } + } + expect( + fs.readdirSync(tracesDir).filter((f) => f.endsWith('.perfetto-trace')) + .length, + ).toBeGreaterThan(0); + + // 2. Start Bigtrace Python backend on a free port. + bigtracePort = await getFreePort(); + const pythonBin = process.env.BIGTRACE_PYTHON || 'python3'; + const serverScript = path.resolve( + __dirname, + '../../../tools/bigtrace_server.py', + ); + const shellPath = + process.env.BIGTRACE_SHELL || + path.resolve(__dirname, '../../../out/release/trace_processor_shell'); + + bigtraceServerProc = spawn( + pythonBin, + [ + serverScript, + tracesDir, + '-p', + String(bigtracePort), + '--shell-path', + shellPath, + ], + { + stdio: 'ignore', + }, + ); + + const backendReady = await waitForServer( + `http://127.0.0.1:${bigtracePort}/v1/`, + 20_000, + ); + expect(backendReady).toBe(true); + + targetUrl = `/bigtrace.html?server=http://127.0.0.1:${bigtracePort}/v1`; + page = await browser.newPage(); + }); + + test.afterAll(async () => { + await killProcess(bigtraceServerProc); + if (tracesDir !== '' && fs.existsSync(tracesDir)) { + try { + fs.rmSync(tracesDir, {recursive: true, force: true}); + } catch { + // Ignore + } + } + if (page !== undefined) { + await page.close(); + } + }); + + test.beforeEach(async () => { + await page.goto(targetUrl, {waitUntil: 'domcontentloaded'}); + await ensureEditorOpen(); + }); + + async function ensureEditorOpen() { + const editor = page.locator('.pf-editor .cm-content'); + const startQueryBtn = page.locator('button:has-text("Start Query")'); + const winner = await Promise.race([ + editor + .waitFor({state: 'visible', timeout: 2000}) + .then(() => 'editor' as const) + .catch(() => null), + startQueryBtn + .waitFor({state: 'visible', timeout: 2000}) + .then(() => 'btn' as const) + .catch(() => null), + ]); + if (winner === 'btn') { + await startQueryBtn.click(); + } + await editor.waitFor({state: 'visible', timeout: 10_000}); + } + + async function setEditorText(sql: string) { + const editor = page.locator('.pf-editor .cm-content'); + await editor.click(); + await page.keyboard.press('ControlOrMeta+A'); + await page.keyboard.press('Backspace'); + await page.keyboard.insertText(sql); + } + + async function clickRun() { + const runBtn = page + .locator('.pf-bt-query-page__toolbar button:has-text("Run")') + .first(); + await expect(runBtn).toBeEnabled(); + await runBtn.click(); + + const overwriteBtn = page.locator('button:has-text("Overwrite")'); + const runningBar = page.locator('.pf-bt-status-bar--running'); + const outcome = await Promise.race([ + overwriteBtn + .waitFor({state: 'visible', timeout: 1500}) + .then(() => 'modal' as const) + .catch(() => null), + runningBar + .waitFor({state: 'visible', timeout: 1500}) + .then(() => 'running' as const) + .catch(() => null), + ]); + if (outcome === 'modal') { + await overwriteBtn.click(); + } + } + + test('connect to server endpoint and open Bigtrace UI', async () => { + test.setTimeout(60_000); + const connBtn = page.locator('.pf-button-group button').first(); + await expect(connBtn).toBeVisible({timeout: 15_000}); + await expect(connBtn.locator('i')).toHaveText('cloud_done'); + }); + + test('run a trace query and see rows in results grid', async () => { + test.setTimeout(60_000); + await setEditorText( + 'SELECT id, name, ts, dur, track_id FROM slice LIMIT 10', + ); + await clickRun(); + + const statusBar = page.locator('.pf-bt-status-bar'); + await expect(statusBar).toBeVisible({timeout: 20_000}); + await expect(page.locator('.pf-bt-status-bar--running')).toHaveCount(0, { + timeout: 30_000, + }); + + const summary = page.locator('.pf-bt-results-summary'); + await expect(summary).toBeVisible({timeout: 10_000}); + await expect(summary).toContainText(/Showing \d+ rows|Returned \d+ rows/); + await expect( + page.locator('.pf-bt-query-page__results').first(), + ).toBeVisible({timeout: 10_000}); + await expect(page.locator('.pf-bt-results-error')).toHaveCount(0); + }); + + test('limits including 0 (= no limit) preserved across rerun', async () => { + test.setTimeout(60_000); + const limitInput = page.locator('.pf-bt-limit-input input'); + await expect(limitInput).toBeVisible({timeout: 10_000}); + await expect(limitInput).toHaveAttribute( + 'title', + 'Maximum rows this query returns (0 = unlimited).', + ); + + await limitInput.click(); + await limitInput.fill('0'); + await clickRun(); + + await expect(page.locator('.pf-bt-status-bar--running')).toHaveCount(0, { + timeout: 30_000, + }); + const statusVal = page.locator('.pf-bt-status-bar-stat-value').last(); + await expect(statusVal).toHaveAttribute('title', /no limit/); + await expect(limitInput).toHaveValue('0'); + await expect(page.locator('.pf-bt-results-summary')).toContainText( + /Showing \d+ rows/, + ); + await expect(page.locator('.pf-bt-results-error')).toHaveCount(0); + }); + + test('multi-query (slices + sched) in one run', async () => { + test.setTimeout(90_000); + const addQueryBtn = page.locator('button:has-text("Add Query")').first(); + await expect(addQueryBtn).toBeVisible({timeout: 10_000}); + await addQueryBtn.click(); + + const subqueryBar = page.locator('.pf-bt-subquery-bar'); + await expect(subqueryBar).toBeVisible({timeout: 10_000}); + const pills = subqueryBar.locator('.pf-bt-subquery-pill'); + await expect(pills).toHaveCount(2); + + // Query #1: already executed in previous test, so double-click rename is locked + await pills.nth(0).locator('button').first().click(); + await pills.nth(0).locator('button').first().dblclick(); + await expect( + pills.nth(0).locator('input.pf-bt-subquery-rename-input'), + ).toHaveCount(0); + await setEditorText( + 'SELECT id, name, ts, dur, track_id FROM slice LIMIT 10', + ); + + // Query #2: not executed yet, so double-click inline rename on the chip works + await pills.nth(1).locator('button').first().click(); + await pills.nth(1).locator('button').first().dblclick(); + const renameInput1 = pills + .nth(1) + .locator('input.pf-bt-subquery-rename-input'); + await expect(renameInput1).toBeVisible({timeout: 5000}); + await renameInput1.fill('sched'); + await renameInput1.press('Enter'); + await expect(pills.nth(1)).toContainText('sched'); + await setEditorText('SELECT id, ts, dur, cpu, utid FROM sched LIMIT 10'); + + // Click Run -> slot #0 already ran so Overwrite Table modal appears; click Overwrite + await clickRun(); + const overwriteBtn = page.locator('.pf-modal button:has-text("Overwrite")'); + if (await overwriteBtn.isVisible({timeout: 3000}).catch(() => false)) { + await overwriteBtn.click(); + } + await expect(page.locator('.pf-bt-status-bar--running')).toHaveCount(0, { + timeout: 40_000, + }); + await expect(page.locator('.pf-bt-results-error')).toHaveCount(0); + + // After running, Query #2 is now locked against rename too + await pills.nth(1).locator('button').first().dblclick(); + await expect( + pills.nth(1).locator('input.pf-bt-subquery-rename-input'), + ).toHaveCount(0); + + // Pill #1: verify row counts and track_id column header + await pills.nth(0).locator('button').first().click(); + await expect(pills.nth(0)).toContainText(/\(\d+\)/); + await expect(page.locator('.pf-bt-results-summary')).toContainText( + /Showing \d+ rows/, + ); + await expect( + page.locator('.pf-bt-query-page__results').first(), + ).toContainText('track_id'); + + // Pill #2: verify row counts and cpu/utid column header + await pills.nth(1).locator('button').first().click(); + await expect(pills.nth(1)).toContainText(/\(\d+\)/); + await expect(page.locator('.pf-bt-results-summary')).toContainText( + /Showing \d+ rows/, + ); + await expect( + page.locator('.pf-bt-query-page__results').first(), + ).toContainText('cpu'); + }); + + test('switch to DuckDB SQL mode and run |> pipeline and pf_interval_intersection over extracted tables', async () => { + test.setTimeout(90_000); + const modeBtn = page.locator('.pf-button-group button').nth(1); + await expect(modeBtn).toBeVisible({timeout: 10_000}); + await modeBtn.click(); + await page + .locator('.pf-popup-content button:has-text("DuckDB SQL")') + .click(); + + // Persistent switch, Rows limit input, and Settings button are hidden in DuckDB Mode + await expect(page.locator('.pf-bt-limit-input')).toHaveCount(0); + + await setEditorText( + `INTERVAL INTERSECTION OF ( + (FROM slice |> WHERE dur >= 0) AS s, + (FROM sched |> WHERE dur >= 0) AS c + ) PER (_trace_uuid) + |> AGGREGATE COUNT(*) AS cnt`, + ); + + await clickRun(); + await expect(page.locator('.pf-bt-status-bar--running')).toHaveCount(0, { + timeout: 40_000, + }); + await expect(page.locator('.pf-bt-status-bar')).toBeVisible(); + await expect(page.locator('.pf-bt-results-error')).toHaveCount(0); + await expect( + page.locator('.pf-bt-query-page__results').first(), + ).toBeVisible({timeout: 10_000}); + }); + + test('cancel query in flight', async () => { + test.setTimeout(60_000); + // Switch to PerfettoSQL if tab was left in DuckDB mode + const modeBtn = page.locator('.pf-button-group button').nth(1); + if ((await modeBtn.textContent())?.includes('DuckDB')) { + await modeBtn.click(); + await page + .locator('.pf-popup-content button:has-text("PerfettoSQL")') + .click(); + await ensureEditorOpen(); + } + + await setEditorText( + 'SELECT s1.id FROM slice s1 JOIN slice s2 ON s1.dur = s2.dur JOIN slice s3 ON s2.dur = s3.dur', + ); + + await clickRun(); + + const cancelBtn = page.locator( + '.pf-bt-query-page__toolbar button:has-text("Cancel")', + ); + await expect(cancelBtn).toBeVisible({timeout: 5000}); + await cancelBtn.click(); + + await expect( + page.locator('.pf-bt-query-page__toolbar button:has-text("Run")').first(), + ).toBeVisible({timeout: 10_000}); + await expect(page.locator('.pf-bt-status-bar--running')).toHaveCount(0); + }); + + test('history panel displays completed executions and supports staging DuckDB queries', async () => { + test.setTimeout(60_000); + const historyItems = page.locator('.pf-query-history__item'); + await expect(historyItems.first()).toBeVisible({timeout: 15_000}); + expect(await historyItems.count()).toBeGreaterThan(0); + + const duckdbIconBtn = page + .locator( + '.pf-query-history__item button:has-text("database"), .pf-query-history__item button[title*="DuckDB"]', + ) + .first(); + await expect(duckdbIconBtn).toBeVisible({timeout: 5000}); + await duckdbIconBtn.click(); + const editor = page.locator('.pf-editor .cm-content').last(); + await expect(editor).toContainText('FROM'); + }); +});
diff --git a/ui/src/test/perfetto_ui_test_helper.ts b/ui/src/test/perfetto_ui_test_helper.ts index 3ba375f..1133bd9 100644 --- a/ui/src/test/perfetto_ui_test_helper.ts +++ b/ui/src/test/perfetto_ui_test_helper.ts
@@ -241,3 +241,32 @@ await this.page.evaluate(() => self.app.raf.scheduleFullRedraw()); } } + +export async function getFreePort(): Promise<number> { + const net = await import('net'); + return new Promise((resolve, reject) => { + const srv = net.createServer(); + srv.listen(0, '127.0.0.1', () => { + const port = (srv.address() as {port: number}).port; + srv.close(() => resolve(port)); + }); + srv.on('error', reject); + }); +} + +export async function waitForServer( + url: string, + timeoutMs = 15000, +): Promise<boolean> { + const start = Date.now(); + while (Date.now() - start < timeoutMs) { + try { + const res = await fetch(url); + if (res.ok) return true; + } catch { + // Retry until server responds. + } + await new Promise((r) => setTimeout(r, 200)); + } + return false; +}
diff --git a/ui/src/widgets/accordion.ts b/ui/src/widgets/accordion.ts index 1382111..a9c5202 100644 --- a/ui/src/widgets/accordion.ts +++ b/ui/src/widgets/accordion.ts
@@ -67,7 +67,12 @@ open: this.isOpen, ...(groupName != null ? {name: groupName} : {}), ontoggle: (e: Event) => { - this.isOpen = (e.target as HTMLDetailsElement).open; + const next = (e.target as HTMLDetailsElement).open; + if (next === this.isOpen) { + (e as Event & {redraw?: boolean}).redraw = false; + return; + } + this.isOpen = next; }, }, m(