blob: 87c4dbaffa2c52812ea970bcd1530251fd5b6ca6 [file] [view]
# Python forkserver
A per-build server that starts the Python interpreter and imports common modules
once, then runs each Python build action in a `fork()`ed copy of itself. Each
action still gets its own process; it just skips interpreter startup and
the imports the server already did.
Design doc: [go/fuchsia-python-forkserver](http://go/fuchsia-python-forkserver).
* `server.py`: the forkserver's controller: warms up, starts the forkers, and
merges their stats when it stops.
* `forker.py`: a forker process: accepts requests and forks a worker for each.
* `stats.py`: the counters, per forker and merged.
* `worker.py`: warms the server up, and turns a forked worker into the process
that runs the script.
* `client.py`: the launcher. `python3 -S client.py SCRIPT ARGS` runs `SCRIPT` on
the server when one is advertised, or execs `python3 -S SCRIPT ARGS`
otherwise.
* `setting.py`: reads the `//.fx/config/python_forkserver` opt-in.
* `example/`: a toy build action, and how to run it on a server by hand.
Linux only.
## Cheat sheet
| Want to... | Do |
|---|---|
| Run a script on the server | `FX_PYTHON_FORKSERVER_SOCKET=NAME python3 -S client.py SCRIPT ARGS` |
| Run it without the server | Unset `FX_PYTHON_FORKSERVER_SOCKET` (same command) |
| Change parallelism | `server.py --forkers N` (default: `min(8, cpu_count)`) |
| Get totals on exit | `server.py --stats FILE` (one JSON line) |
| Stop the server | `SIGTERM` (or `SIGINT`) to the controller (`server.py`) |
## What runs where
The server is a controller process (`server.py`) and `--forkers` forker
processes (`forker.py`). The controller binds the socket, warms up, forks the
forkers, and waits for `SIGTERM` (or `SIGHUP` or `SIGINT`). Each forker accepts
connections and forks a worker per request, which runs `SCRIPT` (`worker.py`).
On `SIGTERM`, the controller shuts the socket down, each forker waits for its
workers and writes its stats to `<stats>.<pid>`, and the controller merges them
into `<stats>` (`stats.py`).
```
server.py (controller)
bind → warm up → ready byte → fork N forkers → sigwait() until SIGTERM/HUP/INT
└─ forker × N loop: accept → check peer uid → send READY → fork worker
│ → send PID
│ on SIGCHLD (via a wakeup fd): reap worker → send STATUS → close
└─ worker (one per action)
receive request + client's stdio fds
→ set argv/env/cwd/umask/sys.path as `python3 -S SCRIPT` would
→ run SCRIPT in-process → exit with its status
client.py (one per action)
connect → wait READY → send request + stdio fds → wait PID
→ forward SIGINT/SIGTERM/SIGHUP to the worker → wait STATUS
→ exit with the same status (or die from the same signal)
```
The client is a messenger: it sends `SCRIPT`, its arguments, the environment,
working directory, umask, and stdio to the server, and nothing about its own
interpreter. Served scripts run with the server's interpreter and startup flags
(`-S -W error::SyntaxWarning` in the build), and `PYTHON*` startup variables
such as `PYTHONHASHSEED` and `PYTHONUTF8` are read once, when the server starts,
not from each request (except for those listed below). The server needs its own
directory on `sys.path`, so don't start it with `-P` or `PYTHONSAFEPATH`. Each
worker gives `SCRIPT` the module search path that `python3 -S SCRIPT` would
compute from the request's `PYTHONPATH`, plus its environment, working
directory, umask, stdio, bytecode settings (`PYTHONDONTWRITEBYTECODE`,
`PYTHONPYCACHEPREFIX`), and a fresh `__main__`. `SIGINT`, `SIGTERM`, and
`SIGHUP` sent to the client reach the worker's process group, and the client
exits with the worker's status (or dies from the same signal).
So in GN, only put `client.py` in `python_exe_src -S` commands, right after
the interpreter flags. A served script gets the server's flags, including
`-W error::SyntaxWarning`, even if its command line only has `-S`.
When `FX_PYTHON_FORKSERVER_SOCKET` is unset or empty, the client warns once per
directory (it creates `.python_forkserver_slow_path` there) and execs `SCRIPT`
with the command line's own interpreter and flags.
Everything else fails the action, with a message saying what to do (usually,
how to turn the forkserver off):
* the server isn't running, closes the connection, or doesn't answer within
2 s,
* the client's stdin, stdout, or stderr is closed, or
* the request comes from another working directory than the one the server
warmed up in (for example, a nested build).
## Protocol
One request per connection on an abstract `AF_UNIX` stream socket. The
constants and `Request`, the request encoding, live in `client.py`, which
`server.py` imports.
Server-to-client messages are 5 bytes: a tag byte and an i32le value.
1. Server to client: `R` (value 0) once the connection is accepted from a
client running as the same user. The client only sends its stdio fds after
this, so a hung server's queue of unaccepted connections never holds pipe
write ends open.
2. Client to server: a u32le length, then NUL-separated fields: the magic
string, cwd, umask, argc, `argv[0..argc)`, and `KEY=VALUE` environment
entries. File descriptors 0, 1, and 2 ride along as `SCM_RIGHTS` on the
first `sendmsg()`. The server closes the connection on a malformed request.
3. Server to client:
* `P`: the worker's pid, which is also its process group.
* `S`: the worker's wait status.
## Testing
The tests run during the build, the same way CQ runs them:
```sh
fx build //build/scripts/python_forkserver:tests
```
To iterate on one test file without a build, run it the way the build does:
with `python3 -S`, `PYTHONPATH` set to the checkout root, and no `PATH`.
```sh
cd build/scripts/python_forkserver
env -i HOME="$HOME" PYTHONPATH="$PWD/../../.." \
../../../prebuilt/third_party/python3/linux-x64/bin/python3 -S client_test.py
```
Build actions may have no `PATH` and may set `PYTHONPYCACHEPREFIX`, so tests
start helper processes with `sys.executable` rather than host tools like
`sleep`, and don't assume where bytecode caches go.
## Debugging
* Server log: whatever you redirect the server's stderr to. It prints one line
at warm-up and a stats line on exit.
* If one of the server's forker processes exits early (for example, on
EMFILE), the server logs which one, stops, and exits 1; actions after that
fail with "isn't running".
* To compare against a plain run, unset `FX_PYTHON_FORKSERVER_SOCKET` and rerun
the same command.