Python forkserver

A per-build server that starts the Python interpreter and imports common modules once, then runs each Python build action in a fork()ed copy of itself. Each action still gets its own process; it just skips interpreter startup and the imports the server already did.

Design doc: go/fuchsia-python-forkserver.

  • server.py: the forkserver's controller: warms up, starts the forkers, and merges their stats when it stops.
  • forker.py: a forker process: accepts requests and forks a worker for each.
  • stats.py: the counters, per forker and merged.
  • worker.py: warms the server up, and turns a forked worker into the process that runs the script.
  • client.py: the launcher. python3 -S client.py SCRIPT ARGS runs SCRIPT on the server when one is advertised, or execs python3 -S SCRIPT ARGS otherwise.
  • setting.py: reads the //.fx/config/python_forkserver opt-in.
  • example/: a toy build action, and how to run it on a server by hand.

Linux only.

Cheat sheet

Want to...Do
Run a script on the serverFX_PYTHON_FORKSERVER_SOCKET=NAME python3 -S client.py SCRIPT ARGS
Run it without the serverUnset FX_PYTHON_FORKSERVER_SOCKET (same command)
Change parallelismserver.py --forkers N (default: min(8, cpu_count))
Get totals on exitserver.py --stats FILE (one JSON line)
Stop the serverSIGTERM (or SIGINT) to the controller (server.py)

What runs where

The server is a controller process (server.py) and --forkers forker processes (forker.py). The controller binds the socket, warms up, forks the forkers, and waits for SIGTERM (or SIGHUP or SIGINT). Each forker accepts connections and forks a worker per request, which runs SCRIPT (worker.py). On SIGTERM, the controller shuts the socket down, each forker waits for its workers and writes its stats to <stats>.<pid>, and the controller merges them into <stats> (stats.py).

server.py (controller)
  bind → warm up → ready byte → fork N forkers → sigwait() until SIGTERM/HUP/INT
  └─ forker × N  loop: accept → check peer uid → send READY → fork worker
      │                → send PID
      │          on SIGCHLD (via a wakeup fd): reap worker → send STATUS → close
      └─ worker (one per action)
           receive request + client's stdio fds
           → set argv/env/cwd/umask/sys.path as `python3 -S SCRIPT` would
           → run SCRIPT in-process → exit with its status

client.py (one per action)
  connect → wait READY → send request + stdio fds → wait PID
  → forward SIGINT/SIGTERM/SIGHUP to the worker → wait STATUS
  → exit with the same status (or die from the same signal)

The client is a messenger: it sends SCRIPT, its arguments, the environment, working directory, umask, and stdio to the server, and nothing about its own interpreter. Served scripts run with the server‘s interpreter and startup flags (-S -W error::SyntaxWarning in the build), and PYTHON* startup variables such as PYTHONHASHSEED and PYTHONUTF8 are read once, when the server starts, not from each request (except for those listed below). The server needs its own directory on sys.path, so don’t start it with -P or PYTHONSAFEPATH. Each worker gives SCRIPT the module search path that python3 -S SCRIPT would compute from the request‘s PYTHONPATH, plus its environment, working directory, umask, stdio, bytecode settings (PYTHONDONTWRITEBYTECODE, PYTHONPYCACHEPREFIX), and a fresh __main__. SIGINT, SIGTERM, and SIGHUP sent to the client reach the worker’s process group, and the client exits with the worker's status (or dies from the same signal).

So in GN, only put client.py in python_exe_src -S commands, right after the interpreter flags. A served script gets the server's flags, including -W error::SyntaxWarning, even if its command line only has -S.

When FX_PYTHON_FORKSERVER_SOCKET is unset or empty, the client warns once per directory (it creates .python_forkserver_slow_path there) and execs SCRIPT with the command line's own interpreter and flags.

Everything else fails the action, with a message saying what to do (usually, how to turn the forkserver off):

  • the server isn‘t running, closes the connection, or doesn’t answer within 2 s,
  • the client's stdin, stdout, or stderr is closed, or
  • the request comes from another working directory than the one the server warmed up in (for example, a nested build).

Protocol

One request per connection on an abstract AF_UNIX stream socket. The constants and Request, the request encoding, live in client.py, which server.py imports.

Server-to-client messages are 5 bytes: a tag byte and an i32le value.

  1. Server to client: R (value 0) once the connection is accepted from a client running as the same user. The client only sends its stdio fds after this, so a hung server's queue of unaccepted connections never holds pipe write ends open.
  2. Client to server: a u32le length, then NUL-separated fields: the magic string, cwd, umask, argc, argv[0..argc), and KEY=VALUE environment entries. File descriptors 0, 1, and 2 ride along as SCM_RIGHTS on the first sendmsg(). The server closes the connection on a malformed request.
  3. Server to client:
    • P: the worker's pid, which is also its process group.
    • S: the worker's wait status.

Testing

The tests run during the build, the same way CQ runs them:

fx build //build/scripts/python_forkserver:tests

To iterate on one test file without a build, run it the way the build does: with python3 -S, PYTHONPATH set to the checkout root, and no PATH.

cd build/scripts/python_forkserver
env -i HOME="$HOME" PYTHONPATH="$PWD/../../.." \
  ../../../prebuilt/third_party/python3/linux-x64/bin/python3 -S client_test.py

Build actions may have no PATH and may set PYTHONPYCACHEPREFIX, so tests start helper processes with sys.executable rather than host tools like sleep, and don't assume where bytecode caches go.

Debugging

  • Server log: whatever you redirect the server's stderr to. It prints one line at warm-up and a stats line on exit.
  • If one of the server‘s forker processes exits early (for example, on EMFILE), the server logs which one, stops, and exits 1; actions after that fail with "isn’t running".
  • To compare against a plain run, unset FX_PYTHON_FORKSERVER_SOCKET and rerun the same command.