commit 4eab1fe02f7823e32c6f14d41172298d47651760
author Oliver Newman <olivernewman@google.com> Wed Jun 07 15:44:59 2023 +0000
committer CQ Bot <fuchsia-internal-scoped@luci-project-accounts.iam.gserviceaccount.com> Wed Jun 07 15:44:59 2023 +0000
tree 744be1ae6ff03e0ae2c1de05a72c610e4e97dfb3
parent c5b323c25d898438995228fc05348bb0ee24d22b

[engine] Make ctx.os.exec() start background process

... and return an object representing the started process, with a
`wait()` method that will block until the process completes.

This makes it possible to run multiple subprocesses in parallel, which
can drastically speed up checks that require running one subprocess for
every file.

TODOs:
1. The pattern of iterating over all the processes sequentially and
   `wait()`ing on them is not super efficient - more efficient would be
   to implement an `os.ctx.select()` iterator that returns the first
   subprocess to complete. But that's a small optimization and not
   necessary yet. If and when we do implement that, it will likely
   require making `subprocess` objects hashable so they can be used as
   keys in a dict with filenames as values, to look up which file a
   per-file subprocess corresponds to after it completes.
2. Limit the number of subprocesses that can be running concurrently, to
   NumCPU()+2 or so.

Change-Id: I43395e7481685460c5a92e02fba439fecf006c2b
Reviewed-on: https://fuchsia-review.googlesource.com/c/shac-project/shac/+/867676
Reviewed-by: Marc-Antoine Ruel <maruel@google.com>
Commit-Queue: Oliver Newman <olivernewman@google.com>