| commit | e01c5481ac7eae91e17b9f3ed9fac0d60cbfc721 | [log] [tgz] |
|---|---|---|
| author | David Fang <fangism@google.com> | Tue Oct 14 09:01:15 2025 -0700 |
| committer | CQ Bot <fuchsia-internal-scoped@luci-project-accounts.iam.gserviceaccount.com> | Tue Oct 14 09:01:15 2025 -0700 |
| tree | 8a003a1ba96d61f441dd0c3ba7a0694615864696 | |
| parent | 77161a38db980acd35ef457fc7bc3b5c038f4f57 [diff] |
[test] Test utils for verifying interrupt handling New sh_wrappers: 'send_sigint' and 'expect_interrupt' work together by sending an interrupt to a wrapped command, and verifying that the interrupt status (exit code 130) propagates where expected. Some inner command will signal when is waiting for interrupt via a wait-file, at which point send_sigint will deliver the interrupt. 'sleepy_cat.sh' is an example that signals with a wait-file. In the follow-up change, fakebuild will use this mechanism so that we can verify end-to-end signal propagation through layers of wrapper scripts. Bug: 390427892 Change-Id: I53dcc162754f8d8dbad7c602d346bb00ed022ea8 Reviewed-on: https://fuchsia-review.googlesource.com/c/rsclient/+/1391953 Commit-Queue: David Fang <fangism@google.com> Reviewed-by: Jay Zhuang <jayzhuang@google.com>
The rsclient repository contains tools that support the use of ResultStore services and API.
rsproxy acts as a bridge between build tools and the ResultStore service.
The first supported build tool is Fuchsia's fork of ninja.
rsclient builds with Bazel using rules_go.
rsclient is supported for Linux, and is yet untested on other platforms.
Before working with the code base, we recommend installing the precommit hook once per git clone with:
./scripts/install_precommit.sh
This runs a variety of checks and automatic regeneration steps for you before committing local changes.
Download prerequisite tools (like bazel, clang, ninja) by running:
./scripts/infra/bootstrap_prebuilts.sh
This will install tools and bazel build files under prebuilt/. This will also create a convenient ./bazel symlink.
A typical build-and-test development cycle is conveniently captured in the following script:
./scripts/infra/presubmit.sh
To build and test everything, use typical bazel commands:
$ ./bazel build ... $ ./bazel test ...
To create a package at bazel-bin/rsclient.tar:
$ ./bazel build //:artifacts_tar
rsproxyrsproxy acts as bridge between a local build tool and a ResultStore service, primarily the uploading subset of its API. It connects to a ResultStore service using its API and gRPC calls. It receives ResultStore UploadRequests through a named fifo, and performs some minor transformations such batching before uploading them to the service.
rsproxy can also upload files to a CAS service, separate from the ResultStore service. This is useful for relaying logs and useful artifacts from the build tool.
rsproxy configuration is managed through command-line flags, environment variables, and an optional configuration file. The precedence of these sources is as follows, from highest to lowest:
rsproxy will check for an environment variable.Flags can be set using environment variables with the RS_ prefix. For example, the --service flag can be set with RS_service=.... As a fallback, the FLAG_ prefix is also supported.
A configuration file can be specified using the --cfg flag. The file should contain one argument per line. Lines starting with # are treated as comments and are ignored. For flags that require a value, the flag name and value can be separated by whitespace or an equals sign (=). For boolean flags, simply including the flag name is sufficient to set it to true. Flag names should be specified without leading dashes (--); any leading dashes will be automatically removed.
Example configuration file (rsproxy.cfg):
# Set the ResultStore service endpoint. rs_service=resultstore.googleapis.com:443 # Enable verbose logging. verbose
This configuration can be used by running rsproxy --cfg rsproxy.cfg.
The following flags are necessary for connecting rsproxy to the necessary remote services:
rs_service: The address of the ResultStore service (e.g., resultstore.googleapis.com:443).rs_instance: The ResultStore instance name (e.g., projects/your-gcp-project/instances/default).cas_service: The address of the Content Addressable Storage (CAS) service (e.g., remotebuildexecution.googleapis.com:443).cas_instance: The CAS instance name (e.g., projects/your-gcp-project/instances/default).These values can be set in a configuration file, as environment variables, or as command-line flags.
rsproxy handles authentication to the ResultStore and CAS services in one of the following ways:
Application Default Credentials (ADC): By default, rsproxy uses ADC. This can be explicitly enabled by setting use_application_default_credentials=true. This option may frequently prompting the user for re-authentication after tokens expire.
External Credential Helper: For automatic credential exchanges between longer-lived credentials such as LOAS2 and OAuth2 tokens, an external tool can be used. This requires less frequent user intervention. To enable this, set use_application_default_credentials=false. The following flags are also required:
credentials_helper: The path to the credential helper executable. The reclient project provides a credshelper executable.credentials_helper_args: A string of arguments to pass to the credential helper.use_external_auth_token: Must be set to true to use the external helper.For general troubleshooting for authentication issues, consult reclient's documentation.
rsproxy-wrap.shrsproxy-wrap.sh is a convenient wrapper script that starts and stops rsproxy around an arbitrary command (typically a build tool) and returns that command's exit status. The wrapper starts rsproxy in the background, and sets up a named pipe (FIFO) for the wrapped command to send data to rsproxy.
rsproxy-wrap.sh is included as part of the built package.
/path/to/rsproxy-wrap.sh [wrapper_options...] [--rsproxy_options rsproxy_options...] -- command_to_wrap...
--rsproxy <path>: (Required) The path to the rsproxy binary.--log-dir <path>: A directory to store logs and other artifacts from the rsproxy process and the wrapper script itself (e.g., the communication pipe and ready file). It is recommended to use a unique directory for each concurrent invocation. If not provided, a temporary directory will be created.--async-shutdown: If set, the wrapper script will not wait for rsproxy to shut down before exiting.--print-logs-on-failure: If set, the wrapper will dump the rsproxy logs to stderr if the wrapped command fails. This is useful for debugging in test environments.rsproxy options can be passed after --rsproxy_options, and before the -- that precedes the wrapped command.
The wrapper script creates a named pipe (FIFO) and exports its path to the wrapped command via the RSPROXY_FIFO environment variable. The wrapped command is expected to write ResultStore UploadRequests to this pipe.
The first UploadRequest coming from the build tool must be CreateInvocation request, after which any number of generic UploadRequests may follow. Sending FinalizeInvocation will mark the end of the stream of requests and trigger a shutdown of rsproxy. If the pipe is closed before FinalizeInvocation is seen, it will be assumed that the build was interrupted and rsproxy will update the invocation as such before sending its own FinalizeInvocation.
The rs-ninja.sh script is the build tool's counterpart to rsproxy-wrap.sh. It acts as a wrapper around the ninja command, automatically providing the necessary flags for ResultStore integration.
When rs-ninja.sh is run within the environment set up by rsproxy-wrap.sh, it does the following:
RSPROXY_FIFO environment variable (set by rsproxy-wrap.sh).--resultstore_output=<pipe_path> flag into the ninja command.ninja via the NINJA_BUILD_ID environment variable. If NINJA_BUILD_ID is already set in the environment, its value is used instead. Otherwise, a new ID is generated using the uuidgen tool, which must be available in your PATH.rs-ninja.sh expects the first argument to be the ninja executable.
Here is an example of how to use them together:
/path/to/rsproxy-wrap.sh \ --rsproxy /path/to/rsproxy \ --rsproxy_options \ --cfg rsproxy.cfg \ -- \ /path/to/rs-ninja.sh \ /path/to/ninja -C build_output_dir ...
If any other wrapper scripts are involved between rsproxy-wrap.sh and rs-ninja.sh, they need to preserve the RSPROXY_FIFO environment variable and preserve exit statuses of wrapped commands.
This section provides information for developers working on the rsclient codebase.
The core logic is organized into several internal packages:
auth: Handles authentication and credentials for connecting to gRPC services.fakecas: Provides a fake Content Addressable Storage (CAS) server for testing.fakeresultstore: Provides a fake ResultStore server for testing.rsproxy: Contains the core logic for the rsproxy server and client.testenv: Provides a flexible test environment for rsproxy tests.version: Manages the version information for the rsclient tools.Not yet applicable.
Not yet applicable.