tree: 4ff2e1f200f9d68c2b1fb8396326638acc9ce4de [path history] [tgz]
  1. proto/
  2. action_diff.py
  3. action_diff_test.py
  4. bb_fetch_rbe_cas.py
  5. bb_fetch_rbe_cas_test.py
  6. bbtool.py
  7. bbtool_test.py
  8. BUILD.gn
  9. build_summary.py
  10. build_summary_test.py
  11. cl_utils.py
  12. cl_utils_test.py
  13. common-setup.sh
  14. cxx.py
  15. cxx_link_remote_wrapper.py
  16. cxx_link_remote_wrapper_test.py
  17. cxx_remote_wrapper.py
  18. cxx_remote_wrapper_test.py
  19. cxx_test.py
  20. depfile.py
  21. depfile_test.py
  22. detail-diff.sh
  23. dlwrap.py
  24. dlwrap_test.py
  25. fuchsia-reproxy-wrap.sh
  26. fuchsia-reproxy.cfg
  27. fuchsia-rewrapper.cfg
  28. fuchsia.py
  29. fuchsia_test.py
  30. linker.py
  31. linker_test.py
  32. local-only.sh
  33. log-it.sh
  34. output_leak_scanner.py
  35. output_leak_scanner_test.py
  36. pb_message_util.py
  37. pb_message_util_test.py
  38. prebuilt_tool_remote_wrapper.py
  39. prebuilt_tool_remote_wrapper_test.py
  40. python_with_pb_test.sh
  41. README.md
  42. reclient_cxx.sh
  43. relativize_args.py
  44. relativize_args_test.py
  45. remote_action.py
  46. remote_action_test.py
  47. remotetool.py
  48. remotetool.sh
  49. remotetool_test.py
  50. reproxy-compress.cfg
  51. reproxy-no-compress.cfg
  52. reproxy_logs.py
  53. reproxy_logs_test.py
  54. rpl_tool.py
  55. rpl_tool_test.py
  56. rustc.py
  57. rustc_remote_wrapper.py
  58. rustc_remote_wrapper_test.py
  59. rustc_test.py
  60. tablefmt.py
  61. tablefmt_test.py
  62. test_all.sh
  63. textpb.py
  64. textpb_test.py
  65. upload_reproxy_logs.py
  66. upload_reproxy_logs_test.py
build/rbe/README.md

This directory contains support for building the Fuchsia tree with build actions running on RBE (remote build execution).

Remote execution wrappers

The top-level remote execution wrappers are used as command prefixes:

  • reclient_cxx.sh: prefix wrapper for remote compiling C++
    • cxx_remote_wrapper.py: is a similar heavier wrapper with additional features and workarounds.
  • rustc_remote_wrapper.py: prefix wrapper for remote compiling and linking Rust
    • Detects and gathers all inputs and tools needed for remote compiling.
    • Detects extra outputs to download produced by remote compiling.
  • cxx_link_remote_wrapper.py: prefix wrapper for remote ilnking C++
  • prebuilt_tool_remote_wrapper.py: prefix wrapper for simple tools
  • remote_action.py: prefix wrapper for generic remote actions
    • Exists as standalone wrapper and library.
    • Includes detailed diagnostics for certain error conditions.
    • Includes limited fault-tolerance and retries.
    • The C++ and Rust wrappers inherit all of the features of this generic script.
  • fuchsia-reproxy-wrap.sh: automatically start/shutdown reproxy (needed by rewrapper) around any command. Used by fx build.
  • dlwrap.py: downloads artifacts from RBE's CAS using download stub files

More details can be found by running with --help.

Support scripts

  • cl_utils.py: generic command-line operation library
  • cxx.py: for understanding structure of C/C++ compile commands
  • fuchsia.py: Fuchsia-tree specific directory layouts and conventions. Parties interested in re-using wrapper scripts found here should expect to replace this file.
  • output_leak_scanner.py: Check that commands and outputs do not leak the name of the build output directory, for better caching outcomes.
  • relativize_args.py: Attempt to transform commands with absolute paths into equivalent commands with relative paths. This can be useful for build systems that insist on absolute paths, like cmake.
  • rustc.py: for understanding structure of rustc compile commands
  • tablefmt.py: utilities for printing readable ASCII tables
  • textpb.py: generic protobuf text parsing library

Configurations

  • fuchsia-rewrapper.cfg: rewrapper configuration
  • fuchsia-reproxy.cfg: reproxy configuration

Troubleshooting tools

  • action_diff.sh: recursively queries through two reproxy logs to root-cause differences between output and intermediate artifacts. This is useful for examining unexpected digest differences and cache misses.
  • bbtool.py: buildbucket related tools
    • fetch_reproxy_log: retrieve .rrpl log, possibly from subbuild.
  • bb_fetch_rbe_cas.sh: retrieve a remote-built artifact from the RBE CAS using a buildbucket id and the path under the build output directory.
  • detail-diff.sh: attempts to compare human-readable representations of files, including some binary files, by using tools like objdump.
  • remotetool.sh: can lookup actions and artifacts in the RBE CAS. From: https://github.com/bazelbuild/remote-apis-sdks
  • reproxy_logs.sh: subcommand utility
    • diff: report meaningful difference between two reproxy logs
    • output_file_digest: lookup the digest of a remote built artifact
    • bandwidth: report total download and upload bytes from reproxy log
    • plot_download: plot download demand over the course of a build
  • rpl_tool.sh: tools that involve reproxy logs and remotetool together
    • expand_to_rpl: populate .rrpl with command and inputs -> .rpl

All tools have more detailed usage with --help.

GN files

  • build/toolchain/rbe.gni: global args.gn variables for RBE

  • build/toolchain/clang_toolchain.gni and build/toolchain/zircon/zircon_toolchain.gni: use RBE wrappers depending on configuration

  • build/rust/rustc_*.gni: uses RBE wrappers depending on configuration

Metrics and logs

  • build_summary.py: If your environment sets FX_BUILD_RBE_STATS=1, this script will be run after each build and display a summary of cache and download metrics.
  • upload_reproxy_logs.py: pushes RBE metrics and detailed logs to BigQuery.
    • This can be enabled by fx build-metrics.
  • pb_message_util.py: library for translating protobufs to JSON