build/ninja_implicit_inputs/README.md

# Overview

See https://fxbug.dev/37723350 for full context.

A Bazel action that invokes Ninja in a specific sub-build, needs to
know the full list of source inputs that the build artifacts depend on,
otherwise incremental correctness cannot be guaranteed.

Bazel by design doesn't support depfiles, and must know about *all*
possible inputs, which themselves must be listed through actual `BUILD.bazel`
target definition arguments. Over-specifying is ok (but may force un-necessary
Ninja invocations which will do nothing).

Hence, GN targets that are to be built with Bazel must have *no* inputs that
Bazel doesn't know about. The files in this directory help achieve this goal.

# C++ headers

The Ninja build plan generated by GN only includes C++ source files as explicit
inputs for compiler actions. Thus a declaration like:

```
source_set("foo") {
  sources = [
    "bar.cc",
    "bar.h",
    "foo.cc",
    "foo.h",
  ]
}
```

Will generate a Ninja build rule that completely ignores the header, as in:

```
build obj/bar.cc.o: cc_compile ../../bar.cc
build obj/foo.cc.o: cc_compile ../../foo.cc
```

This is intentional to reduce incremental rebuilds. Instead, the depfile
generated by each Ninja action will specific the headers as implicit inputs,
for example:

```
# From obj/bar.cc.o.d
obj/bar.cc.o: ../../bar.h

# From obj/foo.cc.o.d
obj/foo.cc.o: ../../foo.h
```

This is intentional, because it reduces incremental rebuilds, for example
if `foo.h` is modified, only `obj/foo.cc.o` will be recompiled because the
depfile knows exactly which headers are included by which source file(s).

Unfortunately, this also means that the headers are never returned when
invoking a command like `ninja -t inputs obj/foo.cc.o` which is supposed
to return the list of all inputs for that object file, and which works
before building anything with Ninja.

To solve this, all GN C++ target definitions have GN metadata listing the
header they declare in their `public` and `sources` arguments. This information
is then later collected by the `//build/ninja_implicit_inputs:manifest` target,
producing a file that can be used to augment the `-t inputs` output with the
missing information.

See the `_common_cxx_target_wrapper` GN template definition in
`//build/config/BUILDCONFIG.gn`.


# Prebuilt toolchain files

Some toolchain files are invoked by Ninja build commands, and never listed
as inputs at all, such as the compiler or linker binary itself, but also
headers and link time files that are used implicitly by such tools.

To include them in the Bazel inputs, their directories are listed in GN
metadata of dedicated targets under `//build/ninja_implicit_inputs/BUILD.gn`


# Implicit inputs manifest

The `//build/ninja_implicit_inputs:manifest` target generates, at `gn gen`
time, a JSON manifest file describing extra implicit inputs related to a
set of root GN targets, using GN metadata collection.

See the content of `//build/ninja_implicit_inputs/BUILD.gn` for details
about its schema.

By default, all transitive dependencies of `//:default` are visited by
the collection, but this can be overridden by setting
`ninja_implicit_inputs_root_labels` in ` args.gn` to a different label
list. For example when setting up a GN build graph with a reduced
number of targets using `--root-pattern`.

The manifest will be used later by a Bazel repository rule to generate
`BUILD.bazel` file definitions covering the source inputs of the Ninja
artifacts built from Bazel. It is also used by the verification tool
(described below), that can be invoked after a regular `fx build`, to
list all implicit inputs that are not covered by the manifest yet.


# Verification tool

The `//build/ninja_implicit_inputs/check_tool.py` script can be used to
check, after a build, whether a given set of GN target labels and their
transitive dependencies still have implicit inputs that are not covered
by the `//build/ninja_implicit_inputs:manifest`.

Usage example:

```
python3 build/ninja_implicit_inputs/check_tool.py --build //zircon/public/sysroot_sdk
```

After a possibly long time (performing a number of large Ninja queries),
the tool will print a description of all GN target labels that have still
implicit inputs, followed by practical instructions on how to fix the
most common issues (e.g. missing header declarations in GN C++ target
definitions).

**IMPORTANT**: The tool must always be called after building the target of
interest, as it depends on the deps log to give accurate results.

The output would look like:

```
ERROR: The following 236 GN targets use implicit source inputs:

//sdk/lib/c/ctype:ctype(//sdk/lib/c:user.libc_x64)
    sdk/lib/c/include-preempt/src/__support/macros/attributes.h
    third_party/llvm-libc/src/hdr/float_macros.h
    third_party/llvm-libc/src/hdr/limits_macros.h
    ...

...

//build/tools/json_validator:json_validator(//build/toolchain:host_x64)
    src/lib/fxl/build_config.h
    src/lib/fxl/command_line.h
    src/lib/fxl/fxl_export.h
    src/lib/fxl/macros.h
    third_party/rapidjson/include/rapidjson/allocators.h
    third_party/rapidjson/include/rapidjson/document.h
    ...

...
```
Since the report is very long, it is always saved to `/tmp/implicit_inputs.txt`
by default, but the destination can be changed with the `--output=FILE` argument.