gopackagesdriver: fix missing cgo files when RBE is used (#4589)
Bug fix #4588
We notice in our codebase that gopls fails to index some files. The
gopackagesdriver logs show this error:
```
error: unable to load JSON files: unable to resolve imports: open <omitted>/_cgo_gotypes.go: no such file or directory
```
I feed the error to gemini and it proposed this:
<details>
<summary>Gemini report</summary>
## Root Cause
The root cause lies within the `rules_go` Bazel ruleset, specifically in
the `go_pkg_info_aspect` aspect located at
`go/tools/gopackagesdriver/aspect.bzl`.
To understand the issue, it is helpful to understand a few key concepts
in this workflow:
* **`pkg.json`**: A metadata file generated by the `go_pkg_info_aspect`
aspect for each Go target. It contains structured information (like
dependencies, source files, and import paths) that `gopackagesdriver`
uses to inform `gopls` about how the package is constructed.
* **`compiled_go_files`**: A list within the aspect's provider that
tracks all the Go source files destined for compilation. In Bazel,
exposing these files in an output group (like `go_pkg_driver_srcs`)
tells the build system that it must actually execute the actions to
generate these files before the build command finishes.
* **`cgo_out_dir`**: The output directory containing intermediate Go
files generated by the `cgo` tool (such as `_cgo_gotypes.go` and
`_cgo_export.go`). These files are generated from C/C++ sources or Go
files containing `import "C"`.
When the aspect processes a `go_test` target, it traverses the
dependencies to find the archive containing the test sources. While it
correctly generates the `pkg.json` metadata file pointing to the
generated `cgo` output files within the `cgo_out_dir`, it forgot to add
the `cgo_out_dir` to the list of `compiled_go_files`.
Because `cgo_out_dir` was omitted from `compiled_go_files` (which maps
to the `go_pkg_driver_srcs` output group), Bazel did not guarantee that
the `cgo` outputs were actually built and present on disk before
completing the aspect execution. When `gopls` then parsed the `pkg.json`
and attempted to read the generated source files, they were missing,
resulting in the error.
## Resolution
The issue was resolved by patching `rules_go` to ensure that
`cgo_out_dir` is correctly added to `compiled_go_files` when processing
`go_test` dependencies.
```diff
diff --git a/go/tools/gopackagesdriver/aspect.bzl b/go/tools/gopackagesdriver/aspect.bzl
index dbe5663..8dc7d3e 100644
--- a/go/tools/gopackagesdriver/aspect.bzl
+++ b/go/tools/gopackagesdriver/aspect.bzl
@@ -108,6 +108,8 @@ def _go_pkg_info_aspect_impl(target, ctx):
if archive.data.label == dep_archive.data.label:
pkg_json_files.append(make_pkg_json_with_archive(ctx, dep_archive.data.name, dep_archive))
compiled_go_files.extend(dep_archive.source.srcs)
+ if dep_archive.data.cgo_out_dir:
+ compiled_go_files.append(dep_archive.data.cgo_out_dir)
export_files.append(dep_archive.data.export_file)
break
```
</details>
I don't fully understand the root cause but the proposed fix is working
for us. Does the fix make sense?
Signed-off-by: Teo Koon Peng <koonpeng@google.com>
Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>diff --git a/go/tools/gopackagesdriver/aspect.bzl b/go/tools/gopackagesdriver/aspect.bzl
index dbe5663..8dc7d3e 100644
--- a/go/tools/gopackagesdriver/aspect.bzl
+++ b/go/tools/gopackagesdriver/aspect.bzl
@@ -108,6 +108,8 @@
if archive.data.label == dep_archive.data.label:
pkg_json_files.append(make_pkg_json_with_archive(ctx, dep_archive.data.name, dep_archive))
compiled_go_files.extend(dep_archive.source.srcs)
+ if dep_archive.data.cgo_out_dir:
+ compiled_go_files.append(dep_archive.data.cgo_out_dir)
export_files.append(dep_archive.data.export_file)
break