fix(bigtable): eliminate stats-handler MD race in internal/metrics tracer (#20158)

## Summary

- Hoists per-attempt cluster/zone/transport/server-latency extraction
out of `HandleRPC`'s `stats.End` branch and into a new `ingestMetadata`
helper that runs on the `InHeader`/`InTrailer` dispatches. `End` now
reads only already-parsed primitives — never the raw `metadata.MD` — so
the cross-goroutine race on the map is gone.
- Serializes the primitive-field writes vs reads with a per-`Tracer`
`sync.Mutex`. Only held around the field access, never across
`metric.Record` calls. `CreateTracer` now returns `*Tracer` so the
embedded mutex isn't copied out of the factory.
- Adds a deterministic `-race` regression test (`tracer_race_test.go`)
that reproduced the failing interleave on the pre-fix code and passes
cleanly on the fix.

## Race being fixed

Reported in the wild by the nightly integration suite; race dump
captured from `TestIntegration_Presidents`:

```
WARNING: DATA RACE
Read  at 0x…5ac8 by goroutine 5793 (csAttempt.finish → HandleRPC[*stats.End])
  bigtable/internal/metrics/tracer.go:867
Previous write at 0x…5ac8 by goroutine 1478 (http2Client.operateHeaders → HandleRPC[*stats.InTrailer])
  bigtable/internal/metrics/tracer.go:859
```

Two follow-on races on the same run against the underlying `metadata.MD`
map (`MD.Copy` in `operateHeaders` vs `MD.Get` in `extractLocation`).
All three came from the same design flaw: `HandleRPC` stored `ev.Header`
/ `ev.Trailer` on the current attempt from `InHeader` / `InTrailer`,
then re-read them under `stats.End`. gRPC's `stats.Handler` contract
does **not** promise `InHeader → InTrailer → End` dispatch on a single
goroutine — under cancel / deadline / GOAWAY, `csAttempt.finish`
(`google.golang.org/grpc@1.82.0/stream.go:1251`) fires `End` from the
caller goroutine while the transport reader (`http2_client.go:1650`) is
still processing the trailer frame.

Also plausibly implicated in the FlakyBot P1 burst against `30b1dfa0db`
on the same nightly (#20147, #20148, #20150, #20151, #20152), all of
which are TestIntegration_* failures with no attached stack — the Sponge
log is Google-internal, but the timing (all P1s stem from the same
invocation, and the metrics refactor #20099 that introduced this pattern
shipped six days earlier) strongly suggests the same underlying race.

## Design (why this exact fix)

Option A of the several considered — the alternatives were:

- **B: add a mutex around `metadata.MD.Get`** inside
`RecordAttemptCompletion`. Fixes the primitive-field race but not the
map-content race (gRPC's transport owns the map and mutates it inside
`operateHeaders`). Rejected.
- **C: plumb server headers/trailers through `stats.End`.** Requires a
gRPC-side change. Not viable short-term.

Option A eliminates cross-goroutine access to the MD entirely — the raw
map is read exactly once on the goroutine that received it, then
dropped. The mutex only guards the small set of already-parsed primitive
fields that `End` needs to read.

`extractLocation` / `extractPeerInfo` / `extractServerLatency` all check
header first then trailer, so passing one MD at a time (with the
`locationExtracted` / `peerInfoExtracted` booleans + the `serverLatency
== 0` gate) preserves the exact header-preferred-trailer-fallback
behaviour.

## Test plan

- [x] `go test -race -count=5 -run
TestHandleRPC_ConcurrentInTrailerAndEnd_NoRace -timeout=120s
./bigtable/internal/metrics/` — passes; same test tripped the race on
the pre-fix code.
- [x] `go test -race -count=1 -timeout=120s
./bigtable/internal/metrics/...` — full package suite passes under race.
- [x] `go vet ./...` clean.
- [x] `go build ./...` clean.
- [ ] Follow-up: rerun the nightly integration suite to close out the
sibling FlakyBot issues if the fix eliminates them.
6 files changed
tree: 582fce9a064397b0d4980051d8e9d41a06dc9e4f
  1. .gemini/
  2. .github/
  3. .librarian/
  4. accessapproval/
  5. accesscontextmanager/
  6. advisorynotifications/
  7. agentidentity/
  8. agentplatform/
  9. agentregistry/
  10. ai/
  11. aiplatform/
  12. alloydb/
  13. analytics/
  14. apigateway/
  15. apigeeconnect/
  16. apigeeregistry/
  17. apihub/
  18. apikeys/
  19. apiregistry/
  20. appengine/
  21. apphub/
  22. appoptimize/
  23. apps/
  24. area120/
  25. artifactregistry/
  26. asset/
  27. assuredworkloads/
  28. auditmanager/
  29. auth/
  30. automl/
  31. backstory/
  32. backupdr/
  33. baremetalsolution/
  34. batch/
  35. beyondcorp/
  36. biglake/
  37. bigquery/
  38. bigtable/
  39. billing/
  40. binaryauthorization/
  41. capacityplanner/
  42. certificatemanager/
  43. ces/
  44. channel/
  45. chat/
  46. chronicle/
  47. civil/
  48. cloudbuild/
  49. cloudcontrolspartner/
  50. clouddms/
  51. cloudprofiler/
  52. cloudquotas/
  53. cloudsecuritycompliance/
  54. cloudtasks/
  55. commerce/
  56. commerceproducer/
  57. compute/
  58. confidentialcomputing/
  59. config/
  60. configdelivery/
  61. contactcenterinsights/
  62. container/
  63. containeranalysis/
  64. datacatalog/
  65. dataflow/
  66. dataform/
  67. datafusion/
  68. datalabeling/
  69. datamanager/
  70. dataplex/
  71. dataproc/
  72. dataqna/
  73. datastore/
  74. datastream/
  75. deploy/
  76. developerconnect/
  77. devicestreaming/
  78. dialogflow/
  79. discoveryengine/
  80. dlp/
  81. documentai/
  82. domains/
  83. edgecontainer/
  84. edgenetwork/
  85. errorreporting/
  86. essentialcontacts/
  87. eventarc/
  88. filestore/
  89. financialservices/
  90. firestore/
  91. functions/
  92. geminidataanalytics/
  93. gkebackup/
  94. gkeconnect/
  95. gkehub/
  96. gkemulticloud/
  97. gkerecommender/
  98. grafeas/
  99. gsuiteaddons/
  100. httpreplay/
  101. hypercomputecluster/
  102. iam/
  103. iap/
  104. identitytoolkit/
  105. ids/
  106. internal/
  107. iot/
  108. kms/
  109. language/
  110. licensemanager/
  111. lifesciences/
  112. locationfinder/
  113. logging/
  114. longrunning/
  115. lustre/
  116. maintenance/
  117. managedidentities/
  118. managedkafka/
  119. maps/
  120. mediatranslation/
  121. memcache/
  122. memorystore/
  123. metastore/
  124. migrationcenter/
  125. modelarmor/
  126. monitoring/
  127. netapp/
  128. networkconnectivity/
  129. networkmanagement/
  130. networksecurity/
  131. networkservices/
  132. notebooks/
  133. optimization/
  134. oracledatabase/
  135. orchestration/
  136. orgpolicy/
  137. osconfig/
  138. oslogin/
  139. parallelstore/
  140. parametermanager/
  141. phishingprotection/
  142. policysimulator/
  143. policytroubleshooter/
  144. privatecatalog/
  145. privilegedaccessmanager/
  146. productregistry/
  147. profiler/
  148. pubsub/
  149. pubsublite/
  150. rapidmigrationassessment/
  151. recaptchaenterprise/
  152. recommendationengine/
  153. recommender/
  154. redis/
  155. resourcemanager/
  156. retail/
  157. rpcreplay/
  158. run/
  159. saasplatform/
  160. scheduler/
  161. secretmanager/
  162. securesourcemanager/
  163. security/
  164. securitycenter/
  165. securitycentermanagement/
  166. securityposture/
  167. servicecontrol/
  168. servicedirectory/
  169. servicehealth/
  170. servicemanagement/
  171. serviceusage/
  172. shell/
  173. shopping/
  174. spanner/
  175. speech/
  176. storage/
  177. storagebatchoperations/
  178. storageinsights/
  179. storagetransfer/
  180. streetview/
  181. support/
  182. talent/
  183. telcoautomation/
  184. texttospeech/
  185. third_party/
  186. tpu/
  187. trace/
  188. translate/
  189. vectorsearch/
  190. vertexai/
  191. video/
  192. videointelligence/
  193. vision/
  194. visionai/
  195. vmmigration/
  196. vmwareengine/
  197. vpcaccess/
  198. webrisk/
  199. websecurityscanner/
  200. workflows/
  201. workloadmanager/
  202. workstations/
  203. .gitignore
  204. .release-please-bulk-manifest.json
  205. .release-please-individual-manifest.json
  206. CHANGES.md
  207. CODE_OF_CONDUCT.md
  208. CONTRIBUTING.md
  209. debug.md
  210. doc.go
  211. go.mod
  212. go.sum
  213. librarian.yaml
  214. LICENSE
  215. migration.md
  216. README.md
  217. release-please-bulk-config.json
  218. release-please-individual-config.json
  219. RELEASING.md
  220. SECURITY.md
  221. testing.md
README.md

Google Cloud Client Libraries for Go

Go Reference

Go packages for Google Cloud Platform services.

Installation

go get cloud.google.com/go/firestore@latest # Replace firestore with the package you want to use.

NOTE: Some of these packages are under development, and may occasionally make backwards-incompatible changes.

Supported APIs

For an updated list of all of our released APIs please see our reference docs.

Go Versions Supported

Our libraries are compatible with the two most recent major Go releases, the same policy the Go programming language follows. This means the currently supported versions are:

  • Go 1.25
  • Go 1.26

Authentication

By default, each client library will use Application Default Credentials (ADC) to automatically configure the credentials used in calling the API endpoint. When using the libraries in a Google Cloud Platform environment such as Compute Engine, Kubernetes Engine, or App Engine, no additional authentication steps are necessary. See Authentication methods at Google and Authenticate for using client libraries for more information.

client, err := storage.NewClient(ctx)

For applications running elsewhere, such as your local development environment, you can use the gcloud auth application-default login command from the Google Cloud CLI to set user credentials in your local filesystem. Application Default Credentials will automatically detect these credentials. See Set up ADC for a local development environment for more information.

Alternately, you may need to provide an explicit path to your credentials. To authenticate using a service account key file, either set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path to your key file, or programmatically pass option.WithCredentialsFile to the NewClient function of the desired package. For example:

client, err := storage.NewClient(ctx, option.WithCredentialsFile("path/to/keyfile.json"))

You can exert even more control over authentication by using the credentials package to create an auth.Credentials. Then pass option.WithAuthCredentials to the NewClient function:

creds, err := credentials.DetectDefault(&credentials.DetectOptions{...})
...
client, err := storage.NewClient(ctx, option.WithAuthCredentials(creds))

Contributing

Contributions are welcome. Please, see the CONTRIBUTING document for details.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Contributor Code of Conduct for more information.

Links