)]}'
{
  "commit": "c38706633aa13da88e8c54b0b6340b9ce7209007",
  "tree": "582fce9a064397b0d4980051d8e9d41a06dc9e4f",
  "parents": [
    "b29b08a3e037f08a5806e0cb7e75b54e55673f0e"
  ],
  "author": {
    "name": "Sushan Bhattarai",
    "email": "sushantsusan@google.com",
    "time": "Fri Jul 17 13:57:08 2026 -0400"
  },
  "committer": {
    "name": "GitHub",
    "email": "noreply@github.com",
    "time": "Fri Jul 17 13:57:08 2026 -0400"
  },
  "message": "fix(bigtable): eliminate stats-handler MD race in internal/metrics tracer (#20158)\n\n## Summary\n\n- Hoists per-attempt cluster/zone/transport/server-latency extraction\nout of `HandleRPC`\u0027s `stats.End` branch and into a new `ingestMetadata`\nhelper that runs on the `InHeader`/`InTrailer` dispatches. `End` now\nreads only already-parsed primitives — never the raw `metadata.MD` — so\nthe cross-goroutine race on the map is gone.\n- Serializes the primitive-field writes vs reads with a per-`Tracer`\n`sync.Mutex`. Only held around the field access, never across\n`metric.Record` calls. `CreateTracer` now returns `*Tracer` so the\nembedded mutex isn\u0027t copied out of the factory.\n- Adds a deterministic `-race` regression test (`tracer_race_test.go`)\nthat reproduced the failing interleave on the pre-fix code and passes\ncleanly on the fix.\n\n## Race being fixed\n\nReported in the wild by the nightly integration suite; race dump\ncaptured from `TestIntegration_Presidents`:\n\n```\nWARNING: DATA RACE\nRead  at 0x…5ac8 by goroutine 5793 (csAttempt.finish → HandleRPC[*stats.End])\n  bigtable/internal/metrics/tracer.go:867\nPrevious write at 0x…5ac8 by goroutine 1478 (http2Client.operateHeaders → HandleRPC[*stats.InTrailer])\n  bigtable/internal/metrics/tracer.go:859\n```\n\nTwo follow-on races on the same run against the underlying `metadata.MD`\nmap (`MD.Copy` in `operateHeaders` vs `MD.Get` in `extractLocation`).\nAll three came from the same design flaw: `HandleRPC` stored `ev.Header`\n/ `ev.Trailer` on the current attempt from `InHeader` / `InTrailer`,\nthen re-read them under `stats.End`. gRPC\u0027s `stats.Handler` contract\ndoes **not** promise `InHeader → InTrailer → End` dispatch on a single\ngoroutine — under cancel / deadline / GOAWAY, `csAttempt.finish`\n(`google.golang.org/grpc@1.82.0/stream.go:1251`) fires `End` from the\ncaller goroutine while the transport reader (`http2_client.go:1650`) is\nstill processing the trailer frame.\n\nAlso plausibly implicated in the FlakyBot P1 burst against `30b1dfa0db`\non the same nightly (#20147, #20148, #20150, #20151, #20152), all of\nwhich are TestIntegration_* failures with no attached stack — the Sponge\nlog is Google-internal, but the timing (all P1s stem from the same\ninvocation, and the metrics refactor #20099 that introduced this pattern\nshipped six days earlier) strongly suggests the same underlying race.\n\n## Design (why this exact fix)\n\nOption A of the several considered — the alternatives were:\n\n- **B: add a mutex around `metadata.MD.Get`** inside\n`RecordAttemptCompletion`. Fixes the primitive-field race but not the\nmap-content race (gRPC\u0027s transport owns the map and mutates it inside\n`operateHeaders`). Rejected.\n- **C: plumb server headers/trailers through `stats.End`.** Requires a\ngRPC-side change. Not viable short-term.\n\nOption A eliminates cross-goroutine access to the MD entirely — the raw\nmap is read exactly once on the goroutine that received it, then\ndropped. The mutex only guards the small set of already-parsed primitive\nfields that `End` needs to read.\n\n`extractLocation` / `extractPeerInfo` / `extractServerLatency` all check\nheader first then trailer, so passing one MD at a time (with the\n`locationExtracted` / `peerInfoExtracted` booleans + the `serverLatency\n\u003d\u003d 0` gate) preserves the exact header-preferred-trailer-fallback\nbehaviour.\n\n## Test plan\n\n- [x] `go test -race -count\u003d5 -run\nTestHandleRPC_ConcurrentInTrailerAndEnd_NoRace -timeout\u003d120s\n./bigtable/internal/metrics/` — passes; same test tripped the race on\nthe pre-fix code.\n- [x] `go test -race -count\u003d1 -timeout\u003d120s\n./bigtable/internal/metrics/...` — full package suite passes under race.\n- [x] `go vet ./...` clean.\n- [x] `go build ./...` clean.\n- [ ] Follow-up: rerun the nightly integration suite to close out the\nsibling FlakyBot issues if the fix eliminates them.",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "35718642d9c267ccab0efd6224d1e8f7f91c7381",
      "old_mode": 33188,
      "old_path": "bigtable/bigtable.go",
      "new_id": "75d7ed32644634b16e3327eb5093ac4007b83bfe",
      "new_mode": 33188,
      "new_path": "bigtable/bigtable.go"
    },
    {
      "type": "modify",
      "old_id": "490610d8cc75649a69d1b454ce5073e403435e00",
      "old_mode": 33188,
      "old_path": "bigtable/client.go",
      "new_id": "c5efac2a3122673c6ef4824d104f9b836cf716a0",
      "new_mode": 33188,
      "new_path": "bigtable/client.go"
    },
    {
      "type": "modify",
      "old_id": "06783ea10931584a84917752a18c43af614b5945",
      "old_mode": 33188,
      "old_path": "bigtable/internal/metrics/factory.go",
      "new_id": "2a76d60f1d7c582d264685b7ced1f4375a1c68bb",
      "new_mode": 33188,
      "new_path": "bigtable/internal/metrics/factory.go"
    },
    {
      "type": "modify",
      "old_id": "6ff458e7b5d0d9b97a661e1573e900d2bd630bc4",
      "old_mode": 33188,
      "old_path": "bigtable/internal/metrics/tracer.go",
      "new_id": "96220471be6f31be0478b7a9304b151e7a7cf183",
      "new_mode": 33188,
      "new_path": "bigtable/internal/metrics/tracer.go"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "89fd9726a2f059e05f45bdfd51eaad9c23bd01d5",
      "new_mode": 33188,
      "new_path": "bigtable/internal/metrics/tracer_race_test.go"
    },
    {
      "type": "modify",
      "old_id": "8667dff61ae8816e21fa3b7834f563be30353334",
      "old_mode": 33188,
      "old_path": "bigtable/metrics_test.go",
      "new_id": "ac2cc7ce6c0ac6b2719d2c943405248e299c4075",
      "new_mode": 33188,
      "new_path": "bigtable/metrics_test.go"
    }
  ]
}