| # Jiri Package Cache |
| |
| ## 1. Overview & Architecture |
| |
| Jiri manages prebuilt packages (such as toolchains, compilers, and SDKs) |
| distributed via Chrome Infrastructure Package Deployer (CIPD). |
| |
| In multi-worktree development environments or when packages update frequently, |
| installing standalone copies of packages directly into each workspace duplicates |
| tens of gigabytes of identical binaries across worktrees and incurs repeated |
| network downloads and disk space exhaustion. |
| |
| To solve this, Jiri provides an **instance-based, content-addressable package |
| cache** shared across the main checkout and all active worktrees. |
| |
| ```text |
| ┌─────────────────────────────────────────────────────────────────────────────┐ |
| │ PackageCache (.jiri_root/packages/) │ |
| │ │ |
| │ instances/ │ |
| │ ├── <A5DE1AnhsOBwACTgzeukZ-l2FbYuA9Q8tQNrHaFQblYC>/ ─── [Hardlink] ──┐ │ |
| │ │ ├── bin/gn │ │ |
| │ │ └── ... ▼ │ |
| │ │ Workspace │ |
| │ ├── <bdupDu8hA7pXmKNwzF8WTbt1rgDro7L5srqIBXBigisC>/ ─── [Hardlink] ─► Destination |
| │ │ └── bin/clang ▲ │ |
| │ │ │ │ |
| │ └── <SPHCOQ38Z7qb62orRA-gTUIZdEVnEdsb_qx5iKCUVZQC>/ ─── [Hardlink] ──┘ │ |
| │ └── ... │ |
| └─────────────────────────────────────────────────────────────────────────────┘ |
| ``` |
| |
| ### Core Concepts & Terminology |
| |
| * **`PackageCache`**: The central cache manager for the repository-level package |
| cache located at `[root]/.jiri_root/packages`. Manages the global cache lock |
| (`.cache.lock`), CIPD ensure synchronization, and cache directory layout. |
| |
| * **`PackageCacheInstance`**: An immutable, content-addressed CIPD package |
| instance stored in the cache at `[root]/.jiri_root/packages/instances/<instance_id>`, |
| keyed directly by its canonical **CIPD Instance ID**. Once downloaded, its |
| contents are pristine and never modified. |
| |
| * **`WorkspaceDestination`**: A destination directory within a specific checkout |
| or worktree where binaries and tools reside (for example, |
| `<workspaceRoot>/prebuilt/third_party/gn/linux-amd64`). Encapsulates the |
| ordered list of package instances to project into that directory. |
| |
| * **`CacheStamp` (`.jiri_cache_hash`)**: An atomic verification stamp file placed |
| at `<destDir>/.jiri_cache_hash` that records structured JSON metadata |
| describing the target path and package instances currently linked into the |
| destination directory. Enables sub-millisecond fast-path verification and |
| safe cache detachment. |
| |
| * **`CIPDEnsureFile`**: In-memory and on-disk representation of the CIPD ensure |
| specification, mapping every required package instance to `@Subdir instances/<instance_id>`. |
| |
| * **`WorktreePackageIndex`**: The aggregated package registry that discovers all |
| active worktrees, computes the unique union of `PackageCacheInstance` objects, |
| and maps `WorkspaceDestination` targets per workspace. |
| |
| * **Workspace Hardlink Forest**: The projection of one or more cached package |
| instances (`PackageCacheInstance`) into a `WorkspaceDestination` via recursive |
| hard links. |
| |
| --- |
| |
| ## 2. The Three-Stage Lifecycle of `jiri fetch-packages` |
| |
| Every package fetch operation in Jiri (`jiri fetch-packages` or `jiri update`) |
| executes through a single, unified three-stage pipeline: |
| |
| ```text |
| ┌─────────────────────────────────────────────────────────────────────────────┐ |
| │ Stage 1: Migrations & Local Adoption │ |
| ├─────────────────────────────────────────────────────────────────────────────┤ |
| │ • Non-blocking lock on .jiri_root/packages/.cache.lock acquired at entry │ |
| │ • In-place local filesystem optimization; eliminates remote network calls │ |
| │ • Adopts legacy package-group entries (.jiri_root/packages/<hex64>) │ |
| │ • Adopts standalone CIPD installations (<root>/.cipd) into package cache │ |
| │ • Immediately removes legacy directories upon adoption │ |
| │ • Detaches workspace in O(1) time when package-cache=false is selected │ |
| │ • Best-effort: failures are never fatal; Stage 2 guarantees convergence │ |
| └──────────────────────────────────────┬──────────────────────────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────────────┐ |
| │ Stage 2: Native CIPD Cache Synchronization & Pruning │ |
| ├─────────────────────────────────────────────────────────────────────────────┤ |
| │ • Held under .cache.lock across ensure generation, CIPD sync, and pruning │ |
| │ • Accumulates all instances across main tree and active worktrees │ |
| │ • Master ensure file executed via `cipd ensure` with $ParanoidMode │ |
| │ • Self-healing: validates, repairs, and downloads missing instances │ |
| │ • Natively deletes unreferenced instances from instances/<instance_id> │ |
| └──────────────────────────────────────┬──────────────────────────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────────────┐ |
| │ Stage 3: Workspace Hardlink Forest Construction │ |
| ├─────────────────────────────────────────────────────────────────────────────┤ |
| │ • Evaluates each WorkspaceDestination declared in checkout manifest │ |
| │ • FAST-PATH: If destination CacheStamp (.jiri_cache_hash) matches, <1ms │ |
| │ • SLOW-PATH: Reconstructs clean hardlink forest from PackageCacheInstance │ |
| │ • Resolves file collisions in manifest order, matching CIPD semantics │ |
| │ • Writes atomic completion stamp (.jiri_cache_hash) as structured JSON │ |
| │ • Standalone checkouts (package-cache=false) remain untouched │ |
| │ • Held under .cache.lock; released upon completion of Stage 3 via defer │ |
| └─────────────────────────────────────────────────────────────────────────────┘ |
| ``` |
| |
| ### Stage 1: Migrations & Local Adoption |
| |
| Implemented in [`project/cache_migration.go`](file:///usr/local/google/home/awolter/src/jiri/project/cache_migration.go). |
| Stage 1 is a dedicated, zero-network migration phase executed under `.cache.lock` before any sync or linking: |
| |
| * **Zero-Download Legacy Group Adoption (`MigrateLegacyGroupCache`)**: If legacy package-group |
| directories (`.jiri_root/packages/<hex64>`) exist, Jiri inspects their internal CIPD |
| state (`.cipd/pkgs/<index>/<instance_id>`). Because CIPD internally keeps |
| individual package instances isolated in these subdirectories, Jiri |
| hardlinks the pristine files directly into `instances/<instance_id>` and |
| replicates the metadata into `.jiri_root/packages/.cipd/pkgs/`. This converts |
| the existing cache locally in **~1–2 seconds** with **0 bytes of network |
| traffic**, immediately removing the legacy directory upon completion. |
| |
| * **Zero-Download Standalone Adoption (`MigrateStandaloneToPackageCache`)**: When a developer |
| enables the package cache on a previously standalone checkout, Jiri inspects `<root>/.cipd/pkgs/`, |
| hardlinks the pristine instances into `instances/<instance_id>`, seeds CIPD metadata into the |
| shared cache, and retires `<root>/.cipd`. Stage 2 then recognizes the packages as already installed, |
| requiring **0 bytes of network download**. |
| |
| * **Zero-Download Cache Opt-Out (`MigratePackageCacheToStandalone`)**: When switching to |
| `package-cache=false`, Jiri detaches each matching workspace directory in $O(1)$ time (<1ms) |
| by removing `.jiri_cache_hash`, and restores any missing directories from the local package cache |
| via hardlinks without downloading. |
| |
| * **Best-Effort Tolerance**: Any unexpected read error during adoption is |
| logged as debug information and safely skipped; Stage 2 guarantees that all |
| declared instances are cleanly resolved and installed. |
| |
| ### Stage 2: Native CIPD Cache Synchronization & Pruning |
| |
| Implemented in [`project/package_cache.go`](file:///usr/local/google/home/awolter/src/jiri/project/package_cache.go). |
| Stage 2 establishes the ground truth of the cache repository: |
| |
| * **Lock Lifecycle & Concurrency**: |
| * **Acquisition**: Before Stage 1 begins, Jiri acquires an exclusive non-blocking |
| file lock on `.jiri_root/packages/.cache.lock`. If another worktree or concurrent |
| Jiri process is currently synchronizing or linking packages, Jiri fails fast |
| immediately with an actionable message prompting the user to wait and retry. |
| * **Scope**: The lock is held continuously across the entire package cache pipeline: |
| Stage 1 (migration and retirement), Stage 2 (master ensure generation, CIPD execution, |
| and pruning), and Stage 3 (workspace hardlink forest construction and stamping). |
| * **Release**: The lock is released via `defer unlock()` at the completion of Stage 3. |
| This provides complete end-to-end atomicity, eliminating any possible races between |
| concurrent package operations or background pruning. |
| |
| * **Multi-Worktree Collection**: Jiri inspects the active manifests of the |
| main checkout and all registered worktrees, accumulating the complete set |
| of required package instances into a single master ensure file |
| (`instance_master.ensure`). Any unpinned tags or floating versions are |
| resolved via `cipd.ResolveEnsureFile`. Broken or inaccessible worktrees are |
| tolerated with a warning. |
| |
| * **CIPD Native Pruning**: Jiri invokes `cipd ensure` targeting |
| `[root]/.jiri_root/packages`. CIPD verifies all declared instances under |
| `instances/<instance_id>`. Crucially, CIPD natively unlinks and deletes any |
| instance directories under `instances/` that are no longer referenced by the |
| master ensure file, completely eliminating the need for bespoke garbage |
| collection. |
| |
| ### Stage 3: Workspace Hardlink Forest Construction |
| |
| Implemented in [`project/workspace_destination.go`](file:///usr/local/google/home/awolter/src/jiri/project/workspace_destination.go). |
| |
| Stage 3 projects the synchronized instance store into each `WorkspaceDestination` |
| inside the active checkout: |
| |
| * **Fast-Path Verification ($O(1)$)**: If `CacheStamp` (`<destDir>/.jiri_cache_hash`) |
| exists and its contents match the expected target hash via `MatchesTarget`, |
| Jiri immediately returns with 0 I/O. File modification times (`mtime`) are |
| preserved, preventing spurious rebuilds in Ninja and Bazel. |
| |
| * **Clean Forest Reconstruction**: If the verification stamp is missing or |
| mismatched (e.g. following a version roll), Jiri removes the destination |
| directory and reconstructs a clean directory structure by recursively |
| hardlinking from the required `PackageCacheInstance` directories |
| (`instances/<instance_id>`). |
| |
| * **Manifest-Order Collision Resolution**: When multiple packages share the |
| same destination path (for example, common SDK headers and platform-specific |
| binaries), instances are linked sequentially in manifest order. If two |
| packages provide the same file path, the later package in manifest order |
| replaces the file link, faithfully reproducing CIPD's native multi-package |
| semantics. |
| |
| * **Atomic Completion Marker**: The stamp file `.jiri_cache_hash` is written |
| as the final step after all links and symlinks are successfully created. |
| |
| --- |
| |
| ## 3. Why Use CIPD Instance IDs as the Canonical Key? |
| |
| Rather than synthesizing an ad-hoc hash (such as `sha256(name:version)`), the |
| package cache keys each instance directory directly by its **CIPD Instance |
| ID** (for example, `GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C`): |
| |
| 1. **Canonical Content Addressability**: |
| The Instance ID is CIPD's SHA-256 digest of the package archive payload. It |
| represents the canonical ground truth of the files contained within the |
| package. |
| |
| 2. **Deduplication Across Tags and References**: |
| Different git tags or floating refs (for example, `git_revision:abc1234` and |
| `latest`) that point to the exact same build artifact resolve to the |
| **identical Instance ID**. Worktrees referencing different tags pointing to |
| the same build share a single instance directory automatically. |
| |
| 3. **Hermetic Lockfile Alignment**: |
| In Fuchsia, all packages are pinned in `jiri.lock`, which records the |
| `instance_id` for each package. The master ensure file is fully pinned and |
| reproducible: |
| |
| ```text |
| @Subdir instances/GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C |
| gn/gn/linux-amd64 GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C |
| ``` |
| |
| --- |
| |
| ## 4. The Role and Necessity of `.jiri_cache_hash` |
| |
| Because hardlinked files are standard, regular files residing in ordinary |
| directories, there is no OS primitive on Linux or macOS to check whether a |
| directory tree of tens of thousands of files matches a cache directory without |
| recursively walking and statting every inode. |
| |
| `.jiri_cache_hash` is managed by the `CacheStamp` abstraction and serves three essential functions: |
| |
| 1. **$O(1)$ Version Roll Detection**: |
| The stamp file records the structured metadata of the instances linked into |
| a `WorkspaceDestination`: |
| ```json |
| { |
| "format_version": 1, |
| "target_path": "prebuilt/third_party/gn/linux-amd64", |
| "instances": [ |
| { |
| "package": "gn/gn/linux-amd64", |
| "version": "git_revision:b4f59045...", |
| "instance_id": "GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C" |
| } |
| ] |
| } |
| ``` |
| `CacheStamp.MatchesTarget` checks if the installed instances match the expected manifest target. |
| If any package rolls to a new version, Jiri detects this in $O(1)$ time, wipes the directory, and |
| reconstructs the forest. For legacy checkouts, `CacheStamp` automatically falls back to parsing |
| plain-text hashes and instance IDs, upgrading them to structured JSON on the next write. |
| |
| 2. **Build Hermeticity (`mtime` Preservation)**: |
| Fuchsia checkouts contain ~150,000 prebuilt files (~30 GB). Touching or |
| re-linking these files on every `jiri update` or `fx build` updates file |
| modification times (`mtime`), invalidating build system action caches in |
| Ninja and Bazel and triggering multi-hour incremental rebuilds. Checking |
| `.jiri_cache_hash` validates the directory in **< 1 millisecond** with |
| **zero file touches**. |
| |
| 3. **Self-Describing & Reversible Audit Stamp**: |
| Because the stamp records the exact CIPD package name, version, and instance ID, |
| developers and tools can inspect `.jiri_cache_hash` directly with `cat` or `jq` |
| to discover what is installed on disk without parsing XML manifests or running CIPD. |
| |
| --- |
| |
| ## 5. Unified Pipeline for `package-cache=false` and `true` |
| |
| Jiri avoids divergent code paths between cache-enabled and standalone checkouts. |
| Both configurations pass through the same 3-stage lifecycle: |
| |
| <!-- mdformat off(reason: pipeline table) --> |
| | Stage | `package-cache=true` | `package-cache=false` | |
| |---|---|---| |
| | **Stage 1: Local Adoption** | Adopts legacy group cache directories into `instances/<id>`. | Detaches `WorkspaceDestination` in-place by removing `CacheStamp` (<1ms). | |
| | **Stage 2: Cache Sync** | Synchronizes `PackageCache` for the main tree and all worktrees via `WorktreePackageIndex`. | Synchronizes `PackageCache` for any active worktrees that require caching. | |
| | **Stage 3: Workspace Projection** | Projects hardlink forest and writes `CacheStamp`. | Reuses cached instances via `LinkPath` if destination directory is missing; otherwise leaves standalone checkout untouched. | |
| <!-- mdformat on --> |
| |
| --- |
| |
| ## 6. State Machine & Invariants (S1–S18) |
| |
| During package synchronization, Jiri evaluates each workspace destination |
| against an exhaustive state matrix: |
| |
| <!-- mdformat off(reason: state matrix table) --> |
| | State | Config | Cache (`instances/<id>`) | Workspace Directory | Scenario | Expected Behavior | Network Download? | |
| |---|---|---|---|---|---|---| |
| | **S1** | Enabled | Missing | Missing | Fresh checkout | Download instance to cache; hardlink to workspace; write stamp | **Yes** (initial fetch) | |
| | **S2** | Enabled | Valid | Valid Hardlink | Steady-state / no-op | Stamp matches: fast-path return nil (<1ms) | **No** (fast path) | |
| | **S3** | Enabled | Valid | Missing | New worktree or deleted workspace | Reconstruct hardlinks from cache; write stamp | **No** (instant link) | |
| | **S4** | Enabled | Valid | Stale Hardlink | Roll to pre-cached version | Replace workspace with hardlinks to target cache; update stamp | **No** (instant link) | |
| | **S5** | Enabled | Valid | Legacy Symlink | Upgrade legacy symlink checkout | Remove symlink; replace with hardlink forest; write stamp | **No** (local conversion) | |
| | **S6** | Enabled | Missing | Stale Hardlink | Version roll with cache enabled | Clean workspace; download new instance to cache; hardlink to workspace | **Yes** (mandatory) | |
| | **S7** | Enabled | Missing | Valid Standalone | Opt-in without roll | Download/verify clean instance in cache; link to workspace; write stamp | **Yes** (clean seed) | |
| | **S8** | Enabled | Missing | Stale Standalone | Opt-in across a roll | Discard stale standalone; download new version clean to cache; hardlink | **Yes** (mandatory) | |
| | **S9** | Enabled | Missing | Dirty Standalone | Dirty / untracked files in workspace | Discard dirty directory; clean download into cache; hardlink to workspace | **Yes** (clean download) | |
| | **S10** | Enabled | Invalid | Any | Corrupted cache (non-dir or file) | Purge corrupt entry; re-download to cache; reconstruct hardlinks | **Yes** | |
| | **S11** | Disabled | N/A | Missing | Fresh checkout, cache disabled | CIPD downloads standalone package directly to workspace; no stamp | **Yes** | |
| | **S12** | Disabled | N/A | Valid Standalone | Steady-state, cache disabled | CIPD verifies standalone package in-place, no-op | **No** (CIPD no-op) | |
| | **S13** | Disabled | N/A | Stale Standalone | Version roll, cache disabled | CIPD updates standalone workspace directory in-place | **Yes** (update download) | |
| | **S14** | Disabled | Valid | Valid Hardlink | Opt-out of cache (`true -> false`) | Retain existing hardlinks in-place; remove stamp file in O(1) time | **No** (instant detach) | |
| | **S15** | Disabled | Valid | Missing | Cache disabled, missing workspace | Hardlink from cache to workspace via `LinkPath`, avoiding network download | **No** (local hardlink) | |
| | **S16** | Disabled | Missing | Stale Hardlink | Opt-out across a version roll | Remove stale hardlinks and stamp; CIPD downloads new standalone version | **Yes** (mandatory) | |
| | **S17** | Disabled | Valid | Missing (Roll) | Roll while restoring from cache | Restore matching version from cache or clean download if uncached | **No** (if cached) | |
| | **S18** | Multi-WT | Valid | Multiple Worktrees | Multi-worktree roll & GC | Instances used by any worktree are preserved by CIPD; obsolete prebuilts cleaned | **Yes** (for new version) | |
| <!-- mdformat on --> |
| |
| --- |
| |
| ## 7. Garbage Collection & Obsolete Prebuilts |
| |
| With the package cache, garbage collection is completely streamlined: |
| |
| 1. **Retirement of Bespoke Cache GC**: |
| Jiri's legacy custom garbage collector (`PackageGC`), which walked all |
| worktrees, matched directory timestamps, and evaluated mtime cutoff |
| heuristics, has been **completely removed**. Cache pruning is handled |
| natively and authoritatively by CIPD during Stage 2 synchronization based |
| on the master ensure file. |
| |
| 2. **Obsolete Prebuilt Directory Cleanup**: |
| When packages roll or are removed from the manifest, obsolete prebuilt |
| directories or dangling symlinks left behind in destination directories are |
| cleaned up by `cleanObsoletePrebuiltsForWorkspace` during `jiri update -gc` |
| and `jiri worktree prune`. |
| |
| --- |
| |
| ## 8. Verification & Test Suite |
| |
| The package cache implementation is verified by unit and integration test |
| suites: |
| |
| * **`//project/package_cache_test.go`**: |
| Consolidated test suite executing all 18 state machine scenarios (S1–S18), |
| verifying inode relationships (`stat.Ino`), fast-path stamp verification, |
| concurrency locking, multi-worktree rolls, and native CIPD pruning. |
| |
| * **`//integrationtests/package_cache_toggle_test.go`**: |
| End-to-end integration test verifying seamless toggling between |
| `package-cache=true` and `package-cache=false`. |
| |
| * **`//cmd/jiri/subcommands/update_test.go`**: |
| Verifies `jiri update -gc` cleans obsolete cached prebuilt directories. |
| |
| --- |
| |
| ## Appendix A: Historical Context & Legacy Group Cache Analysis |
| |
| ### The Legacy "Package Group" Architecture |
| |
| Originally, Jiri grouped packages sharing a destination path: |
| |
| * Jiri computed a composite group hash: |
| `sha256(pkgA.Name + ":" + pkgA.Version + "\n" + pkgB.Name + ":" + pkgB.Version)`. |
| |
| * CIPD downloaded the combined packages into a temporary staging directory |
| (`tmp/merged-*`). |
| |
| * Jiri renamed `tmp/merged-*` into `.jiri_root/packages/<group_hash>`. |
| |
| * Jiri recursively hardlinked `.jiri_root/packages/<group_hash>` to |
| `workspaceDir`. |
| |
| * Jiri implemented a custom `PackageGC` scanner (~450 lines of Go code) that |
| walked all worktrees using `findUsedCachePaths` to prevent active cache |
| directories from being deleted based on 7-day mtime thresholds. |
| |
| ### Flaws of the Legacy Package Group Design |
| |
| 1. **Redundant Downloads on Partial Rolls**: |
| When one package in a multi-package destination rolled (e.g. host tool |
| updated from v1 to v2), the composite group hash changed from |
| `hash(common:v1, tool:v1)` to `hash(common:v1, tool:v2)`. CIPD was forced |
| to re-download or re-extract `common` (e.g. 1.5 GB SDK common headers), |
| even though `common` had not changed at all. |
| |
| 2. **Duplication Across Architectures**: |
| If Linux and macOS builds shared a platform-independent package (e.g. |
| `sdk_common`), Jiri generated two separate composite directories in the |
| cache (`hash(common, linux)` and `hash(common, mac)`), storing duplicate |
| copies of `common`. |
| |
| 3. **Fragile Bespoke Garbage Collection**: |
| Walking multiple worktrees, matching directory stamps, and guessing |
| activity via mtime heuristics was error-prone and frequently led to active |
| worktree dependencies being deleted from the cache or orphaned directories |
| accumulating indefinitely. |
| |
| 4. **Temporary Staging Orchestration**: |
| Managing `tmp/merged-*` staging directories, atomic renames, and failure |
| rollbacks added substantial complexity and failure modes. |
| |
| The instance-based cache replaces the legacy composite group model with an |
| instance-level store managed natively by CIPD, delivering zero network |
| redundancy, native pruning, and sub-millisecond steady-state performance. |
