Jiri manages prebuilt packages (such as toolchains, compilers, and SDKs) distributed via Chrome Infrastructure Package Deployer (CIPD).
By default (package_cache=false), Jiri installs packages as standalone directories directly into the checkout root (for example, prebuilt/third_party/clang/linux-x64). When multiple worktrees exist or when packages update frequently, this model duplicates gigabytes of identical binaries across worktrees and incurs repeated network downloads and disk usage.
To solve this, Jiri provides a unified, content-addressable package cache located under:
[root]/.jiri_root/packages/<group_hash>/
A single destination path in the checkout may consist of one or more grouped CIPD packages. The group hash is a SHA-256 digest computed from the sorted package declarations:
hash = sha256(
pkg1.Name + ":" + pkg1.Version + "\n" +
pkg2.Name + ":" + pkg2.Version + ...)
Because the hash uniquely identifies the exact packages and versions, the directory .jiri_root/packages/<group_hash> is content-addressable and immutable. Once populated cleanly, it can be safely shared across all worktrees in the Jiri root.
Originally, Jiri populated .jiri_root/packages/<hash> and symlinked the workspace destination to the cache:
destDir -> .jiri_root/packages/<hash>
While simple, symlinked prebuilts caused significant problems in developer workflows:
Compilers and build tools (Clang, Rust, Python) resolve ../ relative to the symlink target rather than the symlink's location in the checkout, breaking relative header searches, runtime library resolution, and build hermeticity.
Tools scanning directories or modifying files could unintentionally alter shared cache files or break symlinks.
To solve this, Jiri transitioned to hard links:
The cache directory .jiri_root/packages/<hash> remains the authoritative master copy.
The workspace destination (destDir) is a real directory containing hard links to the corresponding files in the cache.
Internal relative symlinks inside the package are preserved and adjusted.
.jiri_cache_hashBecause a hard-linked directory appears on disk as a normal, regular directory, Jiri needs a mechanism to distinguish between:
A standalone directory downloaded with package_cache=false (independent files).
A cache-managed directory hard-linked to .jiri_root/packages/<hash>.
Jiri writes a sentinel file inside the destination directory:
<destDir>/.jiri_cache_hash
This file contains the <hash> string of the cache entry to which destDir is currently hard-linked.
During jiri update or jiri fetch-packages, Jiri evaluates each package group against a state machine determined by three dimensions:
Config: package-cache enabled (true) or disabled (false).
Cache Store: .jiri_root/packages/<target_hash> (Missing, Valid, or Invalid).
Destination Directory: <destDir> (Missing, Legacy Symlink, Valid Hardlink, Stale Hardlink, Valid Standalone, Stale Standalone, or Dirty).
| State | Config | Cache | Destination | Scenario | Expected Behavior | Network Download? |
|---|---|---|---|---|---|---|
| S1 | Enabled | Missing | Missing | Fresh checkout | Download from CIPD to tempDir, rename to cache, hardlink to dest, write hash | Yes (initial fetch) |
| S2 | Enabled | Valid | Valid Hardlink | Steady-state / no-op | Hash matches: fast-path return nil | No (fast path) |
| S3 | Enabled | Valid | Missing | New worktree or deleted destDir | Create hardlinks from cache to destDir, write hash | No (instant link) |
| S4 | Enabled | Valid | Stale Hardlink | Roll to already-cached version | Replace destDir with hardlinks to target cache, update hash | No (instant link) |
| S5 | Enabled | Valid | Legacy Symlink | Upgrade legacy symlink checkout | Remove symlink, replace with hardlinks to cache, write hash | No (local conversion) |
| S6 | Enabled | Missing | Stale Hardlink | Roll with cache enabled (Clang 23 -> 24) | Remove stale hardlinks; download new version to cache; hardlink to dest | Yes (mandatory) |
| S7 | Enabled | Missing | Valid Standalone | Opt-in to cache without roll | Verify with CIPD ensure (no-op), migrate to cache, hardlink back | No (local migration) |
| S8 | Enabled | Missing | Stale Standalone | Opt-in across a roll | Update dest via CIPD ensure or download clean to cache; do not migrate stale files | Yes (mandatory) |
| S9 | Enabled | Missing | Dirty Standalone | Pre-existing dirty / untracked files | Clean download into cache; replace destDir; do not poison cache | Yes (clean download) |
| S10 | Enabled | Invalid | Any | Corrupted cache (non-dir or file) | Delete corrupt cache; re-download to cache; hardlink to dest | Yes |
| S11 | Disabled | N/A | Missing | Fresh checkout with cache disabled | CIPD downloads standalone package to destDir, no hash file | Yes |
| S12 | Disabled | N/A | Valid Standalone | Steady-state with cache disabled | CIPD confirms no changes, no-op | No (CIPD no-op) |
| S13 | Disabled | N/A | Stale Standalone | Version roll with cache disabled | CIPD updates destDir in-place | Yes (update download) |
| S14 | Disabled | Valid | Valid Hardlink | Opt-out of cache (true -> false) | Keep hardlinks in destDir, remove hash file | No (instant no-op) |
| S15 | Disabled | Valid | Missing | Fresh destDir but package cached | Link cache to destDir (LinkPath), avoiding network download | No (local hardlink) |
| S16 | Disabled | Missing | Stale Hardlink | Opt-out across a roll | Remove stale hardlinks and hash file; CIPD downloads new version | Yes (mandatory) |
A network download from CIPD is mandatory whenever:
The requested package version is not present anywhere locally (neither in .jiri_root/packages/<target_hash> nor as a valid standalone package in destDir).
A package roll occurs (for example, Clang 23 $\to$ Clang 24) and the machine does not yet have Clang 24.
The existing cache entry is corrupted (non-directory, truncated, or invalid).
Network downloads must be bypassed in favor of local filesystem operations in the following situations:
Cache Hit (S2, S3, S4): The target version already exists in .jiri_root/packages/<target_hash>. Jiri creates hard links from the cache to destDir in milliseconds.
Opt-in with Valid Standalone (S7): The user has already downloaded the correct version standalone (package_cache=false) and enables package_cache=true. Jiri verifies the directory with CIPD ensure (which does 0 network transfer) and moves it into the cache.
Opt-out or Cache-Disabled with Cache Hit (S14, S15): The user has package_cache=false, but the target package is already in .jiri_root/packages/<target_hash>. Jiri links or retains the hardlinks in destDir (osutil.LinkPath), avoiding a full disk copy or re-download.
Root Cause: If cacheDir was missing during a version roll, Jiri previously checked if destExists && !destIsSymlink and blindly executed osutil.Rename(destDir, cacheDir). When rolling from Clang 23 to 24, it renamed the old Clang 23 directory into the Clang 24 cache directory and wrote the Clang 24 hash to .jiri_cache_hash. Jiri then believed Clang 24 was installed and never contacted CIPD, permanently locking the user onto the old compiler.
Invariant: Never migrate destDir into cacheDir based solely on directory existence. Jiri must verify that destDir genuinely matches the target package before migrating, or perform a clean download into cacheDir.
Root Cause: If destDir contains hard links to an older cache entry (indicated by .jiri_cache_hash), running CIPD ensure in-place inside destDir can mutate files shared with .jiri_root/packages/<old_hash>, corrupting other worktrees checking out older commits.
Invariant: Only standalone directories (!destHasCacheHash && !destIsSymlink) may be updated in-place. If destHasCacheHash is present and stale, destDir must be cleanly unlinked/removed before downloading or populating the new version.
Root Cause: CIPD ensure does not delete untracked files by default. Migrating an arbitrary directory into .jiri_root/packages/ can permanently contaminate the content-addressable cache with build artifacts or dirty files.
Invariant: The content-addressable cache must only be populated from clean CIPD installations or verified package roots (possessing .versions/ and .cipd/).
Root Cause: restorePackageFromCache previously checked if !destIsSymlink { return nil }. When opting out of package cache, hardlinked directories were ignored, leaving hardlinks attached to the cache and leaving .jiri_cache_hash in destDir.
Invariant: When package-cache=false, destDir retains hardlinks (or is hardlinked from cache via osutil.LinkPath if missing) without copying, and .jiri_cache_hash must be removed.
Root Cause: findUsedCachePaths previously checked only symlinks and ignored hardlinked packages (directories containing .jiri_cache_hash). When packages rolled in the main tree, PackageGC pruned cache entries that were still required by worktrees.
Invariant: findUsedCachePaths must inspect .jiri_cache_hash files in all registered worktrees to preserve cache directories in active use.
Root Cause: Jiri previously did not remove obsolete hardlinks or symlinks from prebuilt/ when packages rolled or were removed from the manifest, leaving orphaned files in the workspace.
Invariant: FetchPackages and PackageGC remove obsolete hardlinked directories and dangling symlinks within prebuilt/ that are no longer declared in the manifest.
The state machine and failure modes are tested in:
//project/package_cache_test.go
The suite contains unit tests for all states, using:
Inode validation (stat.Ino): Verifies that hard links are established and that local migrations preserve inodes without re-downloading.
Canary file propagation: Verifies that local cache-to-destination restorations copy local cache content rather than falling back to network downloads.