Jiri manages prebuilt packages (such as toolchains, compilers, and SDKs) distributed via Chrome Infrastructure Package Deployer (CIPD).
In multi-worktree development environments or when packages update frequently, installing standalone copies of packages directly into each workspace duplicates tens of gigabytes of identical binaries across worktrees and incurs repeated network downloads and disk space exhaustion.
To solve this, Jiri provides an instance-based, content-addressable package cache shared across the main checkout and all active worktrees.
┌─────────────────────────────────────────────────────────────────────────────┐ │ PackageCache (.jiri_root/packages/) │ │ │ │ instances/ │ │ ├── <A5DE1AnhsOBwACTgzeukZ-l2FbYuA9Q8tQNrHaFQblYC>/ ─── [Hardlink] ──┐ │ │ │ ├── bin/gn │ │ │ │ └── ... ▼ │ │ │ Workspace │ │ ├── <bdupDu8hA7pXmKNwzF8WTbt1rgDro7L5srqIBXBigisC>/ ─── [Hardlink] ─► Destination │ │ └── bin/clang ▲ │ │ │ │ │ │ └── <SPHCOQ38Z7qb62orRA-gTUIZdEVnEdsb_qx5iKCUVZQC>/ ─── [Hardlink] ──┘ │ │ └── ... │ └─────────────────────────────────────────────────────────────────────────────┘
PackageCache: The central cache manager for the repository-level package cache located at [root]/.jiri_root/packages. Manages the global cache lock (.cache.lock), CIPD ensure synchronization, and cache directory layout.
PackageCacheInstance: An immutable, content-addressed CIPD package instance stored in the cache at [root]/.jiri_root/packages/instances/<instance_id>, keyed directly by its canonical CIPD Instance ID. Once downloaded, its contents are pristine and never modified.
WorkspaceDestination: A destination directory within a specific checkout or worktree where binaries and tools reside (for example, <workspaceRoot>/prebuilt/third_party/gn/linux-amd64). Encapsulates the ordered list of package instances to project into that directory.
CacheStamp (.jiri_cache_hash): An atomic verification stamp file placed at <destDir>/.jiri_cache_hash that records structured JSON metadata describing the target path and package instances currently linked into the destination directory. Enables sub-millisecond fast-path verification and safe cache detachment.
CIPDEnsureFile: In-memory and on-disk representation of the CIPD ensure specification, mapping every required package instance to @Subdir instances/<instance_id>.
WorktreePackageIndex: The aggregated package registry that discovers all active worktrees, computes the unique union of PackageCacheInstance objects, and maps WorkspaceDestination targets per workspace.
Workspace Hardlink Forest: The projection of one or more cached package instances (PackageCacheInstance) into a WorkspaceDestination via recursive hard links.
jiri fetch-packagesEvery package fetch operation in Jiri (jiri fetch-packages or jiri update) executes through a single, unified three-stage pipeline:
┌─────────────────────────────────────────────────────────────────────────────┐ │ Stage 1: Migrations & Local Adoption │ ├─────────────────────────────────────────────────────────────────────────────┤ │ • Non-blocking lock on .jiri_root/packages/.cache.lock acquired at entry │ │ • In-place local filesystem optimization; eliminates remote network calls │ │ • Adopts legacy package-group entries (.jiri_root/packages/<hex64>) │ │ • Adopts standalone CIPD installations (<root>/.cipd) into package cache │ │ • Immediately removes legacy directories upon adoption │ │ • Detaches workspace in O(1) time when package-cache=false is selected │ │ • Best-effort: failures are never fatal; Stage 2 guarantees convergence │ └──────────────────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ Stage 2: Native CIPD Cache Synchronization & Pruning │ ├─────────────────────────────────────────────────────────────────────────────┤ │ • Held under .cache.lock across ensure generation, CIPD sync, and pruning │ │ • Accumulates all instances across main tree and active worktrees │ │ • Master ensure file executed via `cipd ensure` with $ParanoidMode │ │ • Self-healing: validates, repairs, and downloads missing instances │ │ • Natively deletes unreferenced instances from instances/<instance_id> │ └──────────────────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ Stage 3: Workspace Hardlink Forest Construction │ ├─────────────────────────────────────────────────────────────────────────────┤ │ • Evaluates each WorkspaceDestination declared in checkout manifest │ │ • FAST-PATH: If destination CacheStamp (.jiri_cache_hash) matches, <1ms │ │ • SLOW-PATH: Reconstructs clean hardlink forest from PackageCacheInstance │ │ • Resolves file collisions in manifest order, matching CIPD semantics │ │ • Writes atomic completion stamp (.jiri_cache_hash) as structured JSON │ │ • Standalone checkouts (package-cache=false) remain untouched │ │ • Held under .cache.lock; released upon completion of Stage 3 via defer │ └─────────────────────────────────────────────────────────────────────────────┘
Implemented in project/cache_migration.go. Stage 1 is a dedicated, zero-network migration phase executed under .cache.lock before any sync or linking:
Zero-Download Legacy Group Adoption (MigrateLegacyGroupCache): If legacy package-group directories (.jiri_root/packages/<hex64>) exist, Jiri inspects their internal CIPD state (.cipd/pkgs/<index>/<instance_id>). Because CIPD internally keeps individual package instances isolated in these subdirectories, Jiri hardlinks the pristine files directly into instances/<instance_id> and replicates the metadata into .jiri_root/packages/.cipd/pkgs/. This converts the existing cache locally in ~1–2 seconds with 0 bytes of network traffic, immediately removing the legacy directory upon completion.
Zero-Download Standalone Adoption (MigrateStandaloneToPackageCache): When a developer enables the package cache on a previously standalone checkout, Jiri inspects <root>/.cipd/pkgs/, hardlinks the pristine instances into instances/<instance_id>, seeds CIPD metadata into the shared cache, and retires <root>/.cipd. Stage 2 then recognizes the packages as already installed, requiring 0 bytes of network download.
Zero-Download Cache Opt-Out (MigratePackageCacheToStandalone): When switching to package-cache=false, Jiri detaches each matching workspace directory in $O(1)$ time (<1ms) by removing .jiri_cache_hash, and restores any missing directories from the local package cache via hardlinks without downloading.
Best-Effort Tolerance: Any unexpected read error during adoption is logged as debug information and safely skipped; Stage 2 guarantees that all declared instances are cleanly resolved and installed.
Implemented in project/package_cache.go. Stage 2 establishes the ground truth of the cache repository:
Lock Lifecycle & Concurrency:
.jiri_root/packages/.cache.lock. If another worktree or concurrent Jiri process is currently synchronizing or linking packages, Jiri fails fast immediately with an actionable message prompting the user to wait and retry.defer unlock() at the completion of Stage 3. This provides complete end-to-end atomicity, eliminating any possible races between concurrent package operations or background pruning.Multi-Worktree Collection: Jiri inspects the active manifests of the main checkout and all registered worktrees, accumulating the complete set of required package instances into a single master ensure file (instance_master.ensure). Any unpinned tags or floating versions are resolved via cipd.ResolveEnsureFile. Broken or inaccessible worktrees are tolerated with a warning.
CIPD Native Pruning: Jiri invokes cipd ensure targeting [root]/.jiri_root/packages. CIPD verifies all declared instances under instances/<instance_id>. Crucially, CIPD natively unlinks and deletes any instance directories under instances/ that are no longer referenced by the master ensure file, completely eliminating the need for bespoke garbage collection.
Implemented in project/workspace_destination.go.
Stage 3 projects the synchronized instance store into each WorkspaceDestination inside the active checkout:
Fast-Path Verification ($O(1)$): If CacheStamp (<destDir>/.jiri_cache_hash) exists and its contents match the expected target hash via MatchesTarget, Jiri immediately returns with 0 I/O. File modification times (mtime) are preserved, preventing spurious rebuilds in Ninja and Bazel.
Clean Forest Reconstruction: If the verification stamp is missing or mismatched (e.g. following a version roll), Jiri removes the destination directory and reconstructs a clean directory structure by recursively hardlinking from the required PackageCacheInstance directories (instances/<instance_id>).
Manifest-Order Collision Resolution: When multiple packages share the same destination path (for example, common SDK headers and platform-specific binaries), instances are linked sequentially in manifest order. If two packages provide the same file path, the later package in manifest order replaces the file link, faithfully reproducing CIPD's native multi-package semantics.
Atomic Completion Marker: The stamp file .jiri_cache_hash is written as the final step after all links and symlinks are successfully created.
Rather than synthesizing an ad-hoc hash (such as sha256(name:version)), the package cache keys each instance directory directly by its CIPD Instance ID (for example, GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C):
Canonical Content Addressability: The Instance ID is CIPD's SHA-256 digest of the package archive payload. It represents the canonical ground truth of the files contained within the package.
Deduplication Across Tags and References: Different git tags or floating refs (for example, git_revision:abc1234 and latest) that point to the exact same build artifact resolve to the identical Instance ID. Worktrees referencing different tags pointing to the same build share a single instance directory automatically.
Hermetic Lockfile Alignment: In Fuchsia, all packages are pinned in jiri.lock, which records the instance_id for each package. The master ensure file is fully pinned and reproducible:
@Subdir instances/GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C gn/gn/linux-amd64 GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C
.jiri_cache_hashBecause hardlinked files are standard, regular files residing in ordinary directories, there is no OS primitive on Linux or macOS to check whether a directory tree of tens of thousands of files matches a cache directory without recursively walking and statting every inode.
.jiri_cache_hash is managed by the CacheStamp abstraction and serves three essential functions:
$O(1)$ Version Roll Detection: The stamp file records the structured metadata of the instances linked into a WorkspaceDestination:
{ "format_version": 1, "target_path": "prebuilt/third_party/gn/linux-amd64", "instances": [ { "package": "gn/gn/linux-amd64", "version": "git_revision:b4f59045...", "instance_id": "GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C" } ] }
CacheStamp.MatchesTarget checks if the installed instances match the expected manifest target. If any package rolls to a new version, Jiri detects this in $O(1)$ time, wipes the directory, and reconstructs the forest. For legacy checkouts, CacheStamp automatically falls back to parsing plain-text hashes and instance IDs, upgrading them to structured JSON on the next write.
Build Hermeticity (mtime Preservation): Fuchsia checkouts contain ~150,000 prebuilt files (~30 GB). Touching or re-linking these files on every jiri update or fx build updates file modification times (mtime), invalidating build system action caches in Ninja and Bazel and triggering multi-hour incremental rebuilds. Checking .jiri_cache_hash validates the directory in < 1 millisecond with zero file touches.
Self-Describing & Reversible Audit Stamp: Because the stamp records the exact CIPD package name, version, and instance ID, developers and tools can inspect .jiri_cache_hash directly with cat or jq to discover what is installed on disk without parsing XML manifests or running CIPD.
package-cache=false and trueJiri avoids divergent code paths between cache-enabled and standalone checkouts. Both configurations pass through the same 3-stage lifecycle:
| Stage | package-cache=true | package-cache=false |
|---|---|---|
| Stage 1: Local Adoption | Adopts legacy group cache directories into instances/<id>. | Detaches WorkspaceDestination in-place by removing CacheStamp (<1ms). |
| Stage 2: Cache Sync | Synchronizes PackageCache for the main tree and all worktrees via WorktreePackageIndex. | Synchronizes PackageCache for any active worktrees that require caching. |
| Stage 3: Workspace Projection | Projects hardlink forest and writes CacheStamp. | Reuses cached instances via LinkPath if destination directory is missing; otherwise leaves standalone checkout untouched. |
During package synchronization, Jiri evaluates each workspace destination against an exhaustive state matrix:
| State | Config | Cache (instances/<id>) | Workspace Directory | Scenario | Expected Behavior | Network Download? |
|---|---|---|---|---|---|---|
| S1 | Enabled | Missing | Missing | Fresh checkout | Download instance to cache; hardlink to workspace; write stamp | Yes (initial fetch) |
| S2 | Enabled | Valid | Valid Hardlink | Steady-state / no-op | Stamp matches: fast-path return nil (<1ms) | No (fast path) |
| S3 | Enabled | Valid | Missing | New worktree or deleted workspace | Reconstruct hardlinks from cache; write stamp | No (instant link) |
| S4 | Enabled | Valid | Stale Hardlink | Roll to pre-cached version | Replace workspace with hardlinks to target cache; update stamp | No (instant link) |
| S5 | Enabled | Valid | Legacy Symlink | Upgrade legacy symlink checkout | Remove symlink; replace with hardlink forest; write stamp | No (local conversion) |
| S6 | Enabled | Missing | Stale Hardlink | Version roll with cache enabled | Clean workspace; download new instance to cache; hardlink to workspace | Yes (mandatory) |
| S7 | Enabled | Missing | Valid Standalone | Opt-in without roll | Download/verify clean instance in cache; link to workspace; write stamp | Yes (clean seed) |
| S8 | Enabled | Missing | Stale Standalone | Opt-in across a roll | Discard stale standalone; download new version clean to cache; hardlink | Yes (mandatory) |
| S9 | Enabled | Missing | Dirty Standalone | Dirty / untracked files in workspace | Discard dirty directory; clean download into cache; hardlink to workspace | Yes (clean download) |
| S10 | Enabled | Invalid | Any | Corrupted cache (non-dir or file) | Purge corrupt entry; re-download to cache; reconstruct hardlinks | Yes |
| S11 | Disabled | N/A | Missing | Fresh checkout, cache disabled | CIPD downloads standalone package directly to workspace; no stamp | Yes |
| S12 | Disabled | N/A | Valid Standalone | Steady-state, cache disabled | CIPD verifies standalone package in-place, no-op | No (CIPD no-op) |
| S13 | Disabled | N/A | Stale Standalone | Version roll, cache disabled | CIPD updates standalone workspace directory in-place | Yes (update download) |
| S14 | Disabled | Valid | Valid Hardlink | Opt-out of cache (true -> false) | Retain existing hardlinks in-place; remove stamp file in O(1) time | No (instant detach) |
| S15 | Disabled | Valid | Missing | Cache disabled, missing workspace | Hardlink from cache to workspace via LinkPath, avoiding network download | No (local hardlink) |
| S16 | Disabled | Missing | Stale Hardlink | Opt-out across a version roll | Remove stale hardlinks and stamp; CIPD downloads new standalone version | Yes (mandatory) |
| S17 | Disabled | Valid | Missing (Roll) | Roll while restoring from cache | Restore matching version from cache or clean download if uncached | No (if cached) |
| S18 | Multi-WT | Valid | Multiple Worktrees | Multi-worktree roll & GC | Instances used by any worktree are preserved by CIPD; obsolete prebuilts cleaned | Yes (for new version) |
With the package cache, garbage collection is completely streamlined:
Retirement of Bespoke Cache GC: Jiri's legacy custom garbage collector (PackageGC), which walked all worktrees, matched directory timestamps, and evaluated mtime cutoff heuristics, has been completely removed. Cache pruning is handled natively and authoritatively by CIPD during Stage 2 synchronization based on the master ensure file.
Obsolete Prebuilt Directory Cleanup: When packages roll or are removed from the manifest, obsolete prebuilt directories or dangling symlinks left behind in destination directories are cleaned up by cleanObsoletePrebuiltsForWorkspace during jiri update -gc and jiri worktree prune.
The package cache implementation is verified by unit and integration test suites:
//project/package_cache_test.go: Consolidated test suite executing all 18 state machine scenarios (S1–S18), verifying inode relationships (stat.Ino), fast-path stamp verification, concurrency locking, multi-worktree rolls, and native CIPD pruning.
//integrationtests/package_cache_toggle_test.go: End-to-end integration test verifying seamless toggling between package-cache=true and package-cache=false.
//cmd/jiri/subcommands/update_test.go: Verifies jiri update -gc cleans obsolete cached prebuilt directories.
Originally, Jiri grouped packages sharing a destination path:
Jiri computed a composite group hash: sha256(pkgA.Name + ":" + pkgA.Version + "\n" + pkgB.Name + ":" + pkgB.Version).
CIPD downloaded the combined packages into a temporary staging directory (tmp/merged-*).
Jiri renamed tmp/merged-* into .jiri_root/packages/<group_hash>.
Jiri recursively hardlinked .jiri_root/packages/<group_hash> to workspaceDir.
Jiri implemented a custom PackageGC scanner (~450 lines of Go code) that walked all worktrees using findUsedCachePaths to prevent active cache directories from being deleted based on 7-day mtime thresholds.
Redundant Downloads on Partial Rolls: When one package in a multi-package destination rolled (e.g. host tool updated from v1 to v2), the composite group hash changed from hash(common:v1, tool:v1) to hash(common:v1, tool:v2). CIPD was forced to re-download or re-extract common (e.g. 1.5 GB SDK common headers), even though common had not changed at all.
Duplication Across Architectures: If Linux and macOS builds shared a platform-independent package (e.g. sdk_common), Jiri generated two separate composite directories in the cache (hash(common, linux) and hash(common, mac)), storing duplicate copies of common.
Fragile Bespoke Garbage Collection: Walking multiple worktrees, matching directory stamps, and guessing activity via mtime heuristics was error-prone and frequently led to active worktree dependencies being deleted from the cache or orphaned directories accumulating indefinitely.
Temporary Staging Orchestration: Managing tmp/merged-* staging directories, atomic renames, and failure rollbacks added substantial complexity and failure modes.
The instance-based cache replaces the legacy composite group model with an instance-level store managed natively by CIPD, delivering zero network redundancy, native pruning, and sub-millisecond steady-state performance.