Jiri Package Cache

1. Overview & Architecture

Jiri manages prebuilt packages (such as toolchains, compilers, and SDKs) distributed via Chrome Infrastructure Package Deployer (CIPD).

In multi-worktree development environments or when packages update frequently, installing standalone copies of packages directly into each workspace duplicates tens of gigabytes of identical binaries across worktrees and incurs repeated network downloads and disk space exhaustion.

To solve this, Jiri provides an instance-based, content-addressable package cache shared across the main checkout and all active worktrees.

┌─────────────────────────────────────────────────────────────────────────────┐
                   PackageCache (.jiri_root/packages/)                       
                                                                             
  instances/                                                                 
  ├── <A5DE1AnhsOBwACTgzeukZ-l2FbYuA9Q8tQNrHaFQblYC>/  ─── [Hardlink] ──┐    
     ├── bin/gn                                                            
     └── ...                                                               
                                                                  Workspace 
  ├── <bdupDu8hA7pXmKNwzF8WTbt1rgDro7L5srqIBXBigisC>/  ─── [Hardlink] ─► Destination
     └── bin/clang                                                         
                                                                           
  └── <SPHCOQ38Z7qb62orRA-gTUIZdEVnEdsb_qx5iKCUVZQC>/  ─── [Hardlink] ──┘    
      └── ...                                                                
└─────────────────────────────────────────────────────────────────────────────┘

Core Concepts & Terminology

  • PackageCache: The central cache manager for the repository-level package cache located at [root]/.jiri_root/packages. Manages the global cache lock (.cache.lock), CIPD ensure synchronization, and cache directory layout.

  • PackageCacheInstance: An immutable, content-addressed CIPD package instance stored in the cache at [root]/.jiri_root/packages/instances/<instance_id>, keyed directly by its canonical CIPD Instance ID. Once downloaded, its contents are pristine and never modified.

  • WorkspaceDestination: A destination directory within a specific checkout or worktree where binaries and tools reside (for example, <workspaceRoot>/prebuilt/third_party/gn/linux-amd64). Encapsulates the ordered list of package instances to project into that directory.

  • CacheStamp (.jiri_cache_hash): An atomic verification stamp file placed at <destDir>/.jiri_cache_hash that records structured JSON metadata describing the target path and package instances currently linked into the destination directory. Enables sub-millisecond fast-path verification and safe cache detachment.

  • CIPDEnsureFile: In-memory and on-disk representation of the CIPD ensure specification, mapping every required package instance to @Subdir instances/<instance_id>.

  • WorktreePackageIndex: The aggregated package registry that discovers all active worktrees, computes the unique union of PackageCacheInstance objects, and maps WorkspaceDestination targets per workspace.

  • Workspace Hardlink Forest: The projection of one or more cached package instances (PackageCacheInstance) into a WorkspaceDestination via recursive hard links.


2. The Three-Stage Lifecycle of jiri fetch-packages

Every package fetch operation in Jiri (jiri fetch-packages or jiri update) executes through a single, unified three-stage pipeline:

┌─────────────────────────────────────────────────────────────────────────────┐
                 Stage 1: Migrations & Local Adoption                        
├─────────────────────────────────────────────────────────────────────────────┤
  Non-blocking lock on .jiri_root/packages/.cache.lock acquired at entry    
  In-place local filesystem optimization; eliminates remote network calls   
  Adopts legacy package-group entries (.jiri_root/packages/<hex64>)         
  Adopts standalone CIPD installations (<root>/.cipd) into package cache    
  Immediately removes legacy directories upon adoption                      
  Detaches workspace in O(1) time when package-cache=false is selected      
  Best-effort: failures are never fatal; Stage 2 guarantees convergence     
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       
                                       
┌─────────────────────────────────────────────────────────────────────────────┐
             Stage 2: Native CIPD Cache Synchronization & Pruning            
├─────────────────────────────────────────────────────────────────────────────┤
  Held under .cache.lock across ensure generation, CIPD sync, and pruning   
  Accumulates all instances across main tree and active worktrees           
  Master ensure file executed via `cipd ensure` with $ParanoidMode          
  Self-healing: validates, repairs, and downloads missing instances         
  Natively deletes unreferenced instances from instances/<instance_id>      
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       
                                       
┌─────────────────────────────────────────────────────────────────────────────┐
             Stage 3: Workspace Hardlink Forest Construction                 
├─────────────────────────────────────────────────────────────────────────────┤
  Evaluates each WorkspaceDestination declared in checkout manifest         
  FAST-PATH: If destination CacheStamp (.jiri_cache_hash) matches, <1ms    
  SLOW-PATH: Reconstructs clean hardlink forest from PackageCacheInstance   
  Resolves file collisions in manifest order, matching CIPD semantics       
  Writes atomic completion stamp (.jiri_cache_hash) as structured JSON      
  Standalone checkouts (package-cache=false) remain untouched               
  Held under .cache.lock; released upon completion of Stage 3 via defer     
└─────────────────────────────────────────────────────────────────────────────┘

Stage 1: Migrations & Local Adoption

Implemented in project/cache_migration.go. Stage 1 is a dedicated, zero-network migration phase executed under .cache.lock before any sync or linking:

  • Zero-Download Legacy Group Adoption (MigrateLegacyGroupCache): If legacy package-group directories (.jiri_root/packages/<hex64>) exist, Jiri inspects their internal CIPD state (.cipd/pkgs/<index>/<instance_id>). Because CIPD internally keeps individual package instances isolated in these subdirectories, Jiri hardlinks the pristine files directly into instances/<instance_id> and replicates the metadata into .jiri_root/packages/.cipd/pkgs/. This converts the existing cache locally in ~1–2 seconds with 0 bytes of network traffic, immediately removing the legacy directory upon completion.

  • Zero-Download Standalone Adoption (MigrateStandaloneToPackageCache): When a developer enables the package cache on a previously standalone checkout, Jiri inspects <root>/.cipd/pkgs/, hardlinks the pristine instances into instances/<instance_id>, seeds CIPD metadata into the shared cache, and retires <root>/.cipd. Stage 2 then recognizes the packages as already installed, requiring 0 bytes of network download.

  • Zero-Download Cache Opt-Out (MigratePackageCacheToStandalone): When switching to package-cache=false, Jiri detaches each matching workspace directory in $O(1)$ time (<1ms) by removing .jiri_cache_hash, and restores any missing directories from the local package cache via hardlinks without downloading.

  • Best-Effort Tolerance: Any unexpected read error during adoption is logged as debug information and safely skipped; Stage 2 guarantees that all declared instances are cleanly resolved and installed.

Stage 2: Native CIPD Cache Synchronization & Pruning

Implemented in project/package_cache.go. Stage 2 establishes the ground truth of the cache repository:

  • Lock Lifecycle & Concurrency:

    • Acquisition: Before Stage 1 begins, Jiri acquires an exclusive non-blocking file lock on .jiri_root/packages/.cache.lock. If another worktree or concurrent Jiri process is currently synchronizing or linking packages, Jiri fails fast immediately with an actionable message prompting the user to wait and retry.
    • Scope: The lock is held continuously across the entire package cache pipeline: Stage 1 (migration and retirement), Stage 2 (master ensure generation, CIPD execution, and pruning), and Stage 3 (workspace hardlink forest construction and stamping).
    • Release: The lock is released via defer unlock() at the completion of Stage 3. This provides complete end-to-end atomicity, eliminating any possible races between concurrent package operations or background pruning.
  • Multi-Worktree Collection: Jiri inspects the active manifests of the main checkout and all registered worktrees, accumulating the complete set of required package instances into a single master ensure file (instance_master.ensure). Any unpinned tags or floating versions are resolved via cipd.ResolveEnsureFile. Broken or inaccessible worktrees are tolerated with a warning.

  • CIPD Native Pruning: Jiri invokes cipd ensure targeting [root]/.jiri_root/packages. CIPD verifies all declared instances under instances/<instance_id>. Crucially, CIPD natively unlinks and deletes any instance directories under instances/ that are no longer referenced by the master ensure file, completely eliminating the need for bespoke garbage collection.

Stage 3: Workspace Hardlink Forest Construction

Implemented in project/workspace_destination.go.

Stage 3 projects the synchronized instance store into each WorkspaceDestination inside the active checkout:

  • Fast-Path Verification ($O(1)$): If CacheStamp (<destDir>/.jiri_cache_hash) exists and its contents match the expected target hash via MatchesTarget, Jiri immediately returns with 0 I/O. File modification times (mtime) are preserved, preventing spurious rebuilds in Ninja and Bazel.

  • Clean Forest Reconstruction: If the verification stamp is missing or mismatched (e.g. following a version roll), Jiri removes the destination directory and reconstructs a clean directory structure by recursively hardlinking from the required PackageCacheInstance directories (instances/<instance_id>).

  • Manifest-Order Collision Resolution: When multiple packages share the same destination path (for example, common SDK headers and platform-specific binaries), instances are linked sequentially in manifest order. If two packages provide the same file path, the later package in manifest order replaces the file link, faithfully reproducing CIPD's native multi-package semantics.

  • Atomic Completion Marker: The stamp file .jiri_cache_hash is written as the final step after all links and symlinks are successfully created.


3. Why Use CIPD Instance IDs as the Canonical Key?

Rather than synthesizing an ad-hoc hash (such as sha256(name:version)), the package cache keys each instance directory directly by its CIPD Instance ID (for example, GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C):

  1. Canonical Content Addressability: The Instance ID is CIPD's SHA-256 digest of the package archive payload. It represents the canonical ground truth of the files contained within the package.

  2. Deduplication Across Tags and References: Different git tags or floating refs (for example, git_revision:abc1234 and latest) that point to the exact same build artifact resolve to the identical Instance ID. Worktrees referencing different tags pointing to the same build share a single instance directory automatically.

  3. Hermetic Lockfile Alignment: In Fuchsia, all packages are pinned in jiri.lock, which records the instance_id for each package. The master ensure file is fully pinned and reproducible:

    @Subdir instances/GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C
    gn/gn/linux-amd64 GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C
    

4. The Role and Necessity of .jiri_cache_hash

Because hardlinked files are standard, regular files residing in ordinary directories, there is no OS primitive on Linux or macOS to check whether a directory tree of tens of thousands of files matches a cache directory without recursively walking and statting every inode.

.jiri_cache_hash is managed by the CacheStamp abstraction and serves three essential functions:

  1. $O(1)$ Version Roll Detection: The stamp file records the structured metadata of the instances linked into a WorkspaceDestination:

    {
      "format_version": 1,
      "target_path": "prebuilt/third_party/gn/linux-amd64",
      "instances": [
        {
          "package": "gn/gn/linux-amd64",
          "version": "git_revision:b4f59045...",
          "instance_id": "GvbSt6gklhdVYRp9r7bot-rllr0smxfv9rAElX3K798C"
        }
      ]
    }
    

    CacheStamp.MatchesTarget checks if the installed instances match the expected manifest target. If any package rolls to a new version, Jiri detects this in $O(1)$ time, wipes the directory, and reconstructs the forest. For legacy checkouts, CacheStamp automatically falls back to parsing plain-text hashes and instance IDs, upgrading them to structured JSON on the next write.

  2. Build Hermeticity (mtime Preservation): Fuchsia checkouts contain ~150,000 prebuilt files (~30 GB). Touching or re-linking these files on every jiri update or fx build updates file modification times (mtime), invalidating build system action caches in Ninja and Bazel and triggering multi-hour incremental rebuilds. Checking .jiri_cache_hash validates the directory in < 1 millisecond with zero file touches.

  3. Self-Describing & Reversible Audit Stamp: Because the stamp records the exact CIPD package name, version, and instance ID, developers and tools can inspect .jiri_cache_hash directly with cat or jq to discover what is installed on disk without parsing XML manifests or running CIPD.


5. Unified Pipeline for package-cache=false and true

Jiri avoids divergent code paths between cache-enabled and standalone checkouts. Both configurations pass through the same 3-stage lifecycle:

Stagepackage-cache=truepackage-cache=false
Stage 1: Local AdoptionAdopts legacy group cache directories into instances/<id>.Detaches WorkspaceDestination in-place by removing CacheStamp (<1ms).
Stage 2: Cache SyncSynchronizes PackageCache for the main tree and all worktrees via WorktreePackageIndex.Synchronizes PackageCache for any active worktrees that require caching.
Stage 3: Workspace ProjectionProjects hardlink forest and writes CacheStamp.Reuses cached instances via LinkPath if destination directory is missing; otherwise leaves standalone checkout untouched.

6. State Machine & Invariants (S1–S18)

During package synchronization, Jiri evaluates each workspace destination against an exhaustive state matrix:

StateConfigCache (instances/<id>)Workspace DirectoryScenarioExpected BehaviorNetwork Download?
S1EnabledMissingMissingFresh checkoutDownload instance to cache; hardlink to workspace; write stampYes (initial fetch)
S2EnabledValidValid HardlinkSteady-state / no-opStamp matches: fast-path return nil (<1ms)No (fast path)
S3EnabledValidMissingNew worktree or deleted workspaceReconstruct hardlinks from cache; write stampNo (instant link)
S4EnabledValidStale HardlinkRoll to pre-cached versionReplace workspace with hardlinks to target cache; update stampNo (instant link)
S5EnabledValidLegacy SymlinkUpgrade legacy symlink checkoutRemove symlink; replace with hardlink forest; write stampNo (local conversion)
S6EnabledMissingStale HardlinkVersion roll with cache enabledClean workspace; download new instance to cache; hardlink to workspaceYes (mandatory)
S7EnabledMissingValid StandaloneOpt-in without rollDownload/verify clean instance in cache; link to workspace; write stampYes (clean seed)
S8EnabledMissingStale StandaloneOpt-in across a rollDiscard stale standalone; download new version clean to cache; hardlinkYes (mandatory)
S9EnabledMissingDirty StandaloneDirty / untracked files in workspaceDiscard dirty directory; clean download into cache; hardlink to workspaceYes (clean download)
S10EnabledInvalidAnyCorrupted cache (non-dir or file)Purge corrupt entry; re-download to cache; reconstruct hardlinksYes
S11DisabledN/AMissingFresh checkout, cache disabledCIPD downloads standalone package directly to workspace; no stampYes
S12DisabledN/AValid StandaloneSteady-state, cache disabledCIPD verifies standalone package in-place, no-opNo (CIPD no-op)
S13DisabledN/AStale StandaloneVersion roll, cache disabledCIPD updates standalone workspace directory in-placeYes (update download)
S14DisabledValidValid HardlinkOpt-out of cache (true -> false)Retain existing hardlinks in-place; remove stamp file in O(1) timeNo (instant detach)
S15DisabledValidMissingCache disabled, missing workspaceHardlink from cache to workspace via LinkPath, avoiding network downloadNo (local hardlink)
S16DisabledMissingStale HardlinkOpt-out across a version rollRemove stale hardlinks and stamp; CIPD downloads new standalone versionYes (mandatory)
S17DisabledValidMissing (Roll)Roll while restoring from cacheRestore matching version from cache or clean download if uncachedNo (if cached)
S18Multi-WTValidMultiple WorktreesMulti-worktree roll & GCInstances used by any worktree are preserved by CIPD; obsolete prebuilts cleanedYes (for new version)

7. Garbage Collection & Obsolete Prebuilts

With the package cache, garbage collection is completely streamlined:

  1. Retirement of Bespoke Cache GC: Jiri's legacy custom garbage collector (PackageGC), which walked all worktrees, matched directory timestamps, and evaluated mtime cutoff heuristics, has been completely removed. Cache pruning is handled natively and authoritatively by CIPD during Stage 2 synchronization based on the master ensure file.

  2. Obsolete Prebuilt Directory Cleanup: When packages roll or are removed from the manifest, obsolete prebuilt directories or dangling symlinks left behind in destination directories are cleaned up by cleanObsoletePrebuiltsForWorkspace during jiri update -gc and jiri worktree prune.


8. Verification & Test Suite

The package cache implementation is verified by unit and integration test suites:

  • //project/package_cache_test.go: Consolidated test suite executing all 18 state machine scenarios (S1–S18), verifying inode relationships (stat.Ino), fast-path stamp verification, concurrency locking, multi-worktree rolls, and native CIPD pruning.

  • //integrationtests/package_cache_toggle_test.go: End-to-end integration test verifying seamless toggling between package-cache=true and package-cache=false.

  • //cmd/jiri/subcommands/update_test.go: Verifies jiri update -gc cleans obsolete cached prebuilt directories.


Appendix A: Historical Context & Legacy Group Cache Analysis

The Legacy “Package Group” Architecture

Originally, Jiri grouped packages sharing a destination path:

  • Jiri computed a composite group hash: sha256(pkgA.Name + ":" + pkgA.Version + "\n" + pkgB.Name + ":" + pkgB.Version).

  • CIPD downloaded the combined packages into a temporary staging directory (tmp/merged-*).

  • Jiri renamed tmp/merged-* into .jiri_root/packages/<group_hash>.

  • Jiri recursively hardlinked .jiri_root/packages/<group_hash> to workspaceDir.

  • Jiri implemented a custom PackageGC scanner (~450 lines of Go code) that walked all worktrees using findUsedCachePaths to prevent active cache directories from being deleted based on 7-day mtime thresholds.

Flaws of the Legacy Package Group Design

  1. Redundant Downloads on Partial Rolls: When one package in a multi-package destination rolled (e.g. host tool updated from v1 to v2), the composite group hash changed from hash(common:v1, tool:v1) to hash(common:v1, tool:v2). CIPD was forced to re-download or re-extract common (e.g. 1.5 GB SDK common headers), even though common had not changed at all.

  2. Duplication Across Architectures: If Linux and macOS builds shared a platform-independent package (e.g. sdk_common), Jiri generated two separate composite directories in the cache (hash(common, linux) and hash(common, mac)), storing duplicate copies of common.

  3. Fragile Bespoke Garbage Collection: Walking multiple worktrees, matching directory stamps, and guessing activity via mtime heuristics was error-prone and frequently led to active worktree dependencies being deleted from the cache or orphaned directories accumulating indefinitely.

  4. Temporary Staging Orchestration: Managing tmp/merged-* staging directories, atomic renames, and failure rollbacks added substantial complexity and failure modes.

The instance-based cache replaces the legacy composite group model with an instance-level store managed natively by CIPD, delivering zero network redundancy, native pruning, and sub-millisecond steady-state performance.