[pkg-cache] Add blob write retry

This change attempts to workaround an inconsistency between
pkgfs/install and pkg-cache when fetching a package, where pkgfs/install
reports a blob is still needed but pkg-cache has already either decided
it was already present or has written it.  After processing all blobs in
the package, pkg-cache will re-query pkgfs/install and open any
remaining blobs for write.  If pkgfs discovers the blob is present (as
it should be at this point), it will unblock the package install. If
this blob data is somehow still needed, the package cache operation will
fail as it did before.

Bug: 74326
Change-Id: I0ff191d9061a530159bc43e90bdd854a9b05bfa7
Reviewed-on: https://fuchsia-review.googlesource.com/c/fuchsia/+/514654
Fuchsia-Auto-Submit: Kevin Wells <kevinwells@google.com>
Reviewed-by: Aaron Wood <aaronwood@google.com>
Commit-Queue: Auto-Submit <auto-submit@fuchsia-infra.iam.gserviceaccount.com>
diff --git a/src/sys/pkg/bin/pkg-cache/src/cache_service.rs b/src/sys/pkg/bin/pkg-cache/src/cache_service.rs
index 6f2f303..29c9e62 100644
--- a/src/sys/pkg/bin/pkg-cache/src/cache_service.rs
+++ b/src/sys/pkg/bin/pkg-cache/src/cache_service.rs
@@ -409,6 +409,28 @@
                     "BAD_STATE: pkgfs needs more blobs, but all were provided: {:#?}",
                     needs
                 );
+
+                for need in needs {
+                    // open each blob for write.  If the blob already exists, this should fulfill
+                    // it in pkgfs as well, unblocking this package fetch.  Though, the open that
+                    // happened during the package fetch should have already done that.
+                    match open_blob(pkgfs_install, need, BlobKind::Data).await {
+                        Ok(OpenBlobSuccess::AlreadyExists) => {
+                            fx_log_err!(
+                                "BAD_STATE: already written blob needed opened again to commit"
+                            );
+                        }
+                        Ok(OpenBlobSuccess::Needed(_)) => {
+                            fx_log_err!("BAD_STATE: already written blob needs written again");
+                        }
+                        Err(e) => {
+                            fx_log_err!(
+                                "BAD_STATE: error opening already written blob for write: {:#}",
+                                anyhow!(e)
+                            );
+                        }
+                    }
+                }
             }
             Err(e) => {
                 fx_log_err!(