We shipped v1.0.10 and it contained the old assets: a "shipped" log proves nothing

The release that said shipped

Our automated ship script reported “v1.0.9 shipped.” We downloaded the release from object storage and inspected it. The workspaces feature, MCP tools, and language additions we had made that day were missing. We had distributed old code under a new version number.

The quality gate had passed 445 tests. That result was real, but we had treated it as evidence about something the tests had never examined: the code used to build the release. The script’s completion message added another apparent confirmation without establishing what was inside the download.

We followed the release process backward, from the archive to the packager and then to the source it consumed. The tests and the packager were looking at different trees.

We tested one tree and shipped another

The quality gate ran against our development working tree. The packager built from a separate clone. That clone received updates through the remote, while commits accumulated locally in the development tree without reaching it.

We had assumed that passing the gate meant the source was ready to package. In practice, the gate and the packager had no such connection. A passing test result described the development tree. The clone could still contain earlier code, and the release could receive a new version number without receiving the changes we intended to ship.

We changed the ship process to copy the committed development tree into the clone before packaging, then commit and push that update. This addressed the missing connection between the trees.

The word “committed” would matter later. At this point, we had repaired the transfer of committed changes. We had not established that everything we intended to release was committed.

The automation had not been running

The investigation also overturned our assumption that daily automatic shipping was already operating. It had been configured, but the release history contained no automatic releases.

In 5 places, a fullwidth character immediately followed an unbraced shell variable such as $VERSION. Bash consumed part of the multibyte character as part of the variable name. With set -u enabled, the resulting reference failed as an unbound variable.

We changed those references to use braces, such as ${VERSION}, so the variable name had an explicit boundary. The configuration had expressed an intention to run every day. It had not provided evidence that the script could finish.

A dry run exposed another problem. After shipping, the script changed the version, but it saved the tree hash from before that change. On the next run, its own version update would look like new work. It would keep shipping another version every morning.

We moved the hash calculation after the version update and saved that result. We then observed the script exit without shipping when there was no further change.

Our verification of v1.0.10 was incomplete

With those fixes in place, we shipped v1.0.10 and downloaded the archive from object storage. We checked its listing for the feature additions and confirmed that the product record pointed to v1.0.10.

We also treated the asset listing as confirmation that the replacement assets were present. That was where our verification stopped short. We had looked at names and counts, not the contents of the files behind those names.

A colleague downloaded the ZIP and hashed the asset files. The hashes matched an older asset set we had already replaced.

Our statement that the release was verified did not survive that comparison. The archive contained the expected entries, but those entries held older bytes. Listing the ZIP had answered whether files were present. We had used it to answer whether they were the intended versions.

The replacement had never been committed

The replacement involved 37 files that existed only in the working tree. They had never been committed. Another 4 dependent modules were untracked.

Our repair copied the committed tree into the packaging clone. It therefore copied the older asset set, even though the development working tree contained its replacement. The transfer now worked as implemented; our assumption about its input was still wrong.

We committed the replacement and the dependent modules, quarantined the v1.0.9 and v1.0.10 archives, and shipped again, reusing v1.0.9. This time, we compared the asset files in the object-storage download against the reference by bytes. The result was 16/16 matching the reference, with 0 old assets.

We added that comparison to the ship script, along with a check that the dependent modules existed. The check now covered the distinction the archive listing had missed.

We also ran it against the quarantined v1.0.10 ZIP. It failed with the old assets detected. That negative test showed that the new check rejected the artifact we had previously called verified.

The lesson

Our own “verified” claim was overturned three times that day because we kept allowing evidence about part of the process to stand in for evidence about the release. Passing tests described the tested tree, copying commits described the committed tree, and listing an archive described its entries. We needed to download the distributed artifact and compare its contents by bytes, with the intended replacement committed before packaging. We turned the colleague’s comparison into a ship check and confirmed that it rejected the quarantined release.

Root Cause is written by TechAthletes. The product in this story is Agent Tile, our macOS app for keeping many Claude Code sessions in view at once — one-time purchase, full source included.