CI outages need a temporary source of truth

When CI goes down, the team still has to keep production alive. A fix may need to be built directly on the host. A service may need a local image because the normal publish path is unavailable. That improvisation can be responsible. The risky part starts later, when the urgent artifact remains real in production but unofficial everywhere else.

We think a CI outage needs a temporary source of truth. If the host is now running an artifact that never passed through the usual pipeline, the release record should promote that fact immediately. Not as a footnote in chat. Not as something the deploy owner simply remembers. As an explicit trust record that says what is running, why it exists, and what would need to happen before the normal pipeline becomes authoritative again.

Without that record, teams enter an especially dangerous kind of normal. Production is healthy enough. The incident pressure eases. A few days later someone restarts a service, pulls from the registry, or rebuilds a host under the assumption that official state still lives where it usually lives. If the registry never received the emergency artifact, that routine action can quietly move production backward.

The emergency artifact stops being safe the moment it becomes invisible

A host-only build can be the right move during an outage. The problem is not the exception. The problem is leaving the exception undocumented after the service is stable again. Once the artifact exists only inside one machine and one person's memory, the team loses the ability to reason about recovery cleanly.

The stronger move is to make the exception legible. Running artifact sha256 X was built directly on host because registry publishing was unavailable. Registry artifact sha256 Y is older. Normal deploy path regains authority after publish proof for X or a later replacement artifact. That turns emergency state into release memory the whole team can still use later.

This also improves handoff. Someone joining the incident later does not need to infer why the running digest differs from the most recent registry digest. The system already explains the divergence, which means the next person can decide from evidence instead of folklore.

The temporary source of truth should also name the exit condition. Otherwise the outage state lingers indefinitely. The team needs to know whether the next safe step is publishing the emergency image, rebuilding it through CI once CI returns, or replacing it with a new verified artifact. Until that exit is named, routine maintenance remains more dangerous than it looks.

It should also name the operator cost of getting this wrong. If the temporary record disappears too early, every later restart becomes an archaeology exercise. People have to compare digests by hand, search old incident notes, and guess whether the host or the registry is the safer memory. That is avoidable confusion.

Outage recovery should preserve authority boundaries, not blur them

CI trouble already weakens confidence because one normal proof path has disappeared. The release surface should compensate by becoming more explicit about authority, not less. Which artifact is trusted right now. Which system would ordinarily define truth. Why is that system temporarily behind reality. Which proof will restore the usual order. Those are the questions operators need answered before they start treating maintenance as routine again.

That record can stay lightweight. It does not need a grand incident process to be useful. It needs one visible place where the exceptional artifact is named, bounded, and linked to its return path. Small clarity does a lot of work here because it prevents the next quiet command from undoing the emergency fix by accident.

We build Ubriot around release memory because outages are exactly when teams are forced to improvise, and improvisation is exactly when hidden state becomes most expensive later. A temporary source of truth keeps the emergency artifact visible until the normal pipeline can honestly take over again.

CI outages need a temporary source of truth because production cannot safely depend on unofficial memory for very long. Once the running host artifact, the older official artifact, and the exit condition are visible together, recovery becomes steadier and accidental rollback gets much harder to hide inside routine follow-up work.