A passed smoke should say what it did not test

A production smoke passes. The app opens, authentication completes, and one critical screen loads. The release record turns green.

That result is useful, but it is easy to read too much into it. The smoke did not prove every provider path, every account state, every background transition, or every device condition. It proved a narrow journey against a particular artifact and environment.

At Ubriot, we think a passed smoke should say what it did not test. A green result becomes more trustworthy when its boundary is visible beside it, not buried in a script or left to the memory of the operator who ran it.

Write the result as a bounded claim

Passed is too small on its own. A useful result names the artifact, destination, tested path, test identity, and finish condition. It should also name exclusions that matter to the release decision.

result: passed
artifact: app-release-184
destination: production
path: launch > sign in > release list > release detail
finish-condition: detail loaded with current environment
not-tested: new-account onboarding, offline recovery, provider fallback

The exclusions are not an apology for the test. They are part of the proof. An operator can see that the release detail path is current while onboarding remains supported only by earlier evidence. If the change touched onboarding, the gap is obvious. If it touched release detail rendering, the smoke is directly relevant.

This also makes repeated checks comparable. Two green smokes may have exercised different accounts or stopped at different finish conditions. Without a bounded claim, the later result can appear to refresh evidence it never repeated.

Choose exclusions before the run

Teams often write limitations after something goes wrong. That is too late. The operator should know the smoke boundary before pressing run so the release decision can account for the missing paths deliberately.

We prefer three questions at setup. Which changed behavior must this smoke exercise? Which adjacent failure would still escape it? Which older proof, if any, covers that adjacent risk? The answers keep a focused smoke focused without allowing it to masquerade as broad certification.

Consider a mobile release that changes how runtime configuration selects an API endpoint. A smoke against the primary endpoint can prove the main production path. It does not automatically prove offline recovery or fallback routing. If fallback behavior is unchanged and carries current evidence, the release record can inherit that proof. If configuration also changed the fallback, another check is required.

Account state needs the same precision. A smoke performed with an established operator account can prove that existing sessions reach the changed release view. It says nothing about account creation, invitation acceptance, or a user whose organisation has no prior releases. The test identity is therefore part of the boundary, even when the interface path looks identical after sign-in.

This is where honest exclusions save time. The team does not have to run every available test because the screen is vague. It can add the one missing check that follows the changed dependency.

Let uncovered risk drive the next action

A not-tested list should not become a permanent disclaimer attached to every green result. It should influence the decision. Some exclusions are acceptable because they are outside the change, covered elsewhere, or below the agreed release boundary. Others should block promotion.

The product can help by classifying each exclusion. Covered by current evidence. Unchanged and accepted. Requires focused proof. Outside this release contract. Those labels make the gap actionable without pretending software can decide the team's risk tolerance on its own.

An exclusion can also expire. Offline recovery may be outside today's release contract, then become central when a later configuration change touches local caching. We retain the old boundary with the old result and evaluate the new release against its own contract. Editing history to make an earlier smoke look broader would destroy exactly the evidence the team needs to understand how coverage changed.

The same record improves incident review. If a release fails on a path the smoke explicitly excluded, the team can ask whether the release contract was too narrow. If the smoke claimed to cover the path but missed the failure, the check itself needs repair. Those are different problems and should produce different work.

We build Ubriot so release evidence stays attached to the exact artifact, environment, and decision it supports. A passed smoke should say what it did not test because confidence is strongest when the green result remains narrow enough to be true. Name the path, name the finish condition, name the meaningful exclusions, then let the uncovered risk determine what must happen before the release moves again.