I was looking at a perfectly valid audit event and realizing how little of the story it contained. The cryptography was fine; the explanation was missing.
The record said that a plan had been admitted and a VM had started. It was ordered, signed, and chained to the events around it. I could verify that MVM had written it and that nobody had quietly edited the history afterward.
What I could not tell was who had authorized the action, why an exception had been acceptable, which policy or ticket the decision relied on, or what later work had inherited that choice.
The log was intact. The meaning around it was missing, and no amount of signing the same small record would put it back.
That was the point at which I stopped using logging, audit, and provenance as if they were different levels of the same feature.
They are different jobs.
Three records, three questions
A normal log is primarily for operating the system. It helps answer questions such as: Why did this process crash? Which backend was selected? How long did the connection take? What error did the guest return?
Audit evidence has another purpose. It records security-relevant events in a durable, verifiable sequence: which plan was admitted, which image digest launched, which capability was invoked, whether an egress request was allowed, and whether the record has been altered.
Decision provenance adds the part humans usually reconstruct after the fact. Who exercised the authority? Why did they do it? Was this a normal deployment or an emergency exception? Which earlier decision caused this one, and which later decisions should be revisited if it was wrong?
Those layers overlap, but one cannot substitute for the others.
A text log may contain the right facts without making them queryable. A signed event may be tamper-evident while remaining semantically thin. A detailed rationale may explain a decision while carrying no authority at all.
The architecture improves when each layer is allowed to do its own work.
A signature preserves the context you actually record
Consider a minimal event:
{
"event": "plan.admitted",
"plan_id": "sha256:...",
"timestamp": "..."
}Once it is signed and chained, I can prove that MVM recorded those fields at that point in the sequence.
The signature cannot recover the context we omitted. It does not tell me whether a person or an automated service authorized the plan, whether the network grant came from a routine policy or a temporary exception, or whether a later checkpoint depended on the decision.
Cryptographic integrity is extremely valuable. It preserves the semantics we chose to write down; it does not invent them.
That sounds obvious when stated directly. It is easy to forget once “the audit log is signed” becomes the end of the design discussion.
Give the decision an identity of its own
MVM’s provenance layer adds a typed decision record beside the audit event. A simplified shape looks like this:
struct DecisionRecord {
decision_id: DecisionId,
category: DecisionCategory,
actor: ActorRef,
scenario: DecisionScenario,
reasoning: String,
outcome: DecisionOutcome,
causal_links: Vec<CausalLink>,
metadata: DecisionMetadata,
attestation: AttestationBinding,
}The record is content-addressed. Its identifier is derived from a canonical representation of the decision body:
That gives the decision a stable identity independent of the database or index that happens to make it searchable.
The chain-signed audit log remains the authoritative sequence. The decision store is a derived index that can be rebuilt.
That direction was the most important design choice.
The explanation must not become the authority
Structured provenance is attractive because it gathers the actor, rationale, metadata, and relationships in one place. Once the database becomes useful, there is a temptation to let it decide what is allowed.
MVM does not do that.
The signed execution plan, capability grants, admission policy, and host-side enforcement remain authoritative. Provenance records why those authorities were exercised and what followed from the decision.
flowchart TB
P[Signed plan and policy] --> E[Enforcement]
E --> A[Chain-signed audit event]
A --> D[Decision record]
D --> I[Rebuildable decision index]
D -. records .-> P
D -. does not authorize .-> E
If the decision index disappears, MVM can reconstruct it from the chain. If the audit chain disappears, the index cannot promote itself into a replacement source of truth.
I use a simple review question for these layers now:
If this storage system is unavailable, should the security decision change?
For a provenance index or a rendered receipt, the answer is no. For signed admission or required audit durability, the answer may be yes.
That question keeps a reporting feature from quietly turning into a control plane.
The graph is where provenance becomes useful
A flat list of richly structured decisions is still a list. During an incident, people think in causes and consequences.
An admission causes a launch. The launch establishes a workload identity. That identity requests egress. An approval changes the plan. A checkpoint captures the resulting state. A restore creates a child. A later policy change may invalidate the assumptions behind several of those actions.
flowchart LR
A[Admission] --> L[Launch]
L --> E[Egress approval]
L --> C[Checkpoint]
C --> R[Restore as child]
E --> O[External operation]
X[Policy change] -. invalidates .-> E
X -. review impact .-> O
Once those links are explicit, the system can walk backward from an operation to the decisions that made it possible, or forward from a revoked exception to the runs that may need review.
That is much closer to the question an operator actually asks: What else became true because this was allowed?
The actor has to be equally concrete. “Approved by admin” is not useful provenance. A durable record should distinguish a person, an on-call role, an automated service, a signing key, or a control-plane principal. This does not require storing personal details everywhere; it requires enough identity to tell one source of authority from another.
“Why” is sensitive data
The rationale field created a problem I did not want to ignore.
People paste secrets into text boxes. They paste customer names, internal URLs, credentials, incident transcripts, and entire conversations. A durable, chain-connected record is one of the worst places to discover that habit later.
So “record why” cannot mean “record everything.”
The provenance path needs bounds and hygiene: validate known secret patterns, prefer ticket references over copied incident content, avoid embedding raw policy payloads, keep rationale short enough to review, and define retention or encryption where required.
The audit chain is intentionally hard to rewrite. That makes data minimization part of the design, not a cleanup task for later.
Receipts are useful, but they are still receipts
MVM also emits execution receipts—compact summaries that are convenient for users and external systems.
A receipt is evidence about what happened. It is not the durability boundary that decides whether the action was authorized.
If a convenience receipt fails to render after a correctly admitted launch, the launch does not become retroactively unauthorized. If the required chain-signed append fails before the system is allowed to proceed, that may be a different matter.
Separating those failure modes keeps presentation code from becoming an accidental security dependency.
The same principle applies to export formats. MVM can translate its internal provenance model into PROV-O/RDF or a simpler JSON representation, and artifact-specific integrations may use SPDX, in-toto, or C2PA. Those standards are useful interfaces. They should not dictate the runtime’s internal truth model.
The model should describe the decisions MVM actually makes. Exporters can translate it for the systems that need to consume it.
The review changed once the model existed
Before provenance was typed, a review of a new security-relevant path often ended with “add a log.”
Once decisions had a schema, the questions became sharper:
- Which principal is recorded as the actor?
- What rationale is required, and what data is forbidden there?
- Which admission or policy event caused this decision?
- Which artifact digests bind it?
- Is the authoritative event chain-signed?
- Can the search index be rebuilt?
- What downstream decisions will point back to it?
Those are questions the code can answer, not just the documentation.
Reconstruct one decision
Pick one recent launch and try to answer, from stored evidence alone:
- what ran?
- who authorized it?
- why was it allowed?
- which policy and artifacts did it depend on?
- what later work inherited the decision?
If the operator has to open five unrelated logs and guess how they connect, the system has records but not yet a decision history.
For any MVM workload, I want a future operator to be able to reconstruct a coherent story: which immutable artifacts ran, which signed plan admitted them, who exercised the authorizing key, why the action was allowed, what network and service decisions followed, which child or checkpoint came from it, and which later policy change superseded the original assumptions.
A pile of logs may contain all of those facts and still make the operator assemble them by hand.
Provenance is the structure that turns the facts into a decision history.
Logging helps me see the system. Audit helps me prove what the system recorded. Provenance helps me understand why one decision led to the next. MVM needs all three, and the explanation must remain an explanation rather than quietly becoming a new source of authority.