A 1 GiB Filesystem Cost 3.9 GiB of RAM

A 1 GiB root filesystem was costing MVM almost 4 GiB of host memory to build.

That number looked ridiculous because it was ridiculous.

The ext4 builder was pure Rust and deterministic. It did not shell out to filesystem tools. It walked a source tree, planned inodes and blocks, and emitted an image in memory.

The code was clean enough that the peak did not look suspicious.

The ownership model was the problem.

Where the copies came from

The input walk produced a Vec<Node>.

Each file node owned its bytes:

enum Node {
    Dir { path: String, mode: u32 },
    File { path: String, mode: u32, data: Vec<u8> },
    Symlink { path: String, target: String },
}

The image planner borrowed the node list.

Because it only had &[Node], it could not move a file’s Vec<u8> into the plan. It cloned the bytes into a second structure.

Then the emitter built the final image.

At peak, the process held roughly this:

flowchart LR
    S[Source walk<br/>file bytes] -->|clone| P[Layout plan<br/>same file bytes]
    P --> I[Ext4 image buffer]
    S -. still alive .-> I

The rough memory model was:

$$M_{\text{peak}} \approx M_{\text{walk}} + M_{\text{plan}} + M_{\text{image}} + M_{\text{overhead}}$$

For large trees, both the walk and the plan contained almost the full payload.

That is how a 1 GiB input reached 3.8 to 4.1 times its size in resident memory.

The API was telling the implementation to clone

This was not a local clone() mistake that could be removed in isolation.

The public API encoded the problem:

pub fn build_image(nodes: &[Node]) -> Result<Vec<u8>, Ext4Error>

Borrowing said: the caller keeps ownership, and the builder may inspect the nodes.

The implementation needed a stronger contract: the builder consumes the nodes and is allowed to take their storage apart.

pub fn build_image(nodes: Vec<Node>) -> Result<Vec<u8>, Ext4Error>

That one signature change forced every caller to make a decision.

Did it really need the nodes afterward? In this case, no. The node list existed to build one image.

Once the builder owned the vector, file bytes could move into the plan instead of being duplicated.

Moving the bytes broke the second pass

There was a catch.

The original builder walked the node list more than once.

One pass assigned inode numbers and built planned inodes. A later pass revisited the original nodes to wire parent and child directory entries.

After moving each node’s payload into the plan, the original list was gone.

That required a small redesign.

During the consuming pass, the builder now records the structural information needed later:

struct ChildEdge {
    parent: u32,
    name: String,
    child: u32,
    file_type: u8,
    path: String,
}

The payload moves into the planned inode. The directory relationship moves into a compact edge record.

flowchart TB
    N[Owned Node] -->|move file bytes| P[Planned inode]
    N -->|record relationship| E[ChildEdge]
    P --> B[Block layout]
    E --> D[Directory entries]
    B --> O[Ext4 image]
    D --> O

This is a common systems pattern: when you consume the rich source representation, preserve only the metadata future phases actually need.

The later phase should not require the whole original object graph merely because it was convenient.

Normalize once

The ownership change exposed another inefficiency.

Paths were normalized multiple times across sorting, inode assignment, and parent lookup.

The new pipeline normalizes once up front and carries the result beside the owned node:

Vec<(String, Node)>

The vector is sorted by the normalized path. Later phases use the stored value.

This did not dominate memory, but it removed repeated work and made the control flow easier to reason about.

The same edit that eliminated a payload copy also eliminated two redundant normalization passes.

That is usually a sign the data shape is getting closer to the job.

The numbers

On the same host and inputs, peak resident memory changed like this:

Source tree	Before	After
256 MiB	4.12×	2.67×
512 MiB	4.10×	2.64×
1 GiB	3.83×	2.46×

For the 1 GiB case:

before: 3926 MiB
after:  2522 MiB

That is a large improvement.

It is not magic.

The remaining memory has a straightforward explanation:

• the source walk still owns the input bytes before they move
• the output image itself must exist
• metadata, allocator structures, and temporary state add overhead

The image is built in memory, so peak memory cannot approach zero-copy streaming without a larger architectural change.

I prefer that honest explanation to pretending the remaining 2.46× is solved.

Why not pre-size everything?

After a result like this, it is tempting to keep optimizing.

The image buffer could be pre-sized more aggressively. The planner could use tighter structures. The walk could stream. The emitter could write sparse extents directly to a file.

Some of those ideas may be good.

They were not automatically good enough to justify the API and complexity cost.

The remaining growth headroom in the image buffer was small compared with the unavoidable walk plus image. Pre-sizing every case would add code around an effect that was no longer dominant.

The important optimization was removing the full duplicate payload.

After that, the next step should come from a profile, not momentum.

Ownership is an architectural tool

Rust’s ownership system is often described as a memory-safety feature.

Here it was a performance design tool.

The difference between borrowing and consuming determined whether the implementation could reuse the caller’s allocation.

These signatures describe different pipelines:

fn plan(nodes: &[Node]) -> Plan
fn plan(nodes: Vec<Node>) -> Plan

The first says the plan must coexist with the source.

The second allows the plan to become the source.

That can change peak memory by gigabytes.

There was a behavioral change

The new pipeline normalizes paths as it consumes the input list.

That means an invalid path may be reported in input order rather than after a later sorted pass.

The refusal is the same. The order of multiple possible errors can differ.

This is minor, but it is worth documenting.

Performance refactors often change when an error is discovered. Pretending they are always behavior-free makes review harder.

The capacity limit was not a memory limit

The builder already had a structural capacity fallback around very large ext4 layouts.

That did not protect the host from memory exhaustion.

A theoretical 16 TiB filesystem limit says nothing useful about whether a 4 GiB process can build the image in memory.

This is another measurement mistake I see often: a format limit gets mistaken for a resource limit.

The relevant bound is peak working set under realistic inputs.

The part I would do differently

I would have made the ownership contract explicit earlier.

The node list was an intermediate representation with one consumer. Borrowing it felt flexible, but that flexibility had a full-tree memory cost.

For a single-use build pipeline, consuming APIs are often the honest APIs.

They make the transfer of responsibility visible.

The fix was not “use fewer clones.”

The fix was to stop designing the planner as if the source representation needed to survive it.

---

Next: Delete Everything—Except the Ability to Prove Who You Are