Oak Oil: Cargo Plans, We Execute — Lean, Fast Rust Builds That Stay Byte-Identical to Stock Cargo

My 4 TB Mac was down to its last 2% of disk. Not because of models or video: because of target/ directories. A scan found 469 GiB in 142 Cargo target dirs, and 177 GiB of it was byte-for-byte identical to files already stored somewhere else on the same disk.

Rust's answer to this is cargo clean, which throws away the good with the bad and costs you a full rebuild. I wanted the opposite: keep everything Cargo will use again, store it once, drop the rest, and never be slower than Cargo while doing it.

So we built Oak Oil: Cargo plans the build, Oak Oil executes it, and the output is byte-identical to stock Cargo.

It is open source at github.com/oakoliver/oak-oil, and cargo install cargo-oil gets you the cargo oil subcommand.

I – Where 469 GiB Goes

The first thing Oak Oil did was not build anything. It measured. cargo oil measure walks every target dir on the machine and attributes each byte to a kind of waste:

pie showData
    title Bytes in 142 Cargo target dirs on one Mac (GiB)
    "Incremental caches" : 185.4
    "Debug-info objects (deps/*.o)" : 108.6
    "Binaries, dylibs, tests" : 60.5
    "rlib" : 40.9
    "rmeta" : 29.4
    "Build scripts, OUT_DIR" : 9.2
    "Uplifted copies" : 8.9
    "Other" : 26.0

Three findings shaped everything that followed:

177 GiB were identical copies. The same serde, tokio and syn artifacts, compiled again in every project and every worktree. Parallel agent worktrees were the worst case: eight checkouts of one workspace held a separate ~30 GiB target dir each.
185 GiB were incremental caches. All of it belonged to the developers' own crates (Cargo only compiles those incrementally), which a cross-project cache cannot share anyway.
80% of the bytes had been written in the last 7 days. Cleaning by age, the usual advice, frees little. The waste is duplication and superseded versions, not old files.

Rust-specific design explains why this happens. There is no stable ABI, so a dependency cannot be shipped prebuilt and shared like a C library. Every target dir compiles and stores its own copy. Every change of feature set, profile or toolchain produces a new unit hash and leaves the old artifact behind. On macOS, split-debuginfo = "unpacked" keeps every codegen unit's .o file next to the binary that points at it, and incremental builds leave stale ones behind: one deps/ folder we looked at held 19,886 files, 19,405 of them objects.

None of this is a bug in Cargo. It is the price of fast incremental builds and monomorphized generics. But the price is paid in copies, and copies are something a build tool can remove.

II – Borrowing Bun's Architecture

Bun rewrote its runtime in Rust this year, and its build script contains a sentence that reframed the problem for me. In scripts/build/rust.ts: cargo plans, ninja executes. Bun asks Cargo for its unit graph, then runs every crate as its own rustc step in its own scheduler, beside the C++.

A rusted press restored with one drop of oil, while duplicate type blocks go to the furnace and a cabinet keeps one of each

That split is exactly what a disk- and time-saving tool needs. Cargo is very good at deciding what to build: features, profiles, build scripts, the lockfile. It does not need to be replaced. What can be replaced is the part that runs rustc and owns the files. So Oak Oil has two modes:

flowchart TD
    C["cargo build"] -->|"first build: RUSTC_WRAPPER"| W["cargo-oil records
every rustc call"]
    W --> P[("The plan")]
    P -->|"every later build"| E["Oak Oil executor
pipelining, jobserver,
critical path first"]
    E --> S[("Store, one copy
of each artifact")]
    E --> R["stock rustc"]

Planning runs stock Cargo once with Oak Oil as RUSTC_WRAPPER. Every rustc invocation is recorded, with its arguments, its environment and its outputs. Cargo's own JSON messages tell us every unit it considered, fresh or not, and every build-script run. The plan is only rebuilt when something that changes Cargo's decisions changes: manifests, the lockfile, configs, the toolchain, the environment, or a build script's inputs.

Executing runs that plan without Cargo. It is a small scheduler that starts a dependent as soon as its dependencies' .rmeta exists, hands rustc a jobserver so codegen threads never oversubscribe the machine, and starts the longest chain first.

III – Deciding What to Do With One Unit

Each unit goes through the same three questions, cheapest first:

flowchart TD
    A["Unit from the plan"] --> B{"Stamps unchanged?
size, mtime, inode of everything
it read and wrote"}
    B -- yes --> F["Fresh: nothing to do"]
    B -- no --> K["Content key:
rustc version, command, env,
dependency bytes"]
    K --> L{"Store has an entry whose
source hashes still match?"}
    L -- yes --> H["Restore from the store
(APFS clone, no copy)"]
    L -- no --> X["Run stock rustc"]
    X --> Q["Queue a store write
background process, efficiency cores"]

Stamps first. A unit whose inputs and outputs look exactly as they did after its last build costs a few stat calls. No file is read. A no-change build of a 174-unit workspace runs in 0.05 to 0.14 s.

Content keys second. When a stamp differs, the unit gets a key built from everything rustc will see except the sources: the rustc version, the full command line, the environment, and the bytes of every dependency it links. The store keeps, under that key, the sources rustc read last time (from rustc's own dep-info) and their hashes. If they still match, the outputs are restored as APFS clones: no copy, no extra disk. Because keys hash dependency bytes, a dependency rebuilt into identical bytes does not dirty its dependents.

Rustc last. A miss runs stock rustc with the exact recorded command. Storing the result happens later, in a detached process running at background QoS, which on Apple Silicon means the efficiency cores. No build ever waits for the store.

Library artifacts from registry and git crates are shared across every project on the machine. Linked outputs and the developer's own crates stay per project, because they contain absolute paths (more on that below).

IV – The Contract: Byte-Identical to Stock Cargo

A build tool that is fast but subtly different is worse than useless. So the rule for every change was: Oak Oil's output must be byte-identical to stock Cargo's, and stock Cargo must find the target dir fresh afterwards.

That rule needs an oracle. The oracle builds a project with stock Cargo, records the SHA-256 of every file in the target dir, then builds it with Oak Oil at the same path and compares. Then it wipes the outputs, restores them from the store, and compares again. Then it runs plain cargo build and requires zero recompiles.

The first surprise: stock Cargo is not byte-reproducible with incremental compilation on. Two clean builds at the same path differ in a dozen files, all from the project's own incrementally compiled crates. With CARGO_INCREMENTAL=0, everything matches. So the oracle runs non-incremental, where stock Cargo is its own reference.

Every bug the oracle caught was real, and almost every one was about Cargo's state rather than rustc's output:

Cargo freshness is mtime ordering. Restored files must be stamped in dependency order, or Cargo rebuilds. With --target, build scripts compile in the host profile dir and run in the target's, so they have to be paired by package, not by directory.
Cargo drops -C extra-filename for cdylib crates. Unit identity cannot rely on the hash in the file name.
Build scripts can hand rustc any environment variable through cargo:rustc-env, which env! then reads. Recording only CARGO_* broke a real workspace.
Linked outputs remember where they were built. macOS debug maps (N_OSO) hold absolute paths to .o files, so a binary from one project's store is not the binary Cargo would have built in another.
Cargo has a race of its own. For a cdylib+rlib crate, it writes one profile-level dep-info for both outputs, and the last write wins: its first line names the .dylib or the .rlib at random, in plain stock builds too. The oracle compares that file with that one token normalized.

The oracle runs on every push, in CI, on a small workspace that has one of each tricky unit. Breaking the restore on purpose makes it fail on six files.

V – Getting Out of the Way

The first version of the executor was correct and slow. On a 128-unit rebuild it reached only 0.63× the speed of Cargo. The fix was not a faster algorithm. It was removing everything that was not rustc from the path Cargo or the scheduler waits on:

A jobserver, like Cargo's, so 18 rustc processes stop spawning 18 sets of codegen threads.
Critical path first, so the longest chain of dependents starts early.
Stat-only freshness, instead of hashing every input on every build.
Outputs from rustc's artifact notices, instead of scanning a folder with 19,000 stale objects after every unit.
Store writes moved out of the build entirely, to a background process.

The same discipline fixed clean builds, which started 14% slower than Cargo because the wrapper stored every output before Cargo could move on. Moving those writes to a spool and the background drain brought a clean build to 1.01× stock.

VI – Results

Median of three runs, stock Cargo and Oak Oil alternating within each run, same machine and settings, incremental compilation on. Project C has 138 workspace crates; the edits change a library about 125 crates depend on. The clean-build row was measured again after the fix in section V.

Scenario	Stock Cargo	Oak Oil	Speedup
Clean build	10.69 s	10.60 s	1.0×
No-change build	0.15 s	0.14 s	1.1×
Comment edit	2.56 s	1.97 s	1.3×
Private code edit	3.02 s	2.79 s	1.1×
Revert the edit	3.23 s	0.58 s	5.5×
`rm -rf target`, build	10.90 s	1.66 s	6.6×
Second checkout of the same project	10.96 s	9.52 s	1.15×

xychart-beta
    title "Speedup over stock Cargo, Project C"
    x-axis ["Clean", "No-op", "Comment", "Private", "Revert", "Wipe", "Copy"]
    y-axis "× faster than stock" 0 --> 7
    bar [1.0, 1.1, 1.3, 1.1, 5.5, 6.6, 1.15]

(Wipe is rm -rf target then build; Copy is a second checkout of the same project.)

The pattern is the point. Oak Oil is never slower than Cargo, and it is much faster whenever the work has been done before: switching back to a branch, wiping a target, reverting a change. On a smaller workspace a wiped target came back 16× faster; with incremental compilation off, a revert was 300× faster, because there is nothing left to compile at all.

Disk is the other half. cargo oil clean --apply on a real 46 GiB target dir removed 9.9 GiB of superseded unit variants and old incremental sessions. It then re-ran Cargo for the same commands and checked that Cargo recompiled nothing. It decides what is live from the units Cargo itself reports, not from file ages, and it leaves other directories inside target/ alone unless asked.

It is not free, and the numbers say so. Filling the store costs CPU after a build: about 5.5 s on the efficiency cores after a clean build of Project C. Nobody waits for it, but it happens.

VII – Built in Public

A build tool runs on every developer's machine, so where the binary comes from matters as much as what it does. Every Oak Oil release is built and published by a public GitHub Actions workflow, from a tag, after the same checks as CI:

Release binaries carry a signed build provenance attestation. gh attestation verify cargo-oil-v0.1.0-aarch64-apple-darwin.tar.gz --repo oakoliver/oak-oil proves which workflow run and commit produced the file you downloaded.
The crate on crates.io is published by that same run, and records the commit it was built from. After the first release, publishing moves to crates.io trusted publishing: no long-lived token exists anywhere.
Every action in the workflows is pinned to a full commit SHA.

cargo install cargo-oil builds from the published source. cargo binstall cargo-oil downloads the attested binary instead.

VIII – What Rust Could Do Better

Some limits are not Oak Oil's to remove:

Early cutoff needs rustc. Without incremental compilation, even a comment changes .rmeta, because rustc stores source hashes and line tables in it, and the crate hash covers every function body. An interface hash that ignores private bodies would let a build tool skip every dependent of a private edit, and turn a binary's rebuild into a relink.
Absolute paths limit sharing. --remap-path-prefix can make outputs path-independent, which is the next step for sharing a workspace's own crates between worktrees.
Target dirs have no garbage collector. Cargo cleans its download caches; nothing cleans target/. Oak Oil's store needs its own GC next, and the Cargo team's work on shared build caches points the same way.

Oak Oil is macOS-only and experimental today. It has been verified on four real workspaces, not on yours. But the shape is there: Cargo keeps deciding, an executor keeps everything Cargo will use exactly once, and the output does not change by a single byte.

cargo install cargo-oil
cargo oil build
cargo oil clean          # dry run; --apply to remove, then verify

The code, the oracle and the findings are at github.com/oakoliver/oak-oil.