A reproducible RAM-regression workflow in Rust with dhat + heaptrack

Why RAM regressions are common in media pipelines

Media workloads lean hard on the heap. Big frame buffers, codecs that change behavior per stream, and multi-stage paths (decode → transform → encode) add up. Then one “small” tweak—buffer growth policy, a stray clone(), a channel’s backpressure—shifts the peak RSS by hundreds of megabytes and nothing in perf counters screams about it.

Here’s a workflow that has held up in CI and on laptops:

• Use dhat to enforce a peak-heap budget in tests and fail early.
• Use heaptrack in CI to capture allocation-heavy call stacks and quantify where the bytes moved.

You get a drop-in Rust harness and a bit of shell to keep baselines, diff them, and bisect when needed.

Tooling split: dhat vs heaptrack

• dhat (crate: dhat) swaps in a test-time global allocator and records heap stats with minimal noise. It’s built for budget-style tests and reproducible, per-test runs. Output is JSON and HeapStats is available for direct assertions.
• heaptrack (CLI/GUI) interposes on allocations and keeps full backtraces with sizes and lifetimes. It’s heavier but answers “who allocated this 300MB vec?” with a call tree. Perfect for CI artifacts and deep dives.

Use both. dhat blocks regressions, heaptrack explains them.

Make the workload deterministic

Before chasing deltas, cut variability to the bone:

• Pin one Rust toolchain and a crate lockfile.
• Lock CPU features and disable auto-tuning: set RUSTFLAGS="-C target-cpu=x86-64-v3" (or your floor), not native, in CI.
• Prefer the system allocator for heaptrack runs. Only enable DhatAlloc in tests.
• Run single-threaded or with a fixed thread count: RUST_TEST_THREADS=1, RAYON_NUM_THREADS=1 (if you use Rayon).
• Keep media assets stable in-repo or fetch them with pinned checksums.
• Handle caches consistently every run (e.g., a short warmup decode) or turn them off.

Wire in dhat for budgeted tests

The recipe: enable DhatAlloc only in tests, run a deterministic media workload, read HeapStats, then assert the budget.

Cargo.toml (relevant bits):

[dev-dependencies]
dhat = "0.3"
anyhow = "1"
# your media stack, examples:
# ffmpeg-next = "6"
# symphonia = { version = "0.5", features = ["aac", "mp3"] }

lib.rs (or the crate root):

#![cfg_attr(test, allow(dead_code))]

#[cfg(all(test, feature = "dhat-heap"))]
use dhat::{Dhat, DhatAlloc, HeapStats};

#[cfg(all(test, feature = "dhat-heap"))]
#[global_allocator]
static ALLOC: DhatAlloc = DhatAlloc;

#[cfg(not(all(test, feature = "dhat-heap")))]
#[global_allocator]
static ALLOC: std::alloc::System = std::alloc::System;

// A sketch of a deterministic media work unit. Replace with your real pipeline.
pub fn transcode_in_place(input_path: &str, target_bitrate: u32) -> anyhow::Result<Vec<u8>> {
    // Do real work: open input, decode frames, resample, encode, and collect output bytes.
    // Be careful with buffering: use bounded queues and avoid unbounded Vec::reserve growth.
    // For illustration we simulate dynamic growth patterns often seen in media code.
    let mut out = Vec::with_capacity(16 * 1024);
    let chunks = std::fs::read(input_path)?; // deterministic blob

    let step = (target_bitrate as usize).max(64);
    for window in chunks.chunks(step) {
        // Pretend: transform + encode window
        let mut buf = window.to_vec();
        // e.g., format conversion might temporarily double memory
        buf.extend_from_slice(&[0u8; 128]);
        out.extend_from_slice(&buf);
    }

    // Simulate container mux overhead
    out.splice(0..0, [0x52, 0x55, 0x53, 0x54]); // 'RUST'
    Ok(out)
}

#[cfg(test)]
mod tests {
    use super::*;

    // Budget tuned for a known input asset of ~5 MiB.
    // Keep a baseline in the repo and update intentionally via a PR with justification.
    const PEAK_HEAP_BUDGET_BYTES: usize = 64 * 1024 * 1024; // 64 MiB

    #[test]
    #[cfg(feature = "dhat-heap")]
    fn memory_budget_transcode() {
        // Start dhat and ensure single-thread determinism for the test.
        let _dhat = Dhat::start_heap_profiling();
        std::env::set_var("RAYON_NUM_THREADS", "1");

        let out = transcode_in_place("tests/assets/sample.aac", 192_000).unwrap();
        assert!(out.len() > 0);

        let stats = HeapStats::get();
        eprintln!(
            "dhat: total_bytes={}, max_bytes={}, curr_bytes={}",
            stats.total_bytes, stats.max_bytes, stats.curr_bytes
        );

        assert!(
            stats.max_bytes <= PEAK_HEAP_BUDGET_BYTES,
            "peak heap {} > budget {}",
            stats.max_bytes,
            PEAK_HEAP_BUDGET_BYTES
        );
    }
}

How to run:

# Pin rustc and dependencies; CI should use the same toolchain.
rustup override set 1.77.2

# Only tests compiled with the feature use dhat.
cargo test -F dhat-heap -- --nocapture --test-threads=1

Notes:

• DhatAlloc must be the sole #[global_allocator] in the build. The cfg gating above guarantees it.
• dhat writes dhat-heap.json by default; save it as an artifact if you want historical traces.
• Use distinct budgets per workload shape (audio-only vs 4K video), or assert a delta versus a checked-in baseline.

Heaptrack in CI for call stacks and diffs

When the budget fails, heaptrack is the flashlight. It shows allocation sites and lifetimes.

Install and run (Ubuntu/Debian):

sudo apt-get update && sudo apt-get install -y heaptrack

# Always profile a release build with debuginfo for useful stacks.
RUSTFLAGS="-C debuginfo=1 -C target-cpu=x86-64-v3" cargo build --release

mkdir -p artifacts
# Record; `-o` sets the output prefix. The program and args follow `--`.
heaptrack -o artifacts/trace -- \
  ./target/release/your-bin \
  --input tests/assets/sample.aac --bitrate 192000

# Print a textual summary for CI logs and machine parsing.
heaptrack_print artifacts/trace.zst | tee artifacts/trace.txt

A minimal CI gate that compares peak heap between main and PR:

#!/usr/bin/env bash
set -euo pipefail

parse_peak_bytes() {
  # Extract a number of bytes from heaptrack_print output lines like:
  # "peak heap memory consumption: 64.23 MiB"
  awk '/peak heap memory/{print $(NF-1), $NF}' | \
  python3 - "$@" <<'PY'
import sys
val, unit = sys.stdin.read().strip().split()
num = float(val)
mult = {
  'B': 1,
  'KiB': 1024,
  'MiB': 1024**2,
  'GiB': 1024**3,
}.get(unit, 1)
print(int(num * mult))
PY
}

profile_once() {
  local out_prefix=$1
  shift
  heaptrack -o "${out_prefix}" -- "$@" >/dev/null 2>&1
  heaptrack_print "${out_prefix}.zst" > "${out_prefix}.txt"
  parse_peak_bytes < "${out_prefix}.txt"
}

# Build once for determinism
RUSTFLAGS="-C debuginfo=1 -C target-cpu=x86-64-v3" cargo build --release

# Baseline command is the same across runs; load it or regenerate from main.
BASELINE_BYTES=$(profile_once artifacts/baseline ./target/release/your-bin --input tests/assets/sample.aac --bitrate 192000)
PR_BYTES=$(profile_once artifacts/pr ./target/release/your-bin --input tests/assets/sample.aac --bitrate 192000)

DELTA=$(( PR_BYTES - BASELINE_BYTES ))
ABS_DELTA=${DELTA#-}

BUDGET_PCT=${BUDGET_PCT:-5}
THRESH=$(( BASELINE_BYTES * BUDGET_PCT / 100 ))

printf "baseline=%d bytes, pr=%d bytes, delta=%+d bytes, threshold=%d bytes\n" \
  "$BASELINE_BYTES" "$PR_BYTES" "$DELTA" "$THRESH"

if (( ABS_DELTA > THRESH )); then
  echo "ERROR: peak heap moved by > ${BUDGET_PCT}%"
  exit 1
fi

Hook this into CI once the PR artifact builds. Keep the baseline from main as an artifact or regenerate by checking out origin/main in the workflow.

Baselines, budgets, and bisect

• Keep a small JSON next to your test assets with dhat.max_bytes and heaptrack.peak_heap_bytes measured on main.
• Fail PRs if either metric shifts by more than X% or Y bytes.
• When something regresses, run git bisect run with the CI script locally:

git bisect start
git bisect bad HEAD
git bisect good origin/main

git bisect run bash -lc 'cargo build --release >/dev/null && ci/mem_gate.sh'

That pins the first commit where the peak crossed your line.

Allocators and reproducibility gotchas

• System allocator for heaptrack: prefer std::alloc::System in the release build you profile. If you normally ship with mimalloc or jemalloc, build a profiling variant with the system allocator to maximize interposition accuracy.
• Only one global allocator: gate DhatAlloc behind a test-only feature. Mixing allocators in one binary is undefined behavior.
• Debug info: build release with -C debuginfo=1 to keep frame pointers for heaptrack without gutting optimizations.
• Threads: fix RAYON_NUM_THREADS=1 or another constant. Unbounded parallelism changes interleavings and peaks.
• Inputs: small but representative assets work best. A 5–20 MiB clip drives buffering paths without long runtimes.

Interpreting results

• dhat:
  • max_bytes is the anchor for budgets. In media code it approximates “frame queue high-water mark plus working buffers.”
  • If total_bytes swings while max_bytes holds steady, you have churn, and there may be zero-copy wins.
• heaptrack:
  • Sort by cumulative size to find hot allocation sites. Usual suspects: Vec::resize, image::ImageBuffer::from_raw, bytes::BytesMut::reserve.
  • Compare “allocated but not freed at exit” to spot leaks. Watch for Arc cycles or missed drop during teardown.
  • Cross-check peaks with dhat. If they differ a lot, ensure both profiled the same binary and allocator.

What a good PR looks like

• Updates the dhat baseline or explains any budget increase (e.g., enabling B-frames doubles buffering).
• Attaches a heaptrack summary to the PR for review.
• Leaves allocator and thread settings untouched unless the PR intends to change them.
• Records trade-offs (e.g., +8 MiB for a 12% throughput gain) and adjusts budget constants.

Appendix: pinning versions for repeatability

• rust-toolchain.toml:

[toolchain]
channel = "1.77.2"
components = ["rustc", "cargo"]

• CI packages: heaptrack >= 1.4 recommended. dhat crate pinned via Cargo.lock.

With this setup, a memory regression shows up as a normal test failure that you can explain and bisect. dhat provides the guardrail, heaptrack provides the microscope, and together they keep your Rust media app in bounds.