Zero-copy scene caches with memmap2: instant loads, no RSS bloat

Why memory maps fix your “double RSS” load problem

Reading scene assets into heap buffers leaves the kernel’s page cache holding the same bytes. That’s two residents of identical data: page cache plus heap. With a memory map you address the cached pages directly, so “loading” collapses to pointer arithmetic and occasional page faults. You get near-instant open, stable RSS, and eviction handled by the OS.

How do you load large scene assets instantly without doubling memory?

Map the asset pack with memmap2 and parse structures in place instead of copying them into Vecs. Keep a compact index of offsets and lengths per asset, have the packer enforce alignment, and expose typed views via bytemuck/zerocopy. Share one Arc across threads. The load path becomes a tiny index read and a borrowed slice.

Background: OS page cache and RSS gotchas

• read() fills your heap with file data while the same bytes typically remain in the page cache. RSS grows and warm-starts degrade.
• mmap() inserts file pages into your address space. Touching them faults pages in as needed; you don’t allocate heap for the payload.
• With read-only mappings and immutable packs, pages are shared across processes and you avoid extra copies.

A zero-copy scene pack layout

Use a single file “pack” with a fixed header, a tight index, and payload blobs placed with alignment suitable for zero-copy typed views.

• Header: magic, version, entry count
• Entry: offset (u64), length (u32), kind (u16), align (u16), name_hash (u64)
• Payloads: blobs aligned to entry.align (e.g., 16 for f32x4)

This lets you:

• Compute &[u8] slices by offset/length without secondary parsing.
• Cast slices to typed views when alignment matches (e.g., &[f32], &[u16]).

Rust implementation with memmap2

Install deps:

cargo add memmap2 bytemuck anyhow thiserror

Pack structs and a reader that uses zero-copy views:

use std::{fs::File, mem::size_of, path::Path, sync::Arc};
use anyhow::{bail, Context, Result};
use bytemuck::{bytes_of, cast_slice, Pod, Zeroable};
use memmap2::{Mmap, MmapOptions};

#[repr(C)]
#[derive(Clone, Copy, Debug, Pod, Zeroable)]
struct PackHeader {
    magic: [u8; 4],   // b"SCN\0"
    version: u32,     // 1
    count: u32,       // number of entries
}

#[repr(C)]
#[derive(Clone, Copy, Debug, Pod, Zeroable)]
struct PackEntry {
    offset: u64,      // byte offset into file where payload starts
    len: u32,         // payload length in bytes
    kind: u16,        // app-specific (e.g., 1=positions, 2=uvs, 3=index, 100=texture)
    align: u16,       // required alignment of payload
    name_hash: u64,   // stable id or hashed path
}

#[derive(Clone)]
pub struct ScenePack {
    mmap: Arc<Mmap>,
    entries: Arc<[PackEntry]>,
}

impl ScenePack {
    pub fn open(path: impl AsRef<Path>) -> Result<Self> {
        let file = File::open(path)?;
        let mmap = unsafe { MmapOptions::new().map(&file)? };
        if mmap.len() < size_of::<PackHeader>() { bail!("pack too small"); }
        let (hdr_bytes, rest) = mmap.split_at(size_of::<PackHeader>());
        let hdr: &PackHeader = bytemuck::from_bytes(hdr_bytes);
        if &hdr.magic != b"SCN\0" { bail!("bad magic"); }
        if hdr.version != 1 { bail!("unsupported version"); }
        let need = hdr.count as usize * size_of::<PackEntry>();
        if rest.len() < need { bail!("truncated entries"); }
        let entries: &[PackEntry] = cast_slice(&rest[..need]);
        Ok(ScenePack { mmap: Arc::new(mmap), entries: Arc::from(entries) })
    }

    pub fn entry_by_hash(&self, name_hash: u64) -> Option<&PackEntry> {
        self.entries.iter().find(|e| e.name_hash == name_hash)
    }

    pub fn bytes(&self, e: &PackEntry) -> Result<&[u8]> {
        let start = e.offset as usize;
        let end = start.checked_add(e.len as usize).context("overflow")?;
        if end > self.mmap.len() { bail!("slice OOB"); }
        let slice = &self.mmap[start..end];
        // cheap alignment assertion (helpful during bring-up)
        if (slice.as_ptr() as usize) % (e.align.max(1) as usize) != 0 {
            bail!("payload not properly aligned: required {}", e.align);
        }
        Ok(slice)
    }

    // Common typed views
    pub fn as_u16(&self, e: &PackEntry) -> Result<&[u16]> {
        let b = self.bytes(e)?;
        if b.len() % 2 != 0 { bail!("len not multiple of u16"); }
        Ok(bytemuck::try_cast_slice(b)?)
    }

    pub fn as_f32(&self, e: &PackEntry) -> Result<&[f32]> {
        let b = self.bytes(e)?;
        if b.len() % 4 != 0 { bail!("len not multiple of f32"); }
        Ok(bytemuck::try_cast_slice(b)?)
    }
}

Notes:

• Mapping is unsafe when the file can change under you; treat packs as immutable and publish updates by rename.
• The packer ensures alignment; the reader verifies it.
• Arc makes cross-thread sharing trivial. You pass around borrowed slices, not copies.

Packer sketch: aligned payloads, compact index

A minimal packer that aligns payloads and writes header+entries+blobs:

use std::{fs::File, io::{Seek, SeekFrom, Write}, path::Path};

fn align_to(x: u64, a: u64) -> u64 { ((x + a - 1) / a) * a }

pub fn write_pack(out: impl AsRef<Path>, mut blobs: Vec<(u64 /*name_hash*/, u16 /*kind*/, u16 /*align*/, Vec<u8>)>) -> anyhow::Result<()> {
    blobs.sort_by_key(|b| b.2.max(1)); // optional: group by alignment
    let mut file = File::create(out)?;

    let header = PackHeader { magic: *b"SCN\0", version: 1, count: blobs.len() as u32 };
    file.write_all(bytes_of(&header))?;
    let entries_pos = file.seek(SeekFrom::Current(0))?;
    // reserve space for entries
    let zero = PackEntry::zeroed();
    for _ in 0..blobs.len() { file.write_all(bytes_of(&zero))?; }

    let mut entries = Vec::with_capacity(blobs.len());
    let mut cursor = file.seek(SeekFrom::Current(0))?;

    for (name_hash, kind, align, data) in &blobs {
        let a = (*align).max(1) as u64;
        let aligned = align_to(cursor, a);
        if aligned > cursor {
            let pad = (aligned - cursor) as usize;
            file.write_all(&vec![0u8; pad])?;
            cursor = aligned;
        }
        file.write_all(data)?;
        let entry = PackEntry { offset: cursor, len: data.len() as u32, kind: *kind, align: *align, name_hash: *name_hash };
        cursor += data.len() as u64;
        entries.push(entry);
    }

    // Write entries
    file.seek(SeekFrom::Start(entries_pos))?;
    for e in &entries { file.write_all(bytemuck::bytes_of(e))?; }
    Ok(())
}

With this layout, “load” is: open, mmap, cast header and entry table, then borrow slices for each asset. No per-asset syscalls and no heap copies.

When should you choose mmap over read() for asset caching?

Use mmap for large, read-mostly assets with repeated or random access. It avoids duplicating storage in heap and page cache while leaning on the kernel’s readahead. Stick with read() for tiny files, streaming sources, or when you must own buffers independent of the file handle.

Quick comparison

Scenario	mmap	read()
Large, random-access meshes/textures	Excellent	Poor (many small reads/copies)
Instant warm-start (data cached)	Excellent	Good but doubles memory
Tiny config files	Overkill	Fine
Shared across processes	Great (shared pages)	Not shared
Needs mutation	Risky (COW/consistency)	Safer (explicit buffers)

Concurrency, eviction, and partial prefetch

• Sharing: memmap2’s Mmap is Send + Sync. Wrap it in Arc and return typed views; the cost is just slice borrows.
• Eviction: the OS evicts cold pages automatically. Dropping the Arc unmaps your VMA so the kernel can release pages earlier.
• Prefetch: on Linux/macOS, madvise(WILLNEED)/posix_fadvise can warm ranges. In Rust, call libc::madvise on mmap.as_ptr() and length if you need it. Keep it optional; readahead is usually sufficient.

Example: scene graph that hands out typed buffers

Suppose the app wants position buffers (f32 triplets) and index buffers (u16). The cache maps once and returns typed slices by hash.

use std::collections::HashMap;
use anyhow::{Result, bail};

#[derive(Clone)]
pub struct SceneCache {
    pack: ScenePack,
    // maps logical names to entry indices
    lut: HashMap<u64, usize>,
}

impl SceneCache {
    pub fn from_pack(pack: ScenePack) -> Self {
        let lut = pack.entries.iter().enumerate().map(|(i, e)| (e.name_hash, i)).collect();
        Self { pack, lut }
    }

    pub fn positions(&self, name_hash: u64) -> Result<&[f32]> {
        let idx = *self.lut.get(&name_hash).ok_or_else(|| anyhow::anyhow!("missing"))?;
        let e = &self.pack.entries[idx];
        if e.kind != 1 { bail!("not positions"); }
        let f = self.pack.as_f32(e)?;
        if f.len() % 3 != 0 { bail!("positions not multiple of 3"); }
        Ok(f)
    }

    pub fn indices(&self, name_hash: u64) -> Result<&[u16]> {
        let idx = *self.lut.get(&name_hash).ok_or_else(|| anyhow::anyhow!("missing"))?;
        let e = &self.pack.entries[idx];
        if e.kind != 3 { bail!("not indices"); }
        self.pack.as_u16(e)
    }
}

Cold starts fault in what you actually touch; warm starts reuse cached pages without inflating RSS.

Does mmap still help if my assets are compressed?

Yes. Map a seekable, chunk-compressed container (e.g., zstd-seekable) and decompress on demand into small, reusable scratch buffers. You avoid building one giant heap buffer. Keep chunks aligned and bounded (roughly 128–512 KiB) to balance random access against CPU.

A note on glTF and friends

For glTF/GLB, the binary buffer is already contiguous. If you can pack buffers in place and hold alignment, you can cast straight to typed slices. Skip temporary Vecs; borrow ranges from the mapped buffer and validate bounds with the accessors.

Production notes: updates, portability, and safety

• Immutable updates: never modify a mapped file in place. Write a new pack, fsync, then rename over the old path. Keep the old map alive until users drop it; the OS holds the inode.
• Windows sharing: std::fs::File uses sharing flags that usually allow concurrent read/write access. Still avoid in-place writes; prefer replace-on-rename to keep consistency.
• 32-bit processes: address space is tight. Very large maps can fail; keep packs under 2–3 GiB or ship 64-bit.
• Bounds and lifetimes: validate offsets/lengths and tie entries to the Arc<Mmap) owner (as in ScenePack). Don’t leak &'static references.
• Security: treat user-supplied packs as untrusted. Check magic, version, counts, alignment, and all arithmetic.

Micro-optimizations that matter (sometimes)

• Group entries by alignment or by locality to cut TLB misses.
• Place buffers that are read together next to each other to help readahead.
• Consider huge pages only with measurements; random access rarely benefits.
• For massive level swaps, madvise(DONTNEED) on ranges you’re done with. Measure first; the kernel usually makes good choices.

Debugging and measuring

• RSS/VSZ: watch with top/htop. With mmap, RSS should stay steady during “load.”
• mincore/fincore: see which pages are resident.
• perf/ETW: track page faults and I/O.
• Flamegraphs: verify memcpy isn’t on your hot path.

The bottom line

Memory mapping turns asset “loading” into metadata handling. Validate a small header, cast the index, and borrow slices from a single Arc. You avoid redundant copies, keep RSS flat, and get fast warm-starts. Design the pack for alignment and immutability, then let the kernel work for you.