Memory-Mapped Files (`MemoryMappedFile`) for Gigabyte Dataset Processing

1Concept

`MemoryMappedFile` maps large files (10GB+) directly into the process's virtual memory address space. The OS kernel pages bytes in and out of RAM on demand, bypassing managed heap limitations.

2Architecture Diagram

[ 50GB Database File on NVMe SSD ]
                    │
   OS Virtual Memory Mapping (Paging)
                    │
[ 64-Bit Process Virtual Address Space ] ──> Direct Pointer / Span Access!

3Code Example

using System;
using System.IO.MemoryMappedFiles;

public class MemoryMappedFileDemo
{
    public static void Main()
    {
        // Create in-memory memory-mapped file for IPC (Inter-Process Communication)
        using var mmf = MemoryMappedFile.CreateNew("SharedMemoryChannel", 1024);
        using var accessor = mmf.CreateViewAccessor();

        accessor.Write(0, 42); // Write integer at byte offset 0
        int readBack = accessor.ReadInt32(0);

        Console.WriteLine($"Memory-Mapped Value Read Back: {readBack}");
    }
}

4Expected Output

Memory-Mapped Value Read Back: 42

5Key Takeaways

• ✓Enables ultra-fast IPC sharing between distinct OS processes without network sockets.
• ✓Processes multi-gigabyte log and database files without reading them entirely into RAM.
• ✓OS handles caching and page faults automatically.