Part IVMemory Hierarchy

Lab --- ChampSim, DRAMsim3, and gem5 Memory Modeling

August 3, 2026·21 min read·advanced

The project of Chapter 47 built a trace-driven cache simulator that models hits, misses, and AMAT analytically. Analytic AMAT is a useful first-order tool, but it abstracts away timing details that matter for…

The project of Chapter 47 built a trace-driven cache simulator that models hits, misses, and AMAT analytically. Analytic AMAT is a useful first-order tool, but it abstracts away timing details that matter for real performance: queueing delays at the memory controller, row-buffer hits and conflicts in the DRAM, the contention between prefetchers and demand requests, and the cycle-level interaction between the core and the cache. Closing those gaps requires cycle-accurate simulation.

This lab steps up to three production simulators that the research community uses: ChampSim, DRAMsim3, and gem5. Each covers a different point in the speed-accuracy trade-off space. ChampSim is the fastest and the easiest to start with for cache and prefetcher experiments. DRAMsim3 is the canonical DRAM backend, used standalone for memory-controller studies and integrated into both ChampSim and gem5 for end-to-end memory modeling. gem5 is the full-system simulator with the highest detail and the steepest learning curve, modeling the complete memory hierarchy alongside core, ISA, and OS.

The lab is organized as three sequential exercises. Exercise 1 sets up ChampSim and runs cache and prefetcher configurations on the SPEC traces. Exercise 2 sets up DRAMsim3 and integrates it with ChampSim, then explores DDR5 versus HBM3 configurations. Exercise 3 sets up gem5 and runs a small workload in both the ClassicCache mode and the Ruby mode, comparing the results to ChampSim. The lab closes with a comparison section that frames when each simulator is the right tool.

The exercises assume basic familiarity with the C++ build flow established in Chapter 47 and with the RISC-V toolchain set up in Chapter 24. No HDL work is involved. All three simulators are pure software.

01.Setup and Installation

The lab requires three simulator codebases (ChampSim, DRAMsim3, gem5) plus their build prerequisites. ChampSim depends only on a modern C++ compiler, GNU make, and the xz library. DRAMsim3 depends on the same plus CMake. gem5 has a substantially larger dependency list: Python 3, scons, protobuf, hdf5, and the boost development headers, among others.

macOS (Homebrew)

macOS setup via Homebrew

Plain Text
# Common: compiler and build tooling brew install llvm cmake git python xz # gem5-specific dependencies brew install scons protobuf hdf5 boost brew install gperftools # optional, for profiling # Optional: graphviz for gem5's debug visualizations brew install graphviz

Verify each tool:

Verify macOS installations

Plain Text
clang++ --version # Apple clang 16+ or Homebrew clang 18+
cmake --version
scons --version # SCons 4.x
python3 --version # Python 3.12 or later
protoc --version # protobuf 3.x or 5.x

Linux (Arch as canonical)

Arch Linux setup

Plain Text
# Common
sudo pacman -S clang gcc cmake git python xz make ninja
# gem5
sudo pacman -S scons protobuf hdf5 boost
sudo pacman -S gperftools
# Optional: graphviz for gem5's debug visualizations
sudo pacman -S graphviz

For Debian or Ubuntu:

Debian/Ubuntu alternative

Plain Text
sudo apt update
sudo apt install build-essential cmake clang git \
python3 python3-pip xz-utils ninja-build scons \
libprotobuf-dev protobuf-compiler libhdf5-dev \
libboost-all-dev libgoogle-perftools-dev graphviz

For Fedora:

Fedora alternative

Plain Text
sudo dnf install gcc-c++ clang cmake git python3 xz \
ninja-build scons protobuf-devel hdf5-devel \
boost-devel gperftools-devel graphviz

Windows (ArchWSL)

The recommended path on Windows is ArchWSL, the Arch Linux distribution for WSL2. Once ArchWSL is installed, the setup is identical to the Arch Linux instructions above. All three simulators expect a POSIX environment and have not been extensively tested on native Windows.

Windows setup via ArchWSL

Plain Text
# Inside the ArchWSL terminal:
sudo pacman -S clang gcc cmake git python xz make ninja \
scons protobuf hdf5 boost gperftools graphviz

02.Exercise 1: ChampSim

ChampSim is the natural next step after the trace-driven simulator of Chapter 47. The trace format is the same (the SPEC traces from the project are exactly the format ChampSim consumes), the simulator concepts are familiar (cache geometry, replacement policy, prefetcher), and the configuration is straightforward.

Cloning and building

Clone and build ChampSim

Plain Text
git clone https://github.com/ChampSim/ChampSim.git
cd ChampSim
# Initialize the inferred configurations.
./config.sh champsim_config.json
# Build the default binary.
make

The config.sh script reads a JSON file that specifies the simulator’s structure (cores, cache levels per core, branch predictor, prefetchers, replacement policies, page-table walker). It generates C++ source files that compose the selected modules into the simulator binary. This generation step is the most distinctive feature of ChampSim’s architecture: the design choices are not runtime flags, they are compile-time selections.

The champsim_config.json that ships with the repo describes a four-wide out-of-order core with a 32 KiB L1, a 512 KiB L2, and a 2 MiB shared L3. The replacement policy at each level defaults to LRU. The prefetchers default to a "no_instr_data" combination for IL1 and DL1 and a "next_line" for L2. These defaults are good starting points.

Running a SPEC trace

Use one of the SPEC 2017 traces from Chapter 47. The invocation is straightforward.

Run perlbench on the default ChampSim config

Plain Text
./bin/champsim --warmup-instructions 200000000 \
--simulation-instructions 1000000000 \
./traces/600.perlbench_s-210B.champsimtrace.xz

The simulator runs for 200 million warmup instructions (which fill the caches and warm up the predictors) and then 1 billion measurement instructions. Wall-clock time on a modern desktop is roughly 5 to 10 minutes per billion instructions for the default config.

The output is a long block of per-cache and per-prefetcher statistics. The relevant numbers are total instructions, IPC, the per-level hit/miss counts, the per-prefetcher accuracy (fraction of prefetches that were touched by a demand request before eviction), and the DRAM request counts and average latency.

Comparing two configurations

A typical study compares two configurations on the same trace. For instance, the effect of swapping the L3 replacement policy from LRU to SHiP-PC.

Generate a SHiP-PC variant

Plain Text
# Edit champsim_config.json: change the L3 "replacement"
# field from "lru" to "ship".
./config.sh champsim_config.json
make
mv bin/champsim bin/champsim_ship
# Re-run on the same trace.
./bin/champsim_ship --warmup-instructions 200000000 \
--simulation-instructions 1000000000 \
./traces/600.perlbench_s-210B.champsimtrace.xz \
> out_ship.txt 2>&1
# Compare to the LRU baseline.
diff out_lru.txt out_ship.txt | less

The diff shows the per-level numbers shifting. The bottom-line IPC delta depends on the trace. For perlbench (a heavy working set), SHiP-PC typically produces a small IPC bump (roughly 2 to 5%). For mcf (a pointer-chasing workload that exposes replacement policy sharply), the bump is larger (5 to 10%).

The methodology is the heart of cache-and-prefetcher research: identify a hypothesis about a design change, run a controlled experiment on a set of traces, and report the geometric mean of the per-trace effect.

Adding a custom prefetcher

ChampSim’s prefetcher interface is a small C++ class with a fixed signature. Implementing a custom prefetcher is the typical first hands-on exercise.

Minimal stride prefetcher (prefetcher/my_pf/my_pf.cc)

C
// Detect a constant stride per PC; on the next access, prefetch // the next two strides. Minimal implementation; production // prefetchers track many PCs and use confidence counters. #include "champsim.h" #include "cache.h" #include <map> namespace { struct StrideEntry { uint64_t last_addr; int64_t stride; int confidence; }; std::map<CACHE*, std::map<uint64_t, StrideEntry>> table; } void CACHE::prefetcher_initialize() {} uint32_t CACHE::prefetcher_cache_operate(uint64_t addr, uint64_t ip, uint8_t cache_hit, bool useful_prefetch, uint8_t type, uint32_t metadata_in) { auto& entry = table[this][ip]; int64_t observed_stride = static_cast<int64_t>(addr) - static_cast<int64_t>(entry.last_addr); if (observed_stride == entry.stride && observed_stride != 0) { if (entry.confidence < 3) entry.confidence++; if (entry.confidence >= 2) { prefetch_line(addr + observed_stride, true, 0); prefetch_line(addr + 2 * observed_stride, true, 0); } } else { entry.stride = observed_stride; entry.confidence = 0; } entry.last_addr = addr; return metadata_in; } void CACHE::prefetcher_cycle_operate() {} void CACHE::prefetcher_final_stats() {} uint32_t CACHE::prefetcher_cache_fill(uint64_t addr, uint32_t set, uint32_t way, uint8_t prefetch, uint64_t evicted_addr, uint32_t metadata_in) { return metadata_in; }

Drop the file under prefetcher/my_pf/, point the JSON config at the new prefetcher, regenerate, rebuild, and rerun. A typical stride prefetcher of this form lands within a few percentage points of the published stride numbers on streaming SPEC traces.

03.Exercise 2: DRAMsim3

ChampSim’s stub memory controller models DRAM as a fixed-latency queue. That is fine for cache-and-prefetcher studies. Memory- controller studies, comparing DDR5 against HBM3, exploring open- page versus close-page, or evaluating refresh-management strategies, need a real DRAM model. DRAMsim3 is the canonical open-source choice.

Cloning and building

Clone and build DRAMsim3

Plain Text
git clone https://github.com/umd-memsys/DRAMsim3.git
cd DRAMsim3
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j

The build produces a standalone simulator and a shared library. The standalone simulator takes a trace of memory commands (read, write, address) and runs them through the DRAM model, producing a report on average latency, row-buffer hit rate, and bandwidth. The library (libdramsim3.so) is linked into ChampSim or gem5 as the memory backend.

Standalone DRAM trace

A trace file is plain text. Each line is one access, of the form ADDR READ|WRITE CYCLE. To run a single trace:

Run DRAMsim3 on a sample DDR5 config

Plain Text
# The DRAMsim3 repo ships several JEDEC-rate config files in
# configs/. The DDR5_8Gb_x8_3200.ini matches an 8 Gb/3200 MT/s
# device.
./dramsim3main configs/DDR5_8Gb_x8_3200.ini -t my_trace.txt

The output reports per-channel statistics. The headline numbers are the average read latency and the row-buffer hit rate. The former is the latency a single read sees on average. The latter is the fraction of accesses that hit a row already open in the row buffer. A row-buffer hit costs only the column-access strobe (CAS) latency. A row-buffer conflict costs the precharge, the row activation, and the column access (tRP + tRCD + tCAS).

For a stream-heavy trace, the row-buffer hit rate runs in the 80–95% range. For a random-access trace, it falls to single digits. The latency difference between the two regimes is striking and motivates row-buffer-aware request scheduling.

Integrating DRAMsim3 with ChampSim

ChampSim supports DRAMsim3 as a memory backend. To enable, edit the JSON config to set "physical_memory" to "dramsim3" and point at the relevant DRAM config file. After regenerating and rebuilding:

Run ChampSim with DRAMsim3 backend

Plain Text
./bin/champsim --warmup-instructions 200000000 \
--simulation-instructions 1000000000 \
--dram-config DDR5_8Gb_x8_3200.ini \
./traces/605.mcf_s-994B.champsimtrace.xz

The output now includes per-channel DRAM statistics on top of the per-cache stats. The IPC may differ from the stub-backend run by a few percent, since the cycle-accurate DRAM model has a more nuanced latency distribution.

Comparing DDR5 against HBM3

DRAMsim3 ships configurations for both DDR5 and HBM3. The two present radically different bandwidth and latency profiles.

Table 1. Headline parameters: DDR5 vs HBM3 (JEDEC defaults).

DDR5-6400 (1ch)HBM3 (1 stack)
Channels2 (sub-channel)16 (channel)
Bus width per channel32 bits64 bits
Data rate6.4 GT/s6.4 GT/s
Bandwidth per channel25.6 GB/s51.2 GB/s
Peak total bandwidth51.2 GB/s819.2 GB/s
Typical latency (random)80–100 ns80–100 ns
Capacity8–32 GB16–24 GB per stack
Use casedesktop, server mainGPU, AI accelerator

HBM3 trades capacity for bandwidth. A single HBM3 stack provides about 16 times the bandwidth of a single DDR5 channel, with comparable per-access latency (since the row-buffer mechanics are similar). The cost is that HBM3 is physically attached to the package through a silicon interposer, which makes capacity expansion difficult and limits the technology to high-end accelerators and the HBM-enabled Xeon variants studied in Chapter 46.

Sweep across DRAM configurations

A typical research study runs a single trace across several DRAM configurations and tabulates the resulting IPC. To automate, a shell script wraps the simulator invocations:

Sweep over DRAM configs

Plain Text
#!/bin/bash CONFIGS="DDR4_8Gb_x8_3200 DDR5_8Gb_x8_3200 \ DDR5_8Gb_x8_6400 HBM3_8Gb_x128_6400" for cfg in $CONFIGS; do ./bin/champsim --warmup-instructions 200000000 \ --simulation-instructions 1000000000 \ --dram-config configs/$cfg.ini \ ./traces/605.mcf_s-994B.champsimtrace.xz \ > out_$cfg.txt 2>&1 ipc=$(grep "^CPU 0 cumulative IPC" out_$cfg.txt | \ awk '{print $5}') echo "$cfg IPC=$ipc" done

For mcf (memory-bound), HBM3 shows the largest IPC improvement over DDR4 (a factor of 1.5 to 2 is typical). For perlbench (less memory-bound), the improvement is modest (a few percent). The differential is the textbook story of bandwidth’s relevance: workloads that saturate bandwidth benefit, workloads that do not see compute-bound limits before memory matters.

04.Exercise 3: gem5

gem5 is the full-system simulator. Unlike ChampSim, gem5 fetches, decodes, and executes real binaries against a real ISA. Unlike DRAMsim3, gem5 models the full memory hierarchy including the cache and coherence layers. The trade-off is a steeper learning curve and a substantial simulation cost.

Cloning and building

Clone and build gem5 for RISC-V

Plain Text
git clone https://github.com/gem5/gem5.git
cd gem5
# Build the RISC-V target. The X86 and ARM targets follow the
# same pattern with their respective build names.
scons build/RISCV/gem5.opt -j$(nproc)

The build takes 20 to 60 minutes depending on the host machine. The output binary (build/RISCV/gem5.opt) is a single executable that runs a configuration script written in Python. The Python script is the user-facing configuration, not a command-line flag set.

The ClassicCache mode

ClassicCache is the simpler of gem5’s two memory subsystems. It models a hierarchy of crossbars and caches with built-in coherence (MOESI). A minimal config script looks like:

configs/simple_cache.py

Python
import m5 from m5.objects import * system = System() system.clk_domain = SrcClockDomain() system.clk_domain.clock = '3.0GHz' system.clk_domain.voltage_domain = VoltageDomain() system.mem_mode = 'timing' system.mem_ranges = [AddrRange('512MB')] system.cpu = RiscvTimingSimpleCPU() # L1 instruction cache system.cpu.icache = Cache(size='32kB', assoc=8, tag_latency=1, data_latency=1, response_latency=1, mshrs=4, tgts_per_mshr=20) # L1 data cache system.cpu.dcache = Cache(size='32kB', assoc=8, tag_latency=1, data_latency=1, response_latency=1, mshrs=4, tgts_per_mshr=20) # Connect to the L2 via the L2 bus system.l2bus = L2XBar() system.cpu.icache.cpu_side = system.cpu.icache_port system.cpu.dcache.cpu_side = system.cpu.dcache_port system.cpu.icache.mem_side = system.l2bus.cpu_side_ports system.cpu.dcache.mem_side = system.l2bus.cpu_side_ports # L2 cache system.l2cache = Cache(size='1MB', assoc=16, tag_latency=12, data_latency=12, response_latency=2, mshrs=20, tgts_per_mshr=12) system.l2cache.cpu_side = system.l2bus.mem_side_ports system.membus = SystemXBar() system.l2cache.mem_side = system.membus.cpu_side_ports # Memory controller using gem5's native DRAM model # (a DRAMsim3 backend can be swapped in here) system.mem_ctrl = MemCtrl() system.mem_ctrl.dram = DDR4_2400_8x8() system.mem_ctrl.dram.range = system.mem_ranges[0] system.mem_ctrl.port = system.membus.mem_side_ports system.cpu.createInterruptController() system.system_port = system.membus.cpu_side_ports # The workload (a RISC-V ELF binary, compiled from a C program) process = Process() process.cmd = ['my_program'] system.cpu.workload = process system.cpu.createThreads() root = Root(full_system=False, system=system) m5.instantiate() print("Running gem5") exit_event = m5.simulate() print('Exiting @ tick %i because %s' % (m5.curTick(), exit_event.getCause()))

To run:

Run a gem5 ClassicCache simulation

Plain Text
build/RISCV/gem5.opt configs/simple_cache.py

The simulator emits a directory m5out/ with cycle-level statistics in stats.txt. Headline numbers include the CPI, the per-cache miss rate, and the memory controller’s average latency. The output is more detailed than ChampSim’s because gem5 actually executes the workload, capturing the interaction between the program’s instruction mix and the memory subsystem.

The Ruby mode

Ruby is the more detailed memory subsystem. It separates the coherence protocol (written in the SLICC domain-specific language) from the cache structures. A Ruby configuration is more verbose and the protocol selection requires recompiling gem5 with a specific protocol flag.

Build gem5 with MESI Three Level Ruby protocol

Plain Text
scons build/RISCV_MESI_Three_Level/gem5.opt \
PROTOCOL=MESI_Three_Level -j$(nproc)

The protocol flag selects which SLICC protocol is compiled into the binary. The Ruby system in the configuration script then instantiates the corresponding protocol modules.

The differences between ClassicCache and Ruby are most visible on workloads with significant coherence traffic. A single-threaded SPEC trace shows minimal difference. A multi-threaded workload with shared-data access patterns shows the protocol details (write invalidations, dirty-line transfers, silent eviction) clearly. Ruby is the right tool for that.

When to use which gem5 mode

For single-threaded studies and for memory-bandwidth experiments where coherence is not a first-order concern, ClassicCache is the choice. It is faster (typically 2 to 4 times faster than Ruby for the same workload), simpler to configure, and easier to debug. For multicore experiments, especially when the question involves coherence protocol effects, Ruby is the right tool. The trade-off is the steeper learning curve and the substantially larger simulation cost.

For comparing against ChampSim’s results, ClassicCache is the natural counterpart. Both treat the cache hierarchy as a sequence of crossbars and arrays. Both abstract coherence similarly. The numerical results from the two simulators on the same single-threaded workload should agree to within 5 to 10 percent if the configurations are matched.

05.Comparing the Three Simulators

The lab has now exercised three production simulators on the same family of workloads. Each occupies a distinct place in the trade-off space. The summary table below collects the headline characteristics.

Table 2. ChampSim vs DRAMsim3 vs gem5: scope and cost.

ChampSimDRAMsim3gem5
Scopecache + prefetcherDRAM onlyfull system
Inputmemory traceDRAM command traceELF binary
ISAISA-agnosticISA-agnosticRISC-V, x86, ARM
DRAM modelstub or DRAMsim3nativenative or DRAMsim3
CoherencenonoMOESI (Classic) or SLICC (Ruby)
Simulation cost (1B insns)–10 minseconds (DRAM only)–4 hours
ConfigurationJSON + C++.iniPython script
Learning curvegentlegentlesteep
Best forcache, prefetcher, replacement studiesmemory controller, DRAM technologyend-to-end studies, multicore, ISA-level

The three simulators are complementary, not competing. ChampSim is the right tool for the first 80% of cache and prefetcher research questions because it is the fastest and the easiest to modify. DRAMsim3 is the right tool when the question is about the DRAM subsystem itself (memory technology comparison, request scheduler design, refresh management). gem5 is the right tool for the last 20%, where the question requires real ISA execution, coherence protocols, or full-system effects.

The integration paths matter too. ChampSim + DRAMsim3 is the combination that the JILP data-prefetching championships use. gem5 + DRAMsim3 (or gem5 with its own DRAM controller) is the combination that academic full-system research uses. A ChampSim-based cache study can be confirmed against a gem5 run on the same workload to verify the conclusion holds at the higher detail level.

06.Worked Examples

07.Exercises

Book mode
computer-architecturememory-hierarchy
Was this helpful?