Part VAdvanced ILP and Out-of-Order Execution

Lab --- gem5 Out-of-Order Modeling

August 3, 2026·20 min read·advanced

The preceding lab of Chapter 48 set up gem5 alongside ChampSim and DRAMsim3 for memory-system studies. This lab returns to gem5 with a focus on its out-of-order CPU model (O3CPU), the right gem5 module for…

The preceding lab of Chapter 48 set up gem5 alongside ChampSim and DRAMsim3 for memory-system studies. This lab returns to gem5 with a focus on its out-of-order CPU model (O3CPU), the right gem5 module for exploring the mechanisms that Chapters Chapter 52 through Chapter 54 unpacked.

The lab is organized around a single experimental study. The reader will pick a small set of SPEC CPU 2017 benchmark traces, run them on O3CPU configurations that sweep the ROB size, the issue queue size, and the load-store queue size, and plot IPC as a function of each parameter. The goal is to reproduce, qualitatively, the published trade-off curves from Lipasti and Shen’s pioneering ILP studies and from more recent characterization papers. The methodology itself is the deliverable: at the end of the lab, the reader has a reproducible workflow for any gem5 O3CPU parameter sweep, not just the three exercised here.

The lab assumes gem5 is already built per the setup section of Chapter 48. The Setup and Installation section below adds the small additional pieces needed for the SPEC workloads and the plotting environment.

01.Setup and Installation

The lab requires gem5 (built and ready), a Python 3 environment with matplotlib and pandas for plotting, the GCC RISC-V cross toolchain for compiling SPEC if you need to rebuild the binaries, approximately 50 GB of disk space for the SPEC binaries and input sets, and about 1 GB for the gem5 output directories across all the sweep configurations.

Chapter 48 covered the gem5 build on all three platforms. If gem5 is not already built, return to that chapter’s setup section. The instructions below add only the incremental pieces.

macOS (Homebrew)

macOS additions for the lab

Plain Text
# Python data and plotting libraries. python3 -m pip install matplotlib numpy pandas seaborn # RISC-V cross toolchain, if you intend to rebuild SPEC for RISC-V. brew install riscv-gnu-toolchain # large download, optional # Optional: tmux for managing long sweep runs in the background. brew install tmux

Verify gem5 is in place and runnable:

Verify gem5 on macOS

Plain Text
cd path/to/gem5
./build/RISCV/gem5.opt --version # should print the gem5 version
./build/RISCV/gem5.opt configs/example/se.py --help | head -20

Linux (Arch as canonical)

Arch Linux additions for the lab

Plain Text
# Python libraries
sudo pacman -S python-matplotlib python-numpy \
python-pandas python-seaborn
# RISC-V cross toolchain
sudo pacman -S riscv64-linux-gnu-gcc riscv64-linux-gnu-binutils
# tmux for long runs
sudo pacman -S tmux
# AUR alternative for the latest matplotlib (optional)
# yay -S python-matplotlib-bin

For Debian or Ubuntu:

Debian/Ubuntu alternative

Plain Text
sudo apt update
sudo apt install python3-matplotlib python3-numpy \
python3-pandas python3-seaborn \
gcc-riscv64-linux-gnu binutils-riscv64-linux-gnu tmux

For Fedora:

Fedora alternative

Plain Text
sudo dnf install python3-matplotlib python3-numpy \
python3-pandas python3-seaborn \
gcc-riscv64-linux-gnu binutils-riscv64-linux-gnu tmux

Windows (ArchWSL)

Windows additions for the lab

Plain Text
# Inside the ArchWSL terminal:
sudo pacman -S python-matplotlib python-numpy \
python-pandas python-seaborn \
riscv64-linux-gnu-gcc riscv64-linux-gnu-binutils tmux

02.O3CPU Configuration Anatomy

gem5’s O3CPU is exposed as a Python class with dozens of configuration attributes. The most relevant attributes for this lab are summarized below.

Table 1. O3CPU configuration parameters most relevant to the lab. Default values are gem5 v23.0 defaults.

AttributeDefaultRole
fetchWidth8instructions per cycle from L1I
decodeWidth8decoded per cycle
renameWidth8renamed per cycle
issueWidth8issued from IQ per cycle
dispatchWidth8dispatched into ROB/IQ per cycle
commitWidth8retired per cycle
numROBEntries192reorder buffer size
numIQEntries64issue queue capacity
LQEntries32load queue capacity
SQEntries32store queue capacity
numPhysIntRegs256integer PRF size
numPhysFloatRegs256FP PRF size
branchPredTAGEbranch predictor class
fuPooldefaultfunctional unit mix

The defaults are reasonable for many studies but do not match any specific commercial part. The Golden Cove parameters from Chapter 63 (512-entry ROB, 192-entry IQ, 192-entry LQ) are sustainable in gem5, though the simulation slows down at larger ROB sizes because each pipeline stage walks the structures.

03.The Sweep Script

The sweep script is a Python program that iterates over the parameter values and invokes gem5 for each. The script writes each run’s output to a per-run directory and collects the resulting IPC into a CSV.

ROB sweep driver

Python
#!/usr/bin/env python3 """Sweep gem5 O3CPU's ROB size over a fixed workload set.""" import csv import os import re import subprocess import sys from pathlib import Path GEM5_BIN = "./build/RISCV/gem5.opt" SE_CONFIG = "configs/example/se.py" ROB_SIZES = [16, 32, 64, 128, 192, 256, 384, 512] BENCHMARKS = { "perlbench": "spec2017/perlbench_r/perlbench_r", "mcf": "spec2017/mcf_r/mcf_r", "gcc": "spec2017/gcc_r/gcc_r", "xalancbmk": "spec2017/xalancbmk_r/xalancbmk_r", "x264": "spec2017/x264_r/x264_r", } INSTR_LIMIT = 100_000_000 # 100M instructions per run def run_one(bench, binary, rob_size, outdir): outdir.mkdir(parents=True, exist_ok=True) cmd = [ GEM5_BIN, "--outdir", str(outdir), SE_CONFIG, "--cpu-type", "O3CPU", "--num-ROB-entries", str(rob_size), "-c", binary, "-I", str(INSTR_LIMIT), ] print(f"[run] bench={bench} ROB={rob_size}") proc = subprocess.run(cmd, capture_output=True, text=True) if proc.returncode != 0: print("STDERR:", proc.stderr, file=sys.stderr) return None stats = outdir / "stats.txt" return parse_stats(stats) def parse_stats(stats_path): """Extract IPC and a few other metrics from stats.txt.""" metrics = {} pat = re.compile(r"^([a-zA-Z0-9_.:]+)\s+([0-9.e+-]+)") with open(stats_path) as f: for line in f: m = pat.match(line.strip()) if m: key = m.group(1) try: metrics[key] = float(m.group(2)) except ValueError: pass return metrics def main(): out_root = Path("sweep_results") out_root.mkdir(exist_ok=True) csv_path = out_root / "rob_sweep.csv" with open(csv_path, "w") as f: writer = csv.writer(f) writer.writerow(["bench", "rob", "ipc", "bp_mpki", "l1d_miss_rate", "l2_miss_rate"]) for bench, binary in BENCHMARKS.items(): for rob in ROB_SIZES: run_dir = out_root / f"{bench}_rob{rob}" m = run_one(bench, binary, rob, run_dir) if m is None: continue ipc = m.get("system.cpu.ipc", 0.0) # gem5 keys vary slightly across versions. mpki_key = "system.cpu.branchPred.condIncorrect" mpki = m.get(mpki_key, 0.0) instrs = m.get("simInsts", 1.0) bp_mpki = 1000.0 * mpki / instrs l1d = m.get("system.cpu.dcache.demandMissRate", 0.0) l2 = m.get("system.cpu.l2cache.demandMissRate", 0.0) writer.writerow([bench, rob, ipc, bp_mpki, l1d, l2]) f.flush() if __name__ == "__main__": main()

The script makes a few opinionated choices that the reader should understand and adjust. The instruction limit (INSTR_LIMIT) caps each run at 100 million instructions to keep the wall-clock time manageable. The benchmark selection is five representative SPEC CPU 2017 integer workloads covering compute-bound and memory-bound characteristics. The ROB sizes span a 32x range, from a small in-order-equivalent 16 entries to the Golden-Cove-class 512 entries.

04.Parsing stats.txt

gem5’s stats.txt is a key-value text file with one statistic per line. The keys follow a hierarchical naming scheme rooted at system. A snippet from a typical O3CPU run looks like this:

Sample stats.txt excerpt

Plain Text
system.cpu.cpi 0.483520
system.cpu.ipc 2.068167
system.cpu.numCycles 48352001
system.cpu.committedInsts 100000000
system.cpu.iqSquashedInstsExamined 2854391
system.cpu.iqSquashedNonSpecRemoved 18372
system.cpu.iqInstsIssued 145872039
system.cpu.fuBusy 12382011
system.cpu.fuBusyRate 0.084913
system.cpu.branchPred.condPredicted 12846281
system.cpu.branchPred.condIncorrect 382041
system.cpu.dcache.demandHits 13428991
system.cpu.dcache.demandMisses 672948
system.cpu.dcache.demandMissRate 0.047720
system.cpu.l2cache.demandHits 428291
system.cpu.l2cache.demandMisses 244657
system.cpu.l2cache.demandMissRate 0.363560

The parser in the sweep script extracts these key-value pairs into a Python dictionary. Each row in the resulting CSV captures the metrics needed for the IPC-vs-ROB curve.

A useful pattern is to write a more general stats inspector that prints all key-value pairs matching a regex. This helps when exploring which gem5 stats are relevant for a particular study.

Stats inspector

Python
#!/usr/bin/env python3 """Print stats.txt entries matching a regex.""" import re import sys def main(): if len(sys.argv) != 3: print("Usage: inspect-stats.py <stats.txt> <regex>") sys.exit(1) pat = re.compile(sys.argv[2]) with open(sys.argv[1]) as f: for line in f: line = line.strip() if pat.search(line): print(line) if __name__ == "__main__": main()

Run it like this:

Inspect stats for branch predictor metrics

Plain Text
python3 scripts/inspect-stats.py sweep_results/gcc_rob192/stats.txt \
"branchPred"

05.Plotting IPC vs ROB Size

The plotting script reads the CSV from the sweep and produces one figure per benchmark plus a summary figure with all benchmarks overlaid.

Plot IPC vs ROB

Python
#!/usr/bin/env python3 """Plot IPC vs ROB size from rob_sweep.csv.""" import pandas as pd import matplotlib.pyplot as plt from pathlib import Path df = pd.read_csv("sweep_results/rob_sweep.csv") # One figure per benchmark. for bench in df["bench"].unique(): sub = df[df["bench"] == bench].sort_values("rob") plt.figure(figsize=(6, 4)) plt.plot(sub["rob"], sub["ipc"], marker="o", linewidth=2) plt.xlabel("ROB size") plt.ylabel("IPC") plt.title(f"IPC vs ROB size: {bench}") plt.xscale("log", base=2) plt.grid(True, alpha=0.3) plt.tight_layout() plt.savefig(f"sweep_results/ipc_vs_rob_{bench}.pdf") plt.close() # Overlay all benchmarks. plt.figure(figsize=(8, 5)) for bench in df["bench"].unique(): sub = df[df["bench"] == bench].sort_values("rob") plt.plot(sub["rob"], sub["ipc"], marker="o", linewidth=2, label=bench) plt.xlabel("ROB size") plt.ylabel("IPC") plt.title("IPC vs ROB size across SPEC 2017 benchmarks") plt.xscale("log", base=2) plt.legend() plt.grid(True, alpha=0.3) plt.tight_layout() plt.savefig("sweep_results/ipc_vs_rob_overlay.pdf") plt.close()

The shape of the IPC-vs-ROB curve is the central deliverable of the lab. For most SPEC workloads, the curve starts steep (doubling the ROB from 16 to 32 entries gains substantial IPC), flattens through the middle range (64 to 192 entries are the mainstream sweet spot for general-purpose workloads), and plateaus at the high end for compute-bound workloads (the IPC gain from 256 to 512 is modest, often below 5%).

The point at which the curve flattens depends on the workload. Compute-bound workloads (perlbench, gcc) flatten at relatively small ROB sizes. Memory-bound workloads (mcf, xalancbmk) keep gaining IPC up to large ROB sizes because the deeper window covers more cache-miss latency.

06.Sweeping the Issue Queue

A second sweep varies the issue queue size while holding ROB, LQ, and SQ at their default values. The relationship between IQ size and IPC is similar in shape to the ROB curve but saturates earlier. A 64-entry IQ is typically sufficient on most workloads, and the gain from 64 to 128 is small.

The IQ sweep script is structurally identical to the ROB sweep script, with --num-IQ-entries replacing --num-ROB-entries. The values to sweep are 16, 32, 48, 64, 96, and 128.

07.Sweeping the Load-Store Queue

The third sweep varies the LQ and SQ sizes together. The relationship between LSQ size and IPC is more nuanced than the ROB or IQ relationships because the LSQ binds on memory-bound workloads.

For compute-bound workloads, the LSQ is rarely full and the sensitivity to LSQ size is small. For memory-bound workloads (mcf, xalancbmk), the LSQ binds and the IPC scales noticeably with LSQ size. The Golden Cove choice of 192 LQ entries (Chapter 63) is much larger than the Zen 4 choice of 136 LQ entries (Chapter 64), reflecting Intel’s bet that memory-bound workloads benefit from the deeper in-flight memory window.

The LSQ sweep configuration adds two parameters:

LSQ sweep parameter passing

Python
cmd = [
GEM5_BIN,
"--outdir", str(outdir),
SE_CONFIG,
"--cpu-type", "O3CPU",
"--num-LQ-entries", str(lsq_size),
"--num-SQ-entries", str(lsq_size // 2),
"-c", binary,
"-I", str(INSTR_LIMIT),
]

The store queue is conventionally sized to half the load queue on most commercial designs. Holding the LQ/SQ ratio at 2:1 during the sweep matches the industry convention.

08.Reproducing a Published Study

A useful final exercise is to reproduce, qualitatively, the results of a published characterization study. The classic target is Lipasti and Shen’s 1996 study on the limits of ILP, which sweeps several core parameters and reports IPC saturation points. More recent targets include the Karkhanis-Smith analytic model (ISCA 2004) and the various ChampSim-based studies that sweep prefetcher and branch-predictor sizes.

To reproduce qualitatively means to obtain curves of the same shape and the same approximate saturation points, not to reproduce identical absolute numbers. The latter would require matching the exact tool versions, the exact workloads, the exact compiler revisions, and many subtle details that the publishing community does not always disclose. The qualitative reproduction is the appropriate standard.

The table below lists a small set of published results and the rough qualitative shape this lab should obtain.

Table 2. Selected published studies to reproduce qualitatively.

StudyVariableExpected shape
Lipasti & Shen 1996ROB sizemonotonic, saturating at 128-256
Karkhanis & Smith 2004ROB size on mcfstill gaining at 512
Pellauer & Emer 2010IQ sizesaturates at 64-96
Choi & Yeager 2008LSQ sizestrongly workload-dependent

09.Run-time Engineering

A practical issue is wall-clock time. A full sweep across three parameters (ROB, IQ, LSQ), at 8 ROB values, 6 IQ values, and 8 LSQ values, on 5 benchmarks at 100M instructions per run, totals (8+6+8)5=110(8 + 6 + 8) \cdot 5 = 110 runs. At 30 minutes per run on a modern workstation, that is 55 hours of compute. Strategies that bring this down:

Use gem5.fast for the exploratory passes that establish the curve shape. Switch to gem5.opt only for the production runs that go into the figures.

Reduce the instruction limit during exploration. A 10M- instruction run gives a noisier but qualitatively similar result and runs 10x faster.

Use SimPoint clustering to identify the most representative program phase. Run gem5 on just the SimPoint excerpts rather than the entire benchmark. SimPoint is the standard methodology for this trick and is documented in the gem5 wiki.

Parallelize runs across machines. Each run is independent, so the sweep parallelizes embarrassingly. If you have a small cluster or several workstations, dispatch one run per machine.

10.Common Pitfalls

Several pitfalls are easy to fall into during a gem5 sweep study. The lab is the right place to call them out.

Comparing across gem5 versions. The O3CPU default parameters change across gem5 releases. A study that uses gem5 v22.0 in some runs and gem5 v23.0 in others produces inconsistent results. Pin to one version for the whole study.

Mixing SE and FS results. SE mode and FS mode can produce different IPC for the same workload because of differences in syscall overhead and kernel time. A study should use one mode throughout.

Not warming the simulation. gem5’s caches start cold. The first few million instructions of a run are atypical. The baseline sweep scripts above omit warmup to keep the command lines readable, so adding a warmup interval (--warmup-insts) of at least a few million instructions before the measurement window is the first refinement to make to them.

Reporting noisy IPC. Run each configuration two or three times and report the mean. gem5 is deterministic given the same seed, so two runs of identical configuration should give identical results. Repeating helps catch infrastructure-level non-determinism (file-system caching, machine load).

Confusing parameters with their names. gem5 has parameters named numROBEntries but also commitWidth that acts as a logical retire bandwidth limit. A "ROB sweep" should hold all other parameters fixed, including commitWidth.

11.Worked Examples

12.Deliverable Report

The lab’s deliverable is a short report (4 to 6 pages) containing:

A description of the experimental setup. gem5 version, benchmark selection, instruction limit, baseline parameter values. State everything that a third party would need to reproduce the results.

Three figures: IPC vs ROB size, IPC vs IQ size, IPC vs LSQ size. Each figure should overlay all benchmarks on one set of axes and use a log scale on the x-axis. Caption each figure with the saturation point for each benchmark.

A discussion section that interprets the curves in terms of the workload characteristics. Memory-bound workloads should appear to benefit more from larger ROBs and LSQs. Compute-bound workloads should saturate earlier.

A comparison section that places the saturation points against the parameters of Golden Cove (Chapter 63) and Zen 4 (Chapter 64). Discuss whether the saturation points support the commercial design choices.

A brief reproducibility appendix listing the gem5 command lines, the sweep scripts, and the parsing scripts. A reader should be able to re-run the entire study from the appendix alone.

13.Exercises

Book mode
computer-architectureadvanced-ilp-and-out-of-order-execution
Was this helpful?