Performance Modeling and Correlation

01.Part 1, the problem before the method

1.1 A design decision you cannot make by argument

Start with a concrete decision that somebody on a CPU design team has to make, three years before a chip ships.

The reorder buffer, described in Out of Order Execution, is the structure that holds every instruction that has started but not yet retired. How many entries should it have? 128? 256? 512?

Try to answer that by reasoning alone and watch it fail. Bigger is better, because a bigger window means more independent instructions visible, which means more work to do while a cache miss is outstanding. But bigger costs area, and area costs money and leakage power. Bigger also means longer wires, and the ROB is read and written every cycle by every pipeline stage, so a bigger one may not fit in the cycle time, which would force a frequency reduction that costs performance everywhere. And bigger only helps if there is independent work behind the stall, which depends entirely on the program.

Four effects, pulling in different directions, each of which depends on numbers nobody knows yet. There is no argument that settles this. You have to measure something that does not exist yet, and that is what a performance model is for.

1.2 What these roles actually ask for

CPU design roles carry the identical responsibility bullet. "Performance exploration and correlation, explore high-performance strategies and work with the performance verification team to verify that the RTL design meets targeted performance."

Read that as two separate jobs joined by "and."

Exploration is the forward job. Before RTL exists, decide the structure sizes, the widths, the latencies, the policies. This is architecture work and it is done in a simulator.

Correlation is the backward job. After RTL exists, prove it actually delivers what the model promised. When it does not, find out which of the two is lying.

Candidates prepare for the first half and get ambushed by the second. The second half is where a design engineer contributes, because chasing a 4 percent cycle-count gap between a model and RTL is an RTL debugging problem, not an architecture problem. Part 7 is the important part of this note.

1.3 The one number

Everything in this note is ultimately about one equation, which you should be able to write without thinking.

$\text{Time} = \frac{\text{Instructions}}{\text{Program}} \times \frac{\text{Cycles}}{\text{Instruction}} \times \frac{\text{Seconds}}{\text{Cycle}}$

Instruction count is set by the ISA and the compiler. Seconds per cycle is the clock period, set by the timing closure work in STA Synthesis and Physical Design. Cycles per instruction, CPI, is the microarchitecture's contribution, and its reciprocal, IPC, is the number every model in this note exists to estimate.

Put numbers on it so the units are not abstract. A program of 10 billion instructions, running at IPC 2.0, on a 3 GHz clock, takes

$\frac{10 \times 10^9}{2.0} \times \frac{1}{3 \times 10^9} = \frac{5 \times 10^9}{3 \times 10^9} \approx 1.67\ \text{seconds}$

Now notice something that shapes Apple's whole design philosophy. You can halve that time by doubling IPC or by doubling frequency, and the two are not equally priced. Frequency costs power roughly as $f \cdot V^2$ with voltage rising alongside frequency, so the power cost is closer to cubic, as covered in DVFS Droop and Thermal. IPC costs area and design effort but the power cost is closer to linear. In a phone, where the power budget is fixed by what a hand can dissipate, buying performance through IPC is simply a better trade. That is why Apple builds wide, relatively low-clocked cores, and the only way to know whether width is actually buying IPC is to model it.

---

02.Part 2, the modeling hierarchy

2.1 The spectrum, stated up front

There are five broad kinds of model and they span roughly seven orders of magnitude in speed. Here is the whole picture before the details.

Native means running the program on a real machine. A benchmark that takes 60 seconds natively takes about a week in a cycle-approximate model and roughly a century in RTL simulation. Those ratios are the entire reason the hierarchy exists.

2.2 Analytical models, and why you always start here

An analytical model is an equation you evaluate by hand or in a spreadsheet.

The most useful one in the memory hierarchy is average memory access time, from Cache Organization and Prefetching.

$\text{AMAT} = t_{hit} + (\text{miss rate}) \times t_{miss\ penalty}$

Work it. An L1 with a 4-cycle hit time, a 3 percent miss rate, and a 60-cycle penalty to L2 gives

$\text{AMAT} = 4 + 0.03 \times 60 = 4 + 1.8 = 5.8\ \text{cycles}$

Now ask a design question with it. Someone proposes doubling the L1 to cut the miss rate from 3 percent to 2 percent, at the cost of one extra cycle of hit latency.

$\text{AMAT}_{new} = 5 + 0.02 \times 60 = 5 + 1.2 = 6.2\ \text{cycles}$

Worse. The proposal loses. You just killed a design idea in four seconds of arithmetic with no simulator at all, because the hit term dominates and it got bigger. That is what analytical models are for.

The second analytical model worth memorizing is the branch-misprediction CPI adder from CPU Foundations Pipeline and Hazards.

$\text{CPI}_{branch} = (\text{branch frequency}) \times (\text{misprediction rate}) \times (\text{penalty cycles})$

With 20 percent branches, a 3 percent misprediction rate, and a 15-cycle penalty,

$\text{CPI}_{branch} = 0.20 \times 0.03 \times 15 = 0.09\ \text{cycles per instruction}$

If the base CPI is 0.5, this adder is 18 percent of the total, which tells you branch prediction is worth real effort here. Halve the misprediction rate to 1.5 percent and you save 0.045 CPI, roughly 8 percent performance. That estimate is good enough to decide whether the predictor team gets funded.

The third is Little's law, and because it is used constantly and is almost never explained, here it is from zero.

Picture a coffee shop. Customers arrive at 2 per minute. Each customer stays 5 minutes. How many people are in the shop on average? Two arrive every minute and each hangs around for five minutes, so at any snapshot you see roughly five minutes' worth of arrivals standing there, which is $2 \times 5 = 10$ people. That is the whole law.

$L = \lambda \times W$

Occupancy equals arrival rate times residence time. It holds for any stable system with no assumptions about the arrival pattern, which is why it is so useful.

Now point it at hardware. If a core sustains 2 instructions per cycle and the average instruction sits in the ROB for 30 cycles from allocation to retirement, then the average ROB occupancy is $2 \times 30 = 60$ entries. A 64-entry ROB would be right at the average, which means it would be full a great deal of the time, since the average is not the peak. Hold that number, Part 3 builds on it.

What analytical models cannot do. They cannot capture interaction. AMAT quietly assumes every miss stalls the machine, which is false in an out-of-order core where a miss overlaps with other work. It assumes misses are independent, which is false when a prefetcher is running. It has no notion of a queue filling up, of a port conflict, or of a mispredicted branch launching wrong-path loads that pollute the cache. The moment two mechanisms interact, the equation is wrong and you have to simulate.

2.3 Trace-driven simulation

A trace is a recorded list of events from a real program run. For a cache study, it is a list of memory addresses in the order they were accessed. For a branch study, it is a list of branch program counters and their taken or not-taken outcomes.

A trace-driven simulator reads that list and models one structure against it. Project --- A Trace-Driven Cache Simulator in C++ builds exactly this.

The appeal is real. It is fast, because there is no timing to compute, just tag lookups. It is perfectly reproducible, because the same trace gives the same answer every time, which matters enormously when you are comparing 40 configurations and need the difference between them to be the only variable. And it is easy to write, which is why it is a good first project.

2.4 The limitation that gets asked, and the two forms it takes

A trace was recorded on some other machine. That single fact produces two separate failures, and interviewers ask about both.

Failure one, no timing feedback. The trace is a fixed sequence. Nothing in the sequence responds to how fast your model runs. If your design has a slower L2, real software would experience different memory-level parallelism, different queue occupancies, different prefetcher training. The trace does not care. It replays the same addresses at the same relative positions no matter what your model does. Any effect where timing changes the sequence is invisible.

Failure two, no wrong path. This is the sharper one. A trace records what the program actually executed, which by definition is the correct path. A real out-of-order machine predicts branches, and when it predicts wrong it fetches, decodes, and often executes instructions on the wrong path before squashing them, as covered in Front End and Branch Prediction.

Those wrong-path instructions are not free. They issue loads that occupy MSHRs, they bring lines into the cache that the correct path may or may not want, they consume issue-queue entries and execution ports, and they occasionally prefetch something useful by accident. None of that is in the trace.

So here is the question you will get. "Why can a trace-driven simulator not properly evaluate a branch predictor change?" The answer is that a better predictor changes the set of wrong-path instructions, and the trace has no wrong-path instructions at all, so the mechanism you are trying to measure does not exist in the model. You can measure the misprediction rate from a trace, which is fine, but you cannot measure the performance effect of that rate change, because the cost of a misprediction depends on machine state the trace does not carry.

There is a subtler version. Trace-driven cache studies systematically report optimistic miss rates for one reason and pessimistic ones for another, and which dominates depends on the workload. Optimistic because no wrong-path pollution. Pessimistic because no wrong-path accidental prefetch. You cannot tell which without an execution-driven run, which rather defeats the purpose.

2.5 Execution-driven simulation

An execution-driven simulator actually interprets the program's instructions against a model of the machine. It has an architectural state, it fetches, it predicts, it executes down the wrong path, and it squashes. gem5's O3CPU is the standard open-source example and is the subject of Lab --- gem5 Out-of-Order Modeling.

Because it executes, everything the trace could not model comes back. Timing feedback is captured, because the model's own timing determines what happens next. Speculation is modeled, because the model genuinely goes down the wrong path. Structural interactions are captured, because the queues in the model actually fill.

The price is speed. A detailed out-of-order model runs somewhere around 100,000 to 500,000 simulated instructions per second on a modern host. Call it 200 KIPS. A native machine at 3 GHz and IPC 2 executes 6 billion instructions per second. The slowdown is

$\frac{6 \times 10^9}{2 \times 10^5} = 30{,}000\times$

That number drives Part 6 entirely. It is why you cannot simulate a full benchmark and why sampling is not optional.

There is a further split worth knowing. Cycle-approximate models get the structure right and the cycle counts within a few percent, which is plenty for comparing two designs. Cycle-accurate models claim to match the RTL cycle for cycle, which is far harder, much slower, and is the thing correlation in Part 7 tries to achieve. Most architecture exploration happens in cycle-approximate models, and treating a cycle-approximate number as an absolute prediction rather than a relative comparison is a classic mistake.

2.6 RTL simulation, and why it is not the answer to everything

RTL simulation is the ground truth for the design that exists. It captures everything, because it is the design.

It is also somewhere between $10^6$ and $10^8$ times slower than native, it requires the RTL to be written, and it requires the whole design to be integrated before you can measure a full-core number. Running SPEC on a full-core RTL model is not a thing anyone does. Instead you run short traces, tens of thousands to a few million cycles, chosen because they exercise the behavior you care about.

And RTL cannot answer the question that started this note, because the ROB size question must be settled before the RTL is written. That is the whole point of the hierarchy.

2.7 The rule

Use the cheapest model that can answer the question you are actually asking.

That sounds like a platitude until you apply it. Someone asks whether a 3-cycle L1 beats a 4-cycle L1 with a lower miss rate. Do not launch a 200-run gem5 campaign. Write the AMAT equation, get the answer in four seconds, and only escalate if the two options land within noise of each other. Someone asks how deep the store queue should be. That needs an execution-driven model, because store-queue occupancy depends on stall behavior which depends on timing. Someone asks whether the RTL's issue-queue wakeup logic is behaving as designed. That needs RTL, because no model above it contains the wakeup logic.

Escalating too early wastes months. Escalating too late produces a confidently wrong answer.

---

03.Part 3, structure sizing, worked properly

3.1 Why the ROB needs to be large, derived rather than asserted

Go back to Little's law from 2.2 and apply it to the real case.

A load misses all the way to DRAM. Say the latency is 200 cycles. Because the machine retires in order (see Out of Order Execution), that load sits at the head of the ROB for the whole 200 cycles and cannot retire. Every instruction that was fetched after it stays in the ROB too, because they cannot retire past it.

Meanwhile the machine is trying to keep working. If it can sustain 2 instructions per cycle on the independent work behind the miss, then over 200 cycles it wants to allocate

$2 \times 200 = 400\ \text{ROB entries}$

If the ROB holds only 128, it fills after 64 cycles and the machine stalls for the remaining 136 cycles of the miss. The ROB size directly caps how much of a memory latency you can hide.

That derivation gives you the shape of the answer before you run anything. It says the useful ROB size scales with (target IPC) times (the latency you want to cover). It also says the returns must eventually stop, because at some point you are covering the full DRAM latency and there is nothing more to cover.

3.2 The sweep

Now run it. In gem5 or an equivalent, hold everything else fixed and vary only the ROB, measuring IPC on a workload suite. Here is a representative sweep, with the marginal gain and the marginal gain per entry added computed alongside.

Plot it and the shape is the point.

Three regions, and naming them is how you talk about this in an interview.

The steep region, 32 to 96. Every entry you add is doing work. The machine is window-limited, meaning it can see independent instructions but has nowhere to put them. Adding entries here is the cheapest performance in the design.

The knee, around 128 to 192. The curve bends. You are still gaining but the gain per entry has dropped by roughly 4x from the steep region.

The flat region, past 256. You are adding storage that is rarely occupied and, when it is occupied, contains instructions that cannot make progress anyway because they are dependent on the same stalled load.

3.3 Why it flattens, which is the actual question

Do not just say "diminishing returns." Say why, because there are three separate mechanisms and an interviewer is checking whether you know them.

One, the program runs out of independent work. A large window only helps if the instructions behind the stalled load are independent of it. Real programs have dependence chains. If instruction 200 past the miss depends on instruction 50 past the miss which depends on the miss itself, then a 512-entry ROB and a 128-entry ROB behave identically, because everything past entry 50 is blocked regardless. This is the intrinsic ILP limit of the program and no amount of hardware fixes it.

Two, some other structure binds first. The ROB is not the only queue. Sweep it alone and eventually the physical register file, the issue queue, the load queue, or the MSHR count becomes the binding constraint, at which point the ROB curve flattens for reasons that have nothing to do with the ROB.

This is why single-parameter sweeps mislead, and it is worth a table.

Look at row two. Doubling the ROB from 128 to 256 bought 0.03 IPC, and a naive reading says the ROB does not matter. Row five shows it does, once the structures around it are scaled to match. The lesson is that structures must be sized as a set, and a sweep of one parameter with the others held at their old values will systematically underestimate its value. Balanced-design sweeps, where several structures scale together, are the honest version.

Three, the cost is not linear. A ROB entry is not just storage. The ROB is written by allocate, read by retire, searched by recovery, and its occupancy feeds stall logic. Doubling entries can push the structure out of a single cycle, which either costs a pipeline stage or costs frequency, and a 3 percent frequency loss to buy 1 percent IPC is a net loss.

3.4 Picking the design point

The engineering answer is not "where the curve flattens." It is where the marginal IPC stops justifying the marginal cost, and that requires a cost model alongside the performance model.

Suppose a ROB entry costs roughly 0.9 percent of the core area per 16 entries at the sizes involved, and suppose the team's rule of thumb is that a feature must return at least 1 percent performance per 1 percent area. Apply that to the sweep.

The design point is 192 or 256 depending on how much you value the marginal case, and the argument for it is now quantitative rather than aesthetic. That is what the whole exercise was for.

3.5 The other uses

Feature evaluation. Someone proposes a mechanism, say a stride prefetcher for the L2. Model it, measure IPC on the suite, and compare against the area and power estimate. The critical discipline is measuring across a suite, not one benchmark, because almost every prefetcher helps something and hurts something else. Report the geometric mean and the worst regression, and a feature with a 4 percent mean gain and a 9 percent worst-case regression is usually rejected, because shipped products cannot have workloads that get slower.

Bottleneck analysis. Where are the cycles going? Part 4 is entirely about doing this correctly.

Sensitivity analysis. How much does the conclusion depend on an assumption? If your ROB study assumed 200-cycle DRAM latency, rerun the key points at 150 and 250 and see whether the recommended size moves. Rerun with the branch predictor 1 percent better and 1 percent worse. If the recommendation is 192 entries at every assumption, you have a result. If the recommendation swings from 128 to 384 when DRAM latency moves by 25 percent, you do not have a result, you have an artifact of one assumption, and shipping a design based on it is how a chip arrives 10 percent slower than the model said.

---

04.Part 4, top-down analysis

4.1 The naive method, and watching it break

You have a slow program and you want to know why. The obvious approach is to count stall cycles by cause. Add a counter for cycles stalled on instruction-cache misses, one for data-cache misses, one for branch mispredictions, one for divider occupancy, and so on. Then look at which counter is biggest.

Watch this fail on a concrete case.

Take a 1000-cycle window. During it, an L2 miss is outstanding for 40 cycles. Overlapping with that, an instruction-cache miss is outstanding for 25 cycles, 20 of which fall inside the L2 miss window. A branch mispredict recovery takes 15 cycles, 10 of which also overlap the L2 miss. The machine was actually unable to make progress for 55 cycles total.

Now read the counters.

The counters sum to 80 for 55 real cycles, 145 percent of reality. Every overlapping cycle was counted twice or three times.

That is bad enough. The next failure is worse. Suppose you now fix the icache miss entirely, believing it was worth 25 cycles. The L2 miss is still there and still 40 cycles. You recover 5 cycles, not 25, because 20 of the 25 were hidden under the L2 miss and cost nothing. You optimized something whose measured cost was almost entirely fictional, which is exactly how weeks get spent for no gain.

The root problem is that in an out-of-order machine several things are in flight at once by design, and a cycle-based accounting has no way to decide which of the overlapping causes "owns" the cycle. Any assignment you invent is arbitrary.

4.2 The fix, count slots instead of cycles

Top-down microarchitecture analysis, introduced by Ahmad Yasin in a public 2014 paper and now implemented in the performance counters of essentially every modern high-performance core, fixes this by changing the unit of accounting.

Instead of counting cycles, count pipeline slots. A pipeline slot is one opportunity to move one micro-operation through the machine's narrowest issue point, once, in one cycle. A 4-wide machine has 4 slots per cycle. Over 1000 cycles it has 4000 slots.

The crucial property is that a slot is a discrete thing that either got used or did not, and it has exactly one story. There is no overlap possible, because a slot is a single position in a single cycle. Ask of every slot what happened to it, and the answers partition perfectly.

4.3 The four buckets

Every slot goes through one decision tree, and it lands in exactly one of four buckets.

Retiring. The slot delivered a micro-operation that eventually retired and did useful architectural work. This is the good bucket, and its fraction is the machine's efficiency at that width.

Bad speculation. The slot delivered a micro-operation that was later thrown away. Dominated by branch mispredictions from Front End and Branch Prediction, with a smaller contribution from machine clears such as memory-ordering violations described in Load Store and Memory Ordering. Note that this bucket also implicitly charges for the recovery bubbles, because the empty slots during a pipeline flush are attributed here rather than to the front end, which is correct, since the front end was doing exactly what it was told.

Front-end bound. The slot was empty and the back end had room. The front end failed to supply. Causes are instruction-cache misses, iTLB misses, branch resteers, decode bandwidth limits, and fetch alignment.

Back-end bound. The slot was empty and the back end could not have taken anything anyway, because a resource was full or a dependency was unresolved. This is where cache misses, port contention, and dependence chains live.

The four sum to 100 percent by construction. That is not a happy accident, it is the entire design of the method.

4.4 Working real numbers

Take a 4-wide machine over a 1000-cycle window, so 4000 slots.

Sanity-check the retiring number against IPC. 1780 micro-operations retired over 1000 cycles is 1.78 uops per cycle, which on a 4-wide machine is 44.5 percent of peak. That matches the retiring fraction, as it must, and this cross-check is worth doing every time because it catches counter-configuration mistakes.

Now read the result. Back-end bound at 34 percent is the largest non-retiring bucket, so that is where the effort goes. Front-end bound at 13.5 percent is second. Bad speculation at 8 percent is real but is not the story.

Compare that to what the naive method would have told you. The naive counters, summing to more than 100 percent, would very likely have shown branch mispredictions as a large absolute number of cycles, because a 15-cycle penalty times a few thousand mispredicts is a big number, and a team could easily have spent a quarter on the predictor for a 2 percent return.

4.5 The drill-down

Each bucket subdivides, and the same partition property holds at every level. Here is the hierarchy.

Drill the back-end bucket of the worked example and suppose it splits as follows.

And memory bound splits again.

Now you have an actionable answer. 9 percent of all issue slots are lost to DRAM. That is the single biggest identifiable cost, it is bigger than the entire bad-speculation bucket, and it points at prefetching, at the last-level cache, or at the memory controller. That conclusion could not have been reached from the naive counters, because the DRAM stalls would have been shadowed by everything overlapping them.

The workflow discipline is simple. Only drill into the largest bucket. Optimizing a level-3 leaf inside a level-1 bucket that is only 8 percent of slots caps your possible gain at 8 percent before you start.

4.6 How it maps onto real counters

The reason this method matters to a hardware engineer rather than only to a software performance analyst is that it is implemented in hardware counters, so the identical methodology runs on a model and on silicon.

The counters needed are few, and the formulas are arithmetic on them. Using the publicly documented event names as an illustration,

$\text{Total slots} = W \times \text{CPU\_CLK\_UNHALTED}$

$\text{Retiring} = \frac{\text{UOPS\_RETIRED.RETIRE\_SLOTS}}{\text{Total slots}}$

$\text{Bad speculation} = \frac{\text{UOPS\_ISSUED.ANY} - \text{UOPS\_RETIRED.RETIRE\_SLOTS} + W \times \text{RECOVERY\_CYCLES}}{\text{Total slots}}$

$\text{Front-end bound} = \frac{\text{IDQ\_UOPS\_NOT\_DELIVERED.CORE}}{\text{Total slots}}$

$\text{Back-end bound} = 1 - (\text{Retiring} + \text{Bad speculation} + \text{Front-end bound})$

Read the last line carefully, because it is the honest part of the method. Back-end bound is computed as the residual, not measured directly. Counting "slots the back end refused" directly would require knowing, for every empty slot, whether the back end would have accepted a uop had one been offered, and that counterfactual is expensive to build in hardware. Taking it as the remainder is cheap and exact, given that the four buckets partition the space. The cost is that every measurement error in the other three lands in the back-end bucket, which is a real caveat and a good thing to know when someone shows you a top-down chart with 60 percent back-end bound.

The bad-speculation formula deserves the same attention. Issued minus retired counts the uops that were thrown away. The recovery term adds the slots that went empty during the flush, multiplied by the width because every slot in those cycles is lost. Without that second term you would undercount mispredictions badly, since the flush bubble is usually larger than the wrong-path work itself.

This counter mapping is the bridge to DFT and Silicon Debug. The same counters used for post-silicon performance analysis are part of the observability infrastructure that gets designed in, and if the counters are not there or are wrong, this entire method is unavailable on the shipped part.

---

05.Part 5, roofline

5.1 Arithmetic intensity, from an example

Top-down tells you where issue slots went inside a core. Roofline answers a different and coarser question. Is this code limited by how fast the machine computes, or by how fast it can move data?

The concept you need first is arithmetic intensity, and it is simpler than the name suggests. It is the number of useful arithmetic operations performed per byte moved between the processor and memory.

$I = \frac{\text{operations}}{\text{bytes moved}}$

Work a real kernel. Vector add on double-precision values, c[i] = a[i] + b[i], for a large array that does not fit in cache.

Per element, count the arithmetic. One addition. That is 1 flop.

Per element, count the bytes. Read a[i], 8 bytes. Read b[i], 8 bytes. Write c[i], 8 bytes. That is 24 bytes.

$I_{vector\ add} = \frac{1}{24} = 0.042\ \text{flops per byte}$

Now a second kernel for contrast. Dense matrix multiply of two $1024 \times 1024$ double matrices. The arithmetic is $2N^3 = 2 \times 1024^3 \approx 2.15 \times 10^9$ flops. With good cache blocking, the data moved is roughly the three matrices once each, $3 \times 1024^2 \times 8 \approx 2.5 \times 10^7$ bytes.

$I_{matmul} = \frac{2.15 \times 10^9}{2.5 \times 10^7} \approx 86\ \text{flops per byte}$

Two thousand times higher. That difference is the entire reason these two kernels behave completely differently on the same machine.

5.2 The roof

Now describe the machine with exactly two numbers. Peak compute, say 200 GFLOP/s. Peak memory bandwidth, say 100 GB/s.

Achievable performance is capped by both.

$P \le \min(P_{peak},\ I \times BW_{peak})$

The first term is a horizontal ceiling. The second is a sloped ceiling, because if you only move bytes at 100 GB/s and each byte carries $I$ operations of work, you cannot exceed $I \times 100$ GFLOP/s no matter how fast the ALUs are.

The two ceilings cross at the ridge point, where $I \times BW = P_{peak}$, so

$I_{ridge} = \frac{200}{100} = 2\ \text{flops per byte}$

Place the two kernels. Vector add at $I = 0.042$ is far left of the ridge, so its ceiling is $0.042 \times 100 = 4.2$ GFLOP/s, which is 2.1 percent of the machine's peak compute. Matrix multiply at $I = 86$ is far right of the ridge, so its ceiling is the full 200 GFLOP/s.

5.3 What that buys you

The conclusion is blunt and immediately actionable. For vector add, doubling the number of FPUs does nothing. Not a little, nothing. The kernel is at 2 percent of compute peak because bandwidth caps it, and adding compute moves a ceiling that was never binding. The only levers that help are ones that reduce bytes moved, such as fusing the loop with a neighbouring one so an array is not written and reread, using single precision to halve the bytes, or improving cache reuse to keep data on-chip.

For matrix multiply the opposite holds. Bandwidth upgrades are wasted and wider FMA units pay off directly.

That is the value proposition of roofline in one sentence. IPC tells you how well the machine is doing. Roofline tells you which resource to spend money on. A kernel running at 30 percent of peak compute sounds bad until you notice its arithmetic intensity puts its ceiling at 32 percent, at which point it is running at 94 percent of what is achievable and the engineer should go work on something else.

Real roofline plots add more ceilings. A lower horizontal line for scalar-only code without SIMD, another for code without FMA, and lower sloped lines for memory access patterns that do not achieve peak bandwidth such as strided or random access. Each ceiling you are stuck under names a specific optimization.

---

06.Part 6, workloads and the sampling problem

6.1 A conclusion is only as good as what you ran it on

Every number in Parts 3 through 5 came from running something. Change what you run and the numbers change, sometimes enough to reverse the design decision. This is where a great deal of performance work quietly goes wrong.

SPEC CPU is the industry-standard single-thread suite, split into integer and floating-point subsets, with run rules strict enough that published results are comparable across vendors. Its virtue is that it is well understood, widely reported, and hard to argue with. Its criticisms are real. It is compute-heavy relative to modern software, several of its components have small working sets by contemporary standards, and compilers have been tuned against it for decades so it partly measures compiler effort.

Consumer benchmarks such as GeekBench are what reviewers quote and therefore what customers see. They are short, which makes them sensitive to boost behavior and thermal state in ways that matter commercially and can mislead architecturally. A design tuned to win a 3-second burst benchmark is not necessarily the design that renders a web page fastest.

Real applications are what a company shipping consumer devices actually cares about. Browser page loads and JavaScript execution, video encode and decode, photo processing, compilation, application launch, UI responsiveness. These have properties SPEC does not, notably very large instruction footprints. A browser can have an instruction working set of several megabytes, which stresses the instruction cache and the branch predictor in ways no SPEC component does, and that is why front-end structures in Apple cores are as large as they are.

Microbenchmarks isolate exactly one behavior. A pointer-chase loop to measure load-to-use latency. A dependent-add chain to measure ALU latency. A wide independent stream to measure issue width. They are essential for characterization and for correlation in Part 7, because they produce a number you can predict analytically and therefore check. They are terrible for judging a design, because no real program looks like them and optimizing for them produces a machine that is fast at nothing anyone runs.

The discipline that follows is a suite, always, with the geometric mean reported alongside the per-workload spread and the worst regression called out explicitly.

6.2 Workload characterization

Before you can reason about why two workloads respond differently to a design change, you have to measure what they are.

Now the point of the table. Two workloads with identical IPC can require completely different machines. Workload X runs at IPC 2.0 with 8 percent branches that are 99.5 percent predictable, a 200 KB working set that fits in L2, and long dependence chains. Workload Y also runs at IPC 2.0, with 24 percent branches at 96 percent predictable, a 40 MB working set, and abundant independent work. X wants deeper pipelining and more execution latency tolerance. Y wants a much better predictor, a much bigger last-level cache, and more MSHRs. Report only IPC and these two look interchangeable, which is why characterization exists.

6.3 Sampling, because full runs are impossible

Now confront the arithmetic from 2.5.

A single SPEC CPU component executes on the order of $5 \times 10^{11}$ instructions on its reference input. At 200,000 simulated instructions per second,

$\frac{5 \times 10^{11}}{2 \times 10^5} = 2.5 \times 10^6\ \text{seconds} \approx 29\ \text{days}$

That is one benchmark, one configuration. A suite of 20 benchmarks across the 9-point ROB sweep from Part 3 is $20 \times 9 = 180$ runs, which is

$180 \times 29\ \text{days} \approx 14\ \text{years of machine time}$

Farms are parallel, so with 500 machines that is about ten days, but the sweep in 3.3 had four parameters, and you wanted sensitivity runs on top. The full-run approach does not survive contact with reality. You must sample.

The naive sample is to simulate the first billion instructions. This is badly wrong, because the first billion instructions of almost any program are initialization, which has nothing to do with the program's steady-state behavior. The second naive approach is to skip a fixed 10 billion and then sample, which is better and still arbitrary, because it happens to land wherever it lands.

6.4 SimPoint

SimPoint solves this properly, and the idea is worth understanding rather than name-dropping.

Programs execute in phases. A compiler spends a while in lexing, then in parsing, then in optimization, and each phase has a characteristic instruction mix, working set, and branch behavior. If you can identify the phases and how much time is spent in each, you can simulate one representative slice of each phase and weight the results.

The mechanism is the clever part. Chop the program's execution into intervals of, say, 100 million instructions. For each interval, record a basic block vector, which is simply a histogram counting how many times each static basic block in the program was executed during that interval, weighted by block length. That vector is a fingerprint of what code ran.

The insight is that intervals executing the same code in the same proportions behave the same way microarchitecturally, regardless of the actual data. So cluster the basic block vectors with k-means. Each cluster is a phase. Pick the interval nearest each cluster centroid as the representative, and weight it by the fraction of total intervals in that cluster.

Concretely, a program with 5000 intervals of 100 million instructions each, so 500 billion instructions total, might cluster into 10 phases.

Simulate 1 billion instructions instead of 500 billion, a 500x reduction, and estimate whole-program CPI as the weighted average

$\text{CPI}_{est} = \sum_i w_i \cdot \text{CPI}_i$

Published SimPoint accuracy on CPI is typically within a few percent of the full run, which is entirely good enough for the relative comparisons that Part 3 is doing. The 29-day run becomes about 90 minutes.

6.5 Warmup, which is where samples go wrong

Here is the part people underestimate, and it deserves numbers.

You have picked an interval 300 billion instructions into the program. You cannot execute the preceding 300 billion in detail, that was the whole point. So you fast-forward functionally, which means executing instructions for their architectural effect only, with no timing model, at maybe 100 million instructions per second, and then you switch to detailed mode and start measuring.

The problem is that at the moment you switch, your caches and predictors are empty. The real machine at that point in the program has warm caches, a trained branch predictor, a populated TLB, and a trained prefetcher. Yours has none of it.

Quantify it. Take a 100 million instruction sample with 20 percent loads, so 20 million memory accesses. Consider an 8 MB last-level cache with 64-byte lines, which holds

$\frac{8 \times 2^{20}}{64} = 131{,}072\ \text{lines}$

If the steady-state LLC miss count over that sample would have been about 60,000, then starting cold forces up to 131,072 additional compulsory misses as the cache fills. At a 200-cycle DRAM penalty that is

$131{,}072 \times 200 = 26.2\ \text{million cycles of phantom stall}$

The sample at IPC 2.0 should take about 50 million cycles. You have added 26 million cycles of pure artifact, so the reported IPC is roughly

$\frac{100 \times 10^6}{50 \times 10^6 + 26 \times 10^6} = 1.32$

instead of 2.0. A 34 percent error, entirely from cold state, and it is not a random error, it is a systematic pessimism that will make every large-cache configuration look better than it is because the large cache takes longer to warm. Cold-start bias does not cancel out across configurations, it actively distorts the comparison you are running.

Branch predictors are worse in a different way. A TAGE predictor with tens of thousands of entries needs several million branches to reach steady-state accuracy. A sample that starts with a cleared predictor spends a meaningful fraction of its length training, reporting an inflated misprediction rate throughout.

Three standard defenses exist.

Functional warming. During fast-forward, do not just execute architecturally. Also feed every memory access to the cache models and every branch to the predictor models, updating their state without computing any timing. It is slower than pure fast-forward but far faster than detailed simulation, and it arrives at the sample with genuinely warm structures.

Checkpointing. Save the complete state at the sample boundary once, including architectural state and cache tags, predictor tables, TLB contents, and prefetcher state. Every subsequent run restores from the checkpoint instantly. This is what makes a 180-run sweep tractable, since the expensive warm-up is paid once rather than 180 times. The catch is that a checkpoint of microarchitectural state is only valid for the configuration it was taken on, so a ROB sweep can share checkpoints but a cache-size sweep cannot.

Detailed warming. Run detailed simulation for some millions of instructions before the measurement window opens and discard those results. Simple, correct, and expensive.

The rule to carry into an interview is short. A sample with cold state does not report a slightly wrong answer, it reports a meaningless one, and the error is biased rather than random.

---

07.Part 7, correlation

7.1 What it is and why it is the design engineer's half

The model said the core would run SPECint at IPC 2.19. The RTL exists now. Someone runs a trace through both and the RTL reports 2.02.

That gap is 8 percent. It is far too large to ignore, and somebody has to find out where it went. That is correlation, and it is where an RTL engineer contributes to performance work, because finding the answer means reading RTL, reading waveforms, and reasoning about what the logic actually does cycle by cycle. Architecture background helps. RTL debugging skill is what closes it.

7.2 The method

Correlation is a discipline, not an act of inspiration.

One, run identical stimulus. Same instruction sequence, same starting state, same memory contents. This sounds trivial and is the most common source of false discrepancies, because the model and the RTL testbench usually have completely different mechanisms for loading a program and initializing memory.

Two, compare aggregate cycle count first. If the total matches within a fraction of a percent, you are done for that test. If it does not, you have a number to chase.

Three, compare event counts, not just cycles. Instructions retired, branch mispredicts, L1 misses, L2 misses, stall cycles by source, structure full events, port utilizations. This localizes the discrepancy without any waveform work. If cycles differ by 8 percent but every event count matches, the difference is in latency or arbitration rather than in behavior. If the L2 miss counts differ by 40 percent, you have your answer's neighbourhood.

Four, compare per-instruction retire timestamps. This is the sharpest tool. Emit, from both the model and the RTL, a record of the cycle at which each instruction retired. Then diff them and find the first instruction where the two diverge.