Lab --- Toolchain Setup: Spike, QEMU, and Cross-Toolchains
August 3, 2026·23 min read·intermediate
The assembler project in Chapter 23 produced raw machine-code bytes from hand-written assembly. The next step is to run those bytes on something that behaves like a RISC-V processor. This lab sets up two…
The assembler project in Chapter 23 produced raw machine-code bytes from hand-written assembly. The next step is to run those bytes on something that behaves like a RISC-V processor. This lab sets up two RISC-V execution environments (Spike and QEMU), the GNU and LLVM cross-compilation toolchains, and walks the reader through compiling a C program, examining its assembly output, running it on both simulators, and single-stepping through individual instructions.
By the end of this lab, the reader will have a complete, local, zero-cost RISC-V development environment that works on macOS, Linux, or Windows. This environment is used throughout Parts III through V whenever the text asks the reader to “compile and run” a RISC-V program or to inspect generated assembly.
01.Setup and Installation
Four components are needed: the RISC-V GNU cross-toolchain (riscv64-unknown-elf-gcc and its companion tools), the Spike ISA simulator, the RISC-V proxy kernel (pk), and QEMU. An optional fifth component is LLVM/Clang for cross- compilation.
macOS (Homebrew)
macOS setup via Homebrew
| # RISC-V GNU toolchain (bare-metal) | |
| brew tap riscv-software-src/riscv | |
| brew install riscv-tools | |
| # The tap provides: | |
| # riscv64-unknown-elf-gcc | |
| # riscv64-unknown-elf-as | |
| # riscv64-unknown-elf-objdump | |
| # riscv64-unknown-elf-gdb | |
| # spike (ISA simulator) | |
| # pk (proxy kernel) | |
| # QEMU with RISC-V support | |
| brew install qemu | |
| # Optional: LLVM (includes Clang with RISC-V target) |
Verify macOS installations
| riscv64-unknown-elf-gcc --version | |
| spike --help 2>&1 | head -1 | |
| qemu-system-riscv32 --version # 32-bit machine emulator | |
| qemu-system-riscv64 --version # 64-bit machine emulator | |
| qemu-riscv64 --version # user-mode |
Linux (Arch as canonical)
Arch Linux setup
| # GNU cross-toolchain (bare-metal) | |
| sudo pacman -S riscv64-elf-gcc riscv64-elf-binutils \ | |
| riscv64-elf-newlib riscv64-elf-gdb | |
| # Spike (from AUR) | |
| yay -S spike | |
| # Proxy kernel (from AUR) | |
| yay -S riscv-pk | |
| # QEMU | |
| sudo pacman -S qemu-system-riscv qemu-user | |
| # Optional: LLVM/Clang | |
| sudo pacman -S clang lld |
For Debian/Ubuntu:
Debian/Ubuntu alternative
| sudo apt install gcc-riscv64-unknown-elf \ | |
| binutils-riscv64-unknown-elf \ | |
| qemu-system-misc qemu-user | |
| # Spike and pk: build from source | |
| # (see github.com/riscv-software-src/riscv-isa-sim | |
| # and github.com/riscv-software-src/riscv-pk) |
For Fedora:
Fedora alternative
| sudo dnf install gcc-riscv64-linux-gnu \ | |
| binutils-riscv64-linux-gnu qemu-system-riscv | |
| # These packages target RISC-V Linux, not bare metal. The | |
| # bare-metal newlib toolchain used by the lab exercises has | |
| # to be built from source, as do Spike and pk. |
Windows (ArchWSL)
The RISC-V toolchain, Spike, and QEMU are Linux-native tools. On Windows, the recommended approach is to install them inside WSL2 with ArchWSL.
Windows setup via ArchWSL
| # Inside the ArchWSL terminal: | |
| sudo pacman -S riscv64-elf-gcc riscv64-elf-binutils \ | |
| riscv64-elf-newlib riscv64-elf-gdb | |
| yay -S spike riscv-pk | |
| sudo pacman -S qemu-system-riscv qemu-user |
Native Windows builds of QEMU exist (installable via winget install SoftwareFreedomConservancy.QEMU), but Spike and the GNU toolchain do not have maintained native Windows installers. The WSL2 path provides the most consistent experience.
LLVM/Clang is an exception: it runs natively on Windows and supports RISC-V as a cross-compilation target out of the box. Install via winget install LLVM.LLVM or download from the LLVM releases page.
02.Lab Exercise 1: Compiling C to RISC-V Assembly
The first exercise compiles a simple C program to RISC-V assembly and examines the output. This connects the high-level C source to the instruction encodings studied in Part II.
The source program
A simple C program for cross-compilation
/* sum.c -- sum the integers 1 through N */
#include <stdio.h>
int sum(int n) {
int total = 0;
for (int i = 1; i <= n; i++) {
total += i;
}
return total;
}
int main(void) {
int result = sum(100);
printf("Sum = %d\n", result);
return 0;
}Compiling to assembly
Compiling to RISC-V assembly with GCC
| # Generate assembly output (-S flag) | |
| riscv64-unknown-elf-gcc -march=rv32im -mabi=ilp32 \ | |
| -O1 -S sum.c -o sum.s | |
| # Inspect the generated assembly | |
| cat sum.s |
The -march=rv32im flag targets the RV32IM instruction set (32-bit base plus multiply/divide). The -mabi=ilp32 flag selects the 32-bit integer ABI. The -O1 optimization level produces readable assembly without excessive register spilling.
Reading the generated assembly
Open sum.s in a text editor and find the sum function. At -O1, GCC typically produces a tight loop using add, addi, bge (or blt), and a return sequence. Identify:
-
The function prologue (saving
raands0–s11to the stack, if any). -
The loop body (the
addthat accumulates the sum and the branch that controls the loop). -
The function epilogue (restoring saved registers and returning via
ret).
Compare the generated code with the calling conventions described in Chapter 19. Which registers does GCC use for the function argument (n), the return value, and the loop counter?
Using LLVM/Clang
Compiling to RISC-V assembly with Clang
| clang --target=riscv32 -march=rv32im -mabi=ilp32 \ | |
| -O1 -S sum.c -o sum_clang.s |
Compare sum.s (GCC output) with sum_clang.s (Clang output). The two compilers often make different register allocation and instruction scheduling decisions. Both outputs are correct but may differ in instruction count and register usage.
03.Lab Exercise 2: Assembling and Linking
Assembling and linking
| # Assemble to an object file | |
| riscv64-unknown-elf-gcc -march=rv32im -mabi=ilp32 \ | |
| -c sum.c -o sum.o | |
| # Link to an ELF executable (bare-metal, using pk) | |
| riscv64-unknown-elf-gcc -march=rv32im -mabi=ilp32 \ | |
| sum.o -o sum.elf | |
| # Disassemble the executable | |
| riscv64-unknown-elf-objdump -d sum.elf | less |
The objdump -d output shows every instruction in the executable, including the C runtime startup code (_start) and the library functions linked in. The sum and main functions appear as labeled sections. Each line shows the address, the hexadecimal encoding, and the disassembled mnemonic.
04.Lab Exercise 3: Running on Spike
Spike is the RISC-V ISA functional simulator. It executes RISC-V instructions one at a time, faithfully implementing the ISA specification. Spike is not cycle-accurate: it does not model pipeline stages, caches, or branch prediction. Its purpose is correctness verification.
Running a program
Running on Spike with the proxy kernel
| # sum.elf is an RV32IM binary, but Spike defaults to RV64, | |
| # so the ISA string has to be given explicitly and pk has to | |
| # be the RV32 build of the proxy kernel. | |
| spike --isa=rv32im pk sum.elf |
The proxy kernel (pk) provides a minimal runtime environment that handles printf, malloc, and program exit by forwarding system calls to the host operating system. The output should print Sum = 5050.
Interactive debugging
Spike supports an interactive debug mode that is invaluable for understanding instruction-level behavior.
Spike interactive debug mode
| spike -d --isa=rv32im pk sum.elf |
At the : prompt, the following commands are available:
Table 1. Spike debug commands
| Command | Action |
|---|---|
run N | Execute N instructions |
reg 0 | Print all integer registers |
reg 0 a0 | Print register a0 |
pc 0 | Print the program counter |
mem 0x80000000 | Print memory at address |
until pc 0 0x80000100 | Run until PC reaches address |
quit | Exit the debugger |
Walkthrough: single-stepping through the sum function. Start Spike in debug mode. Use until pc 0 <addr> to run to the sum function (find the address from the objdump output). Then step one instruction at a time with run 1 and watch the registers change.
After each run 1, print the PC (pc 0) and the registers involved in the current instruction. Trace the loop counter in a0 (or whichever register GCC chose) and the accumulator. Confirm that the final value in the return register is 5050.
05.Lab Exercise 4: Running on QEMU
QEMU provides two modes for RISC-V execution.
User-mode emulation
User-mode QEMU translates RISC-V Linux system calls to host system calls, allowing a RISC-V Linux binary to run directly on the host. This requires a RISC-V Linux executable (compiled with riscv64-unknown-linux-gnu-gcc rather than the bare-metal riscv64-unknown-elf-gcc).
QEMU user-mode emulation
| # Compile for RISC-V Linux (if the Linux toolchain | |
| # is installed) | |
| riscv64-unknown-linux-gnu-gcc -march=rv64gc \ | |
| -static sum.c -o sum_linux | |
| # Run on QEMU user-mode | |
| qemu-riscv64 sum_linux |
The -static flag produces a statically linked binary, avoiding the need for RISC-V shared libraries on the host. The output is the same: Sum = 5050.
System-mode emulation
System-mode QEMU emulates an entire RISC-V machine: CPU, memory, UART, and other devices. It can boot a full Linux kernel or run bare-metal firmware.
QEMU system-mode (bare-metal, no BIOS)
| # sum.elf is a 32-bit binary, so the RV32 machine emulator | |
| # is the right one. An rv64gc build needs qemu-system-riscv64. | |
| qemu-system-riscv32 -machine virt -nographic \ | |
| -bios none -kernel sum.elf |
System-mode emulation is more involved than user-mode because the binary must include its own startup code (or be loaded by pk or OpenSBI). For this lab, user-mode emulation is sufficient. System-mode becomes essential in Part III when the reader builds a CPU and needs a full machine model.
06.Lab Exercise 5: Examining the Binary with objdump
The objdump utility is the bridge between the binary world and the human-readable assembly world. This exercise practices several objdump invocations.
Useful objdump invocations
| # Full disassembly | |
| riscv64-unknown-elf-objdump -d sum.elf | |
| # Disassemble a single function | |
| riscv64-unknown-elf-objdump -d \ | |
| --disassemble=sum sum.elf | |
| # Show section headers (text, data, bss sizes) | |
| riscv64-unknown-elf-objdump -h sum.elf | |
| # Show the symbol table | |
| riscv64-unknown-elf-objdump -t sum.elf | |
| # Intermix source lines with assembly | |
| # (requires -g debug info during compilation) | |
| riscv64-unknown-elf-objdump -dS sum.elf |
Exercise: decode by hand. Pick any three instructions from the objdump output. For each, write down the hexadecimal encoding, identify the format (R, I, S, B, U, J), and manually decode the fields. Confirm that your decoding matches the disassembled mnemonic. This exercise connects the binary encoding work from Chapter 23 to the toolchain output.
07.Lab Exercise 6: Compiler Optimization Effects
Compile sum.c at four optimization levels and compare the generated assembly.
Compiling at different optimization levels
| for opt in O0 O1 O2 O3; do | |
| riscv64-unknown-elf-gcc -march=rv32im \ | |
| -mabi=ilp32 -$opt -S sum.c -o sum_$opt.s | |
| done |
At -O0, the compiler generates straightforward code with every variable stored on the stack. At -O1, most variables live in registers. At -O2 and -O3, the compiler may unroll the loop, use strength reduction (replacing multiplication by repeated addition), or even compute the sum at compile time using the closed-form formula .
Count the number of instructions in the sum function at each level. Record your results in a table. Pay particular attention to -O2 and -O3: if the compiler recognizes the pattern , it may replace the entire loop with a single multiply and shift, reducing the function body to three or four instructions.
The stack frame size is also instructive. At -O0, every local variable (total, i, and the function argument n) is stored on the stack, requiring a sp-adjustment prologue and epilogue. At -O1 and above, the variables live in registers and the stack frame may shrink to zero if the function is a leaf (no calls to other functions).
Table 2. Instruction count versus optimization level (fill in)
| Metric | -O0 | -O1 | -O2 | -O3 |
|---|---|---|---|---|
Instructions in sum | ||||
| Stack frame size (bytes) |
08.Lab Exercise 7: GDB Remote Debugging
Source-level debugging with GDB is the standard workflow for finding bugs in cross-compiled programs. QEMU can act as a GDB server, accepting connections directly from the RISC-V GDB client. Spike reaches GDB by a longer route, described below.
Starting the GDB server
Starting QEMU with a GDB server
| # Start QEMU user-mode with GDB server on port 1234 | |
| qemu-riscv64 -g 1234 sum_linux & | |
| # In a separate terminal, connect GDB | |
| riscv64-unknown-elf-gdb sum_linux | |
| (gdb) target remote :1234 | |
| (gdb) break sum | |
| (gdb) continue |
When the breakpoint hits, GDB shows the current instruction. The reader can inspect registers (info registers), examine memory (x/4x $sp), single-step (stepi), and set watchpoints on memory addresses.
Spike GDB connection
Spike does not implement the GDB remote protocol itself. What it exposes is a remote-bitbang JTAG port, which OpenOCD attaches to and then presents to GDB as a conventional GDB server. The connection is therefore a three-program chain of Spike, OpenOCD, and GDB rather than the direct target remote attachment that QEMU supports.
Spike with a remote-bitbang JTAG port
| # Start Spike with its remote-bitbang JTAG port on 9824 | |
| spike -H --rbb-port=9824 --isa=rv32im pk sum.elf & | |
| # OpenOCD attaches to port 9824 and publishes a GDB server | |
| # of its own, which GDB then connects to. For simple | |
| # inspection, spike's built-in -d debug mode is easier. |
For most debugging tasks in this lab, Spike’s built-in interactive debugger (the -d flag from Exercise 3) is simpler than the GDB remote protocol. GDB becomes essential in Part III when debugging CPU designs that run on QEMU’s system-mode emulator.
09.Lab Exercise 8: Compressed Instructions in Practice
Compile with the C extension enabled and compare the binary size.
Comparing RV32IM versus RV32IMC
| # Without C extension | |
| riscv64-unknown-elf-gcc -march=rv32im -mabi=ilp32 \ | |
| -O2 sum.c -o sum_no_c.elf | |
| # With C extension | |
| riscv64-unknown-elf-gcc -march=rv32imc -mabi=ilp32 \ | |
| -O2 sum.c -o sum_c.elf | |
| # Compare sizes | |
| riscv64-unknown-elf-size sum_no_c.elf sum_c.elf |
Disassemble both executables and count the 16-bit instructions (they appear with only 4 hex digits in the encoding column of objdump output, compared to 8 hex digits for 32-bit instructions). What fraction of instructions are compressed?
10.Lab Exercise 9: Examining the Symbol Table and Sections
Every ELF executable contains metadata beyond the machine code. This exercise explores the symbol table and section layout.
Examining symbols and sections
| # Section headers: text, data, bss, rodata, etc. | |
| riscv64-unknown-elf-readelf -S sum.elf | |
| # Symbol table: functions, global variables, sizes | |
| riscv64-unknown-elf-readelf -s sum.elf | |
| # Program headers: load segments, entry point | |
| riscv64-unknown-elf-readelf -l sum.elf | |
| # Hex dump of the .rodata section (string literals) | |
| riscv64-unknown-elf-readelf -x .rodata sum.elf |
The readelf output reveals several things that objdump does not show clearly. The section headers table lists every section with its type, address, offset in the file, and size. The .text section contains the machine code. The .rodata section contains read-only data (string literals like "Sum = %d\n"). The .bss section reserves space for uninitialized global variables without occupying space in the file.
The symbol table lists every function and global variable with its address, size, binding (local or global), and section. The sum function should appear as a global symbol in the .text section. Its size field tells how many bytes the function occupies.
11.Understanding the Toolchain Pipeline
A cross-compilation toolchain is a pipeline with four stages, and understanding how the stages connect prevents a large class of common errors.
Stage 1: Preprocessor. The C preprocessor (cpp) expands #include directives, #define macros, and conditional compilation blocks. The output is a single preprocessed .i file with no directives remaining. Run riscv64-unknown-elf-gcc -E sum.c -o sum.i to see the preprocessed output.
Stage 2: Compiler. The compiler (cc1 inside GCC, or clang in LLVM) translates the preprocessed C source into RISC-V assembly. The -S flag stops after this stage. The output is a human-readable .s file.
Stage 3: Assembler. The assembler (riscv64-unknown-elf-as, or the integrated assembler in LLVM) translates the assembly into an ELF object file (.o). This is the stage that the project in Chapter 23 reimplemented from scratch. The -c flag tells gcc to stop after assembly.
Stage 4: Linker. The linker (riscv64-unknown-elf-ld, invoked automatically by gcc) combines one or more object files with the C runtime startup code and libraries (newlib for bare-metal, glibc for Linux) to produce a final ELF executable.
Running each stage separately
| # Stage 1: Preprocess | |
| riscv64-unknown-elf-gcc -E sum.c -o sum.i | |
| # Stage 2: Compile to assembly | |
| riscv64-unknown-elf-gcc -march=rv32im -mabi=ilp32 \ | |
| -O2 -S sum.i -o sum.s | |
| # Stage 3: Assemble to object file | |
| riscv64-unknown-elf-as -march=rv32im sum.s -o sum.o | |
| # Stage 4: Link | |
| riscv64-unknown-elf-gcc -march=rv32im -mabi=ilp32 \ | |
| sum.o -o sum.elf |
Running the stages separately is useful for debugging. If the generated assembly looks wrong, the bug is in Stage 2 (compiler flags or source code). If the object file has wrong encodings, the bug is in Stage 3. If the executable crashes at startup before reaching main, the bug is in Stage 4 (a missing library, a wrong linker script, or an ABI mismatch).
12.Common Pitfalls
Architecture and ABI mismatch. Compiling with -march=rv32im but linking with a 64-bit library (or vice versa) produces a linker error about incompatible ELF classes. The -march and -mabi flags must be consistent across all stages. For 32-bit: -march=rv32im -mabi=ilp32. For 64-bit: -march=rv64gc -mabi=lp64d.
Missing pk for Spike. Running spike sum.elf without pk in front does not fail at load time. Spike reads the ELF program headers and jumps to the entry point exactly as it does for pk itself, which is also an ELF executable. The failure comes later. With no proxy kernel underneath it, nothing services the newlib system calls that printf, malloc, and program exit issue, so the ecall traps into machine mode with no handler installed and the program aborts or hangs instead of printing. Always run programs that use C library functions under the proxy kernel.
Static versus dynamic linking. QEMU user-mode needs access to the target’s shared libraries if the binary is dynamically linked. The simplest workaround is to compile with -static, which embeds the entire C library into the executable. The binary is larger but runs anywhere without library dependencies.
Floating-point ABI confusion. The ilp32 ABI passes floats in integer registers (soft float). The ilp32f and ilp32d ABIs pass floats in floating-point registers. Mixing ABIs across object files produces silent data corruption (a float value is written to an integer register in one function and read from a float register in the callee). Always use the same -mabi flag for all compilation units.
13.Looking Ahead
This lab chapter completes Part II of the book. The reader now has a full understanding of instruction set architectures (what instructions exist, how they are encoded, how the ISA organizes privilege levels and exceptions, and how vector and SIMD extensions exploit data-level parallelism) and a working cross-compilation and simulation environment for RISC-V.
Part III turns to the hardware that executes these instructions. Chapter 25 develops the single-cycle RISC-V datapath block by block, and Chapter 26 turns the instruction encodings from Part II into the control decoder that steers it. Chapter 28 introduces pipelining, and Chapter 30 tackles the hazards that arise when instructions overlap in the pipeline. Chapter 34 pulls the sequence together into a pipelined RV32IM core written in Chisel. The Spike and QEMU environments from this lab become the reference against which the reader verifies the CPU designs in the project chapters of Part III.