Part I·Architectural Foundations·Chapter 8 of 66

Part IArchitectural Foundations

Floating-Point Arithmetic

August 3, 2026·18 min read·beginner

Integer arithmetic handles whole numbers. Scientific computation, graphics, machine learning, and signal processing demand numbers with fractional parts and a dynamic range spanning many orders of magnitude: from the mass of an electron ( $9.1 \times 10^{-31}$ kg) to the distance to the nearest galaxy ( $2.5 \times 10^{22}$ m). No fixed-width integer format can cover that range without wasting almost all of its bits on unused positions. The solution is floating-point representation, where the radix point moves (“floats”) under the control of an exponent field, concentrating the available bits around the significant digits of the value.

This chapter develops the IEEE 754 floating-point standard from first principles: the encoding of sign, exponent, and significand into a fixed-width bit pattern, the special values (zeros, infinities, NaNs, denormals), the five rounding modes, and the hardware that adds, multiplies, divides, and fuses multiply-add operations on floating-point operands. It closes with a survey of the newer reduced-precision formats (BFloat16, FP8, TensorFloat-32) that are reshaping machine-learning hardware.

01.Why Floating Point?

A 32-bit unsigned integer can represent values from $0$ to $2^{32} - 1 \approx 4.3 \times 10^{9}$ . Suppose a scientist needs to represent $6.02 \times 10^{23}$ (Avogadro’s number). A plain 32-bit integer cannot reach it. Neither can a 64-bit integer ( $2^{64} \approx 1.8 \times 10^{19}$ is still too small), and even a 128-bit integer ( $2^{128} \approx 3.4 \times 10^{38}$ ), which has the range, has no way to represent the fractional part of $6.02214076 \times 10^{23}$ .

Fixed-point arithmetic places the radix point at a predetermined position within the integer, dedicating some bits to the integer part and the remaining bits to the fractional part. A 32-bit Q16.16 format, for example, gives 16 integer bits and 16 fractional bits, covering the range $[0, 65535]$ with a resolution of $2^{-16} \approx 1.5 \times 10^{-5}$ . Fixed point is fast (it uses the same adder and multiplier as integer arithmetic) but inflexible: the programmer must choose the radix point position at design time, and values that fall outside the chosen range overflow silently.

Floating-point representation separates the question of range from the question of precision. A 32-bit IEEE 754 single-precision number covers a range from roughly $\pm 1.2 \times 10^{-38}$ to $\pm 3.4 \times 10^{38}$ with about 7 decimal digits of precision. The cost is more complex hardware for every arithmetic operation, since the exponents must be aligned before addition and renormalized after every operation.

02.The IEEE 754 Standard

The IEEE 754 standard [1] defines the encoding, arithmetic, and exception model for floating-point computation. Before IEEE 754, every manufacturer had its own floating-point format and rounding rules, making it impossible to port numerical software between machines and expect the same results. The standard was first published in 1985, revised in 2008 to add decimal formats and fused multiply-add, and revised again in 2019 with minor clarifications.

Binary encoding

An IEEE 754 binary floating-point number is stored as three fields packed into a fixed-width word:

$\text{value} = (-1)^{s} \times 1.f \times 2^{e - \text{bias}}$

where $s$ is the sign bit (0 for positive, 1 for negative), $f$ is the fraction (the explicit bits of the significand), and $e$ is the biased exponent. The leading $1$ in $1.f$ is implicit and is not stored, gaining one extra bit of precision for free.

Table 1. IEEE 754 binary formats. “Sig bits” counts the implicit leading 1 plus the explicit fraction bits.

Name	Bits	Sign	Exponent	Fraction	Bias	Sig bits
Half (binary16)	16	1	5	10	15	11
Single (binary32)	32	1	8	23	127	24
Double (binary64)	64	1	11	52	1023	53
Quad (binary128)	128	1	15	112	16383	113

Special values

IEEE 754 reserves two exponent-field patterns for special values:

Table 2. Special value encoding in IEEE 754. e_{} denotes the all-ones exponent field.

Exponent	Fraction	Meaning
$0$	$0$	$\pm 0$ (positive or negative zero)
$0$	$\neq 0$	Denormal (subnormal) number
$e_{\max}$	$0$	$\pm \infty$
$e_{\max}$	$\neq 0$	NaN (Not a Number)

Zeros. IEEE 754 has both $+0$ and $-0$ . They compare as equal ( $+0 = -0$ ) but produce different results when used as divisors: $1 / (+0) = +\infty$ and $1 / (-0) = -\infty$ .

Denormals (subnormals). When the exponent field is zero and the fraction is nonzero, the implicit leading bit is $0$ instead of $1$ , and the true exponent is $1 - \text{bias}$ (not $0 - \text{bias}$ ). This creates a set of evenly spaced values between zero and the smallest normal number, preventing an abrupt gap (called the “underflow gap”) around zero. Denormal arithmetic is slower on many processors because the hardware datapath is optimized for the normal case where the leading bit is $1$ .

Infinities. Infinities result from overflow (a finite computation that exceeds the maximum representable value) or from explicit division by zero. They propagate through arithmetic: $\infty + 5 = \infty$ , $\infty \times (-3) = -\infty$ .

NaN (Not a Number). NaN results from mathematically undefined operations: $0 / 0$ , $\infty - \infty$ , $\sqrt{-1}$ . IEEE 754 distinguishes quiet NaN (qNaN), which propagates through computations without raising an exception, from signaling NaN (sNaN), which raises an invalid-operation exception when used as an operand.

Rounding

The exact result of a floating-point operation can have more significant bits than the destination format provides. The result must be rounded to the nearest representable value. IEEE 754 defines five rounding modes:

Round to nearest, ties to even (default). If the exact result is equidistant between two representable values, round to the one whose least significant bit is $0$ . This avoids systematic bias in long computation chains.
Round toward $+\infty$ (ceiling). Always round toward positive infinity.
Round toward $-\infty$ (floor). Always round toward negative infinity.
Round toward zero (truncation). Always round toward zero.
Round to nearest, ties away from zero. If equidistant, round away from zero. Required only for decimal formats.

To implement rounding, the hardware computes three extra bits beyond the target precision:

Guard bit ( $G$ ). The first bit beyond the target precision.
Round bit ( $R$ ). The second bit beyond the target precision.
Sticky bit ( $S$ ). The OR of all remaining bits beyond the round bit. The sticky bit records whether any precision was lost, which is needed to break ties.

03.Floating-Point Addition and Subtraction

Adding two floating-point numbers is more involved than adding two integers because the exponents must be aligned before the significands can be added.

The hardware pipeline for floating-point addition has four stages:

Exponent comparison. Compute the difference of the two exponents $\Delta = e_{A} - e_{B}$ . The number with the smaller exponent must be shifted.
Alignment shift. Shift the significand of the smaller-exponent operand right by $|\Delta|$ positions, filling the vacated positions with zeros and tracking the shifted-out bits in the sticky bit.
Significand addition or subtraction. Add (or subtract, if the signs differ) the aligned significands using a fixed-point adder.
Normalization and rounding. If the result’s leading bit is not in the expected position, shift the result left or right (adjusting the exponent accordingly) until the leading $1$ is in the implicit-bit position. Then round using the guard, round, and sticky bits.

The alignment shifter is a barrel shifter (from Chapter 5) whose shift amount is the exponent difference. For double precision, this is a 52-position barrel shifter, which is a significant piece of hardware. The normalization step uses a leading-zero counter followed by a left-shift, both of which also use barrel-shifter-like structures.

04.Floating-Point Multiplication

Floating-point multiplication is structurally simpler than addition because no alignment shift is needed. The algorithm is:

Sign. The result sign is $s_{A} \oplus s_{B}$ .
Exponent. Add the true exponents: $e_{\text{result}} = (e_{A} - \text{bias}) + (e_{B} - \text{bias}) + \text{bias} = e_{A} + e_{B} - \text{bias}$ .
Significand. Multiply the two significands (including the implicit leading bits) using the integer multiplier of Chapter 7. The product of two $p$ -bit significands is a $2p$ -bit value.
Normalization. The product $1.xxx \times 1.yyy$ is either $1.zzz$ or $10.zzz$ (at most one bit of right shift is needed). Adjust the exponent if the product’s integer part is $10$ .
Rounding. Round the $2p$ -bit product to $p$ bits using the guard, round, and sticky bits.

The significand multiplication reuses the same Booth-recoded Wallace or Dadda tree multiplier described in Chapter 7. For single precision the significand multiplier is $24 \times 24$ bits. For double precision it is $53 \times 53$ bits. In many processor designs the same physical multiplier serves both integer and floating-point operations, with a mux selecting the appropriate input and output widths.

05.Floating-Point Division and Square Root

Floating-point division computes $A / B$ by dividing the significands and subtracting the exponents. The significand division uses the SRT or Newton-Raphson algorithm from Chapter 7, with the exponent handling layered on top.

Square root is structurally similar to division. An SRT-like algorithm (sometimes called a “digit-recurrence” square root) retires one or two root bits per cycle by maintaining a partial radicand and updating it with trial subtractions. Newton-Raphson can also compute $1 / \sqrt{x}$ (the reciprocal square root) via the iteration $y_{n+1} = \frac{1}{2} y_{n}(3 - x \cdot y_{n}^{2})$ , and the final result $\sqrt{x} = x \cdot (1/\sqrt{x})$ requires one additional multiplication.

Division and square root are the slowest floating-point operations. A typical latency is 10 to 30 cycles for double precision, compared with 3 to 5 cycles for multiply and 3 to 4 cycles for add. Because divisions are relatively rare in most workloads (typically 1 to 3 percent of floating-point instructions), most processors use an iterative non-pipelined divider that occupies less area than a fully pipelined unit.

06.Fused Multiply-Add

The fused multiply-add (FMA) computes $a \times b + c$ as a single operation with a single rounding step at the end. Without the FMA, the computation would require a separate multiply (with rounding) followed by a separate add (with rounding), for two rounding errors instead of one.

The FMA is the single most important floating-point instruction in modern processors. It is the primitive for dot products, matrix multiplications, polynomial evaluations (Horner’s method), and Newton-Raphson iterations. Every major ISA provides an FMA instruction: FMADD in RISC-V, FMADD in ARMv8, VFMADD in x86-64 AVX/FMA.

The hardware for an FMA is essentially a multiplier whose $2p$ -bit product is not rounded before being fed into an adder that adds the third operand $c$ . The unrounded product has $2p$ bits of precision, so the adder and alignment shifter must handle a wider intermediate value than a standalone floating-point adder. The single rounding step at the end produces a result that is correctly rounded as if computed to infinite precision and then rounded once.

07.Reduced-Precision and Emerging Formats

The standard IEEE 754 formats (binary16, binary32, binary64, binary128) were designed for scientific and general-purpose computation where accuracy matters. Machine-learning training and inference have different requirements: the workload is dominated by matrix multiplications, the individual values are noisy (they come from stochastic gradient descent), and throughput matters more than per-element precision. This has driven the creation of several reduced-precision formats that sacrifice significand bits for smaller hardware and higher throughput.

BFloat16

BFloat16 (Brain Floating Point 16) is a 16-bit format with 1 sign bit, 8 exponent bits, and 7 explicit fraction bits, giving 8 significand bits once the implicit leading $1$ is counted. It has the same exponent range as binary32 (8-bit exponent, bias 127) but less than half the significand precision (8 vs 24 bits). The advantage is that any binary32 value can be converted to BFloat16 by simply truncating the lower 16 bits of the significand, which makes hardware conversion trivial.

BFloat16 is supported by Google TPUs, Intel AMX, ARM SME, AMD CDNA, and NVIDIA Ampere and later architectures.

FP8 formats

FP8 refers to a family of 8-bit floating-point formats proposed for machine-learning inference. Two variants are common:

E4M3: 1 sign, 4 exponent, 3 fraction bits. Larger significand for better precision.
E5M2: 1 sign, 5 exponent, 2 fraction bits. Larger exponent range for better dynamic range.

NVIDIA Hopper implements both E4M3 and E5M2 in its Tensor Cores, using E4M3 for the forward pass (where precision matters) and E5M2 for the backward pass (where gradient magnitudes vary widely).

TensorFloat-32

TensorFloat-32 (TF32) is a 19-bit format used internally by NVIDIA Ampere and later Tensor Cores. It has 1 sign bit, 8 exponent bits, and 10 fraction bits. TF32 is not a storage format. Tensor Cores accept binary32 inputs, internally truncate the significand to 10 bits, multiply in TF32, and accumulate in binary32. The effect is the dynamic range of binary32 with the throughput of a format roughly the size of binary16.

Fixed point revisited

For inference on edge devices (phones, microcontrollers, IoT sensors), even FP8 can be too expensive. Quantized inference uses 8-bit or 4-bit integers (INT8, INT4) with per-tensor or per-channel scale factors. The arithmetic is plain integer multiplication and addition, which is the cheapest possible hardware. The scale factors are floating-point but are applied once per tensor, not per element. ARM Cortex-M processors, RISC-V cores with the vector extension and its widening integer multiply-accumulate operations, and dedicated NPU accelerators (Chapter 96) rely heavily on quantized integer inference.

08.Looking Ahead

This chapter has developed the IEEE 754 floating-point representation and the hardware for addition, multiplication, division, and fused multiply-add. Together with the integer arithmetic of Chapter 7, the arithmetic unit of a processor is now complete. The next chapter steps down one level of abstraction, from the gate-level and block-level circuits of Chapters 4 through 8 to the CMOS transistors that implement those gates in silicon.

09.Worked Examples

Example: IEEE 754 Encoding

Encode the decimal value $-6.625$ as an IEEE 754 single-precision (binary32) floating-point number.

Solution. Step 1: Sign bit $s = 1$ (negative).

Step 2: Convert $6.625$ to binary. $6 = 110_{2}$ . $0.625 = 0.101_{2}$ (because $0.5 + 0.125 = 0.625$ ). So $6.625 = 110.101_{2}$ .

Step 3: Normalize. $110.101_{2} = 1.10101_{2} \times 2^{2}$ .

Step 4: Exponent. True exponent $= 2$ . Biased exponent $= 2 + 127 = 129 = 10000001_{2}$ .

Step 5: Fraction. The significand is $1.10101$ . The implicit leading $1$ is not stored. Fraction $= 10101000000000000000000$ (23 bits, padded with zeros).

Final encoding: 1.

Example: Floating-Point Addition

Compute $1.5 + 0.375$ in IEEE 754 single precision, showing all four pipeline stages.

Solution.

$1.5 = 1.1_{2} \times 2^{0}$ , exponent $= 0 + 127 = 127$ .

$0.375 = 1.1_{2} \times 2^{-2}$ , exponent $= -2 + 127 = 125$ .

1. Exponent comparison. $\Delta = 127 - 125 = 2$ . Shift $0.375$ ’s significand right by 2.

2. Alignment shift. $0.375$ : $1.1_{2} \to 0.011_{2}$ (shifted right by 2). No bits shifted out, so sticky $= 0$ .

3. Addition. $1.100 + 0.011 = 1.111_{2}$ .

4. Normalization. The result $1.111_{2} \times 2^{0}$ is already normalized. No rounding needed (no bits beyond the significand).

Result: $1.111_{2} = 1 + 0.5 + 0.25 + 0.125 = 1.875$ . Check: $1.5 + 0.375 = 1.875$ . Correct.

Example: Floating-Point Multiplication

Compute $3.0 \times 2.5$ in IEEE 754, showing the sign, exponent, and significand steps.

Solution.

$3.0 = 1.1_{2} \times 2^{1}$ , $s_{A} = 0$ , $e_{A} = 128$ .

$2.5 = 1.01_{2} \times 2^{1}$ , $s_{B} = 0$ , $e_{B} = 128$ .

Sign. $s = 0 \oplus 0 = 0$ (positive).

Exponent. $e = 128 + 128 - 127 = 129$ .

Significand. $1.1_{2} \times 1.01_{2}$ :

$\begin{array}{ccccc} & & 1 & 1 & 0 \\ \times & & 1 & 0 & 1 \\ \hline & & 1 & 1 & 0 \\ & 0 & 0 & 0 & \\ 1 & 1 & 0 & & \\ \hline 1 & 1 & 1 & 1 & 0 \\ \end{array}$

Product significand $= 1.1110_{2}$ . Since the integer part is $1$ (not $10$ ), no extra normalization is needed.

Round. $1.1110_{2}$ has 4 significant bits after the point. For single precision (23 fraction bits), no rounding is needed.

Result: $1.1110_{2} \times 2^{2} = 111.10_{2} = 7.5$ . Check: $3.0 \times 2.5 = 7.5$ . Correct.

Example: Guard, Round, and Sticky Bits

Two single-precision numbers are added. After alignment, the shifted operand’s significand is $0.00000000000000000000001\;1\;0\;1\ldots$ , where the last three annotated bits are guard ( $G = 1$ ), round ( $R = 0$ ), and the remaining bits contribute to sticky ( $S = 1$ because at least one shifted-out bit is $1$ ). After the addition, the unrounded result in the target precision is $1.11111111111111111111111$ (23 fraction bits). Should the result be rounded up?

Solution. The guard bit is $1$ , so the exact result is more than halfway between two representable values only if $R = 1$ or $S = 1$ . Here $R = 0$ but $S = 1$ , so the result is strictly greater than halfway. Under round-to-nearest-even, round up. The rounded result increments the last fraction bit: the result becomes $10.00000000000000000000000 \times 2^{e}$ , which requires a one-position right shift and an exponent increment.

10.Exercises

Exercises

Decode binary32. The 32-bit pattern 0 represents what decimal value?
Encode binary64. Encode the decimal value $-0.1$ as an IEEE 754 double-precision (binary64) number. Show the repeating binary fraction and the rounding that occurs.
Denormal range. What is the smallest positive denormal value in IEEE 754 binary32? What is the smallest positive normal value? Express both in decimal.
Infinity arithmetic. Evaluate each expression under IEEE 754 rules: (a) $+\infty + 5$ , (b) $-\infty \times -\infty$ , (c) $+\infty - \infty$ , (d) $0 \times +\infty$ .
NaN propagation. If $x$ is a quiet NaN, what is $x + 1$ ? What is $x > 0$ ? What is $x = x$ ?
Addition pipeline. Add $1.0 \times 2^{10}$ and $1.0 \times 2^{1}$ in binary32. Show the exponent comparison, alignment shift, significand addition, and normalization steps. How many significant bits of the smaller operand survive after alignment?
Catastrophic cancellation. Compute $1.0000001 - 1.0000000$ in a hypothetical 8-significant-decimal-digit format. How many significant digits remain in the result? Explain why subtracting nearly equal numbers destroys precision.
FMA advantage. Compute $a \times b + c$ for $a = 1.00000001$ , $b = 1.00000001$ , $c = -1.00000002$ in a hypothetical 9-significant-digit decimal format. Show that the separately rounded multiply-then-add gives a different result from the fused multiply-add.
BFloat16 range. What is the largest finite value representable in BFloat16? What is the smallest positive normal value? Compare with binary16 (half precision).
FP8 precision. In the E4M3 FP8 format, how many distinct finite positive values exist? What is the spacing between 1.0 and the next larger representable value?
Rounding modes. The exact result of an addition is $1.10101010101010101010101\;1\;1$ (23 fraction bits followed by guard $= 1$ and round $= 1$ , sticky $= 0$ ). What is the rounded result under each of the four binary rounding modes?
Multiplier sharing. Explain why a single $64 \times 64$ -bit integer multiplier can serve both 64-bit integer multiply and the $53 \times 53$ -bit significand multiply of double-precision floating-point multiply. What additional hardware is needed to handle the floating-point exponent and sign?

References

[1]IEEE (2019). “IEEE.” IEEE. doi:10.1109/IEEESTD.2019.8766229

Book mode