Data Types, Quantization & Precision (FP32, FP16, BF16, INT8)

AI systems trade numerical precision for throughput and memory efficiency. Floating-point types vary in exponent and mantissa allocation: FP32 (8 exp, 23 mantissa), FP16 (5 exp, 10 mantissa), and BF16 (8 exp, 7 mantissa).

Interactive Systems Visualizer

Vectorized SIMD Kernel vs Interpreted Bytecode Loop

Array Size:100,000 elements

Pure Python for-loop--

for x in array:
total += x * 2.0

Per-element PyObject type checks, pointer dereferencing, and evaluation loop overhead.

NumPy Vectorized Kernel--

total = array * 2.0 # AVX-512 SIMD

Single C kernel call executing 8-16 parallel floating-point operations per CPU clock cycle.

Key Systems Engineering Principles

FP32 is standard for training stability; BF16 preserves dynamic range to prevent underflow in deep transformer gradients.
Quantization down to INT8 or INT4 cuts model memory footprint by 4x to 8x, fitting large LLMs onto consumer GPUs.
Subnormal numbers and float16 overflow/underflow cause NaN loss spikes if gradient scaling is omitted.

Mental Model Analogy

“FP32 is a high-resolution 4K architectural blueprint. INT8 is a rough sketch on a napkin: less precise, but 100x lighter to carry and fast enough to build the frame.”

Production Pitfall to Avoid

Casting raw probabilities to FP16 and taking logarithms, resulting in -inf when probabilities round down to true zero.

production_pattern.py

import numpy as np

# Precision comparisons
fp32_val = np.float32(1e-6)
fp16_val = np.float16(1e-6) # Precision lost / subnormal!

print("FP32:", fp32_val)
print("FP16:", fp16_val)

# Memory impact: 7 Billion parameter model
# FP32: 7B * 4 bytes = 28 GB VRAM
# FP16: 7B * 2 bytes = 14 GB VRAM
# INT8: 7B * 1 byte  = 7 GB VRAM
# INT4: 7B * 0.5 byte = 3.5 GB VRAM

Previous Subtopic#8 Strides, Views vs Copies & Memory Layouts Next Subtopic #10 Einstein Summation (einsum) & Matrix Operations

Loading Python for AI Engineering...

Key Systems Engineering Principles

FP32 is standard for training stability; BF16 preserves dynamic range to prevent underflow in deep transformer gradients.

Quantization down to INT8 or INT4 cuts model memory footprint by 4x to 8x, fitting large LLMs onto consumer GPUs.

Subnormal numbers and float16 overflow/underflow cause NaN loss spikes if gradient scaling is omitted.

import numpy as np # Precision comparisons fp32_val = np.float32(1e-6) fp16_val = np.float16(1e-6) # Precision lost / subnormal! print("FP32:", fp32_val) print("FP16:", fp16_val) # Memory impact: 7 Billion parameter model # FP32: 7B * 4 bytes = 28 GB VRAM # FP16: 7B * 2 bytes = 14 GB VRAM # INT8: 7B * 1 byte = 7 GB VRAM # INT4: 7B * 0.5 byte = 3.5 GB VRAM