Contiguous C-Memory Arrays vs Python Lists

NumPy and PyTorch tensors allocate single contiguous blocks of memory where elements sit adjacent to each other. Unlike Python pointer lists, contiguous memory fits into CPU L1/L2 cache lines and enables single-instruction multiple-data (SIMD) vector processing.

Interactive Systems Visualizer

Vectorized SIMD Kernel vs Interpreted Bytecode Loop

Array Size:100,000 elements

Pure Python for-loop--

for x in array:
total += x * 2.0

Per-element PyObject type checks, pointer dereferencing, and evaluation loop overhead.

NumPy Vectorized Kernel--

total = array * 2.0 # AVX-512 SIMD

Single C kernel call executing 8-16 parallel floating-point operations per CPU clock cycle.

Key Systems Engineering Principles

A contiguous array stores raw unboxed numbers sequentially in virtual memory (stride = sizeof(dtype)).
CPU hardware prefetchers load contiguous data effortlessly into cache lines (64 bytes at a time).
Python lists cause pointer-chasing: each element requires fetching an arbitrary heap pointer address, causing CPU cache misses.

Mental Model Analogy

“A contiguous array is a roll of pennies where every coin is touching. A Python list is a box of scavenger hunt clues, each pointing to a coin hidden in a different room.”

Production Pitfall to Avoid

Using Python for-loops to iterate over arrays element-by-element, throwing away 100x hardware vector speedups.

production_pattern.py

import numpy as np

# Contiguous block: exactly 4,000,000 bytes in memory
arr = np.ones(1_000_000, dtype=np.float32)

print("Flags:", arr.flags['C_CONTIGUOUS']) # True
print("Itemsize:", arr.itemsize)            # 4 bytes
print("Total bytes:", arr.nbytes)           # 4,000,000 bytes

Previous Subtopic#5 Memory Profiling, tracemalloc & Zero-Copy Views Next Subtopic #7 Vectorized Operations & Broadcasting Rules

Loading Python for AI Engineering...