memory • Intermediate

String Memory: UTF-8 vs UTF-16, Code Points & Graphemes

Strings in modern systems use variable-length encodings: UTF-8 (1 to 4 bytes per code point) vs UTF-16 (2 or 4 bytes via surrogate pairs). Indexing by byte length != indexing by visual character glyphs.

Intuitive Mental Model

The Accordion Files: ASCII letters take 1 thin folder (1 byte); emojis take 4 expanded folders (4 bytes). Asking for "character #3" by raw byte offset can slice an emoji in half.

C / TypeScript ImplementationHardware & Algorithmic Standard

// JavaScript UTF-16 Surrogate Pair Gotcha:
const emoji = "🔥";
console.log(emoji.length); // 2 (UTF-16 code units, NOT 1!)
console.log([...emoji].length); // 1 (True Unicode code point!)

// In Rust / Go (UTF-8 bytes):
// "🔥".len() == 4 bytes

Key Architectural Takeaways

•UTF-8 Efficiency: Backwards compatible with 7-bit ASCII, using 1 byte for English text.
•Grapheme Clusters: Complex emojis (family emojis, flags) combine multiple code points via Zero Width Joiners (ZWJ).

Common Coding Mistake

Using s.substring(0, N) on strings containing emojis, creating corrupted surrogate half-characters.

Optimal Solution

Use Intl.Segmenter or Array.from(s) for true Unicode character segmentation.

Next: Knuth-Morris-Pratt (KMP) String Pattern Matching & LPS Array