Terabyte per second to Gigabyte per second

Large language models have billions of parameters that must be read from memory for every inference pass. An LLM with 70 billion parameters at 16-bit precision needs 140 GB of data read per forward pass. At 3 TB/s, the H100 can perform roughly 20 inference passes per second — bandwidth directly determines tokens-per-second output.

During LLM inference each token requires reading all model weights from memory. A 70-billion-parameter model at 16-bit precision means 140 GB read per forward pass. At 30 tokens per second, that is 4.2 TB/s of memory reads — right at the limit of an H100's HBM3. This is why AI inference is "memory-bound": the GPU's compute cores sit idle waiting for data. Quantising weights to 8-bit or 4-bit halves or quarters the bandwidth demand, directly increasing tokens per second.

The NVIDIA B200 GPU with HBM3e achieves approximately 8 TB/s of memory bandwidth as of 2025. Each generation roughly doubles bandwidth — from 2 TB/s (A100) to 3.35 TB/s (H100) to 4.8 TB/s (H200) to 8 TB/s (B200). The trajectory suggests 16+ TB/s within a few years.

About 16.7 minutes. A petabyte is 1,000 terabytes, so at 1 TB/s, the math is simple division. For context, the Library of Congress contains roughly 10–20 petabytes of data. Transferring it all at 1 TB/s would take about 3–6 hours.

Yes — petabytes per second (PB/s). Experimental optical interconnects and photonic computing architectures are pushing toward PB/s-class bandwidth. Some supercomputer storage systems already aggregate into the PB/s range when all nodes operate simultaneously. It is the next frontier for AI training clusters.

CPUs constantly shuttle data between RAM and their caches. DDR5-6000 provides about 96 GB/s of bandwidth in dual-channel mode. In games, insufficient RAM bandwidth causes frame drops during complex scenes. In productivity tasks like video encoding, it directly limits how fast the CPU can process data.

Thunderbolt 4 runs at 40 Gbps, which is 5 GB/s. Thunderbolt 5, released in 2024, doubles this to 80 Gbps (10 GB/s) with a burst mode up to 120 Gbps (15 GB/s). This is fast enough to run an external NVMe SSD at near-internal speeds.

Both, depending on generation. A PCIe 3.0 ×4 interface caps at ~3.5 GB/s, bottlenecking modern NAND. PCIe 4.0 ×4 raises this to ~7 GB/s, and PCIe 5.0 ×4 to ~14 GB/s. The drive's NAND flash and controller also have limits — the fastest SSDs and the fastest interfaces are in a constant leapfrog.

GPUs use wide memory buses (256–384 bits) with very fast HBM or GDDR6X memory running at high clock speeds. An RTX 4090 has a 384-bit bus with GDDR6X at 21 Gbps per pin, totalling 1,008 GB/s. HBM3 in data center GPUs achieves 3,000+ GB/s through stacked memory with 4096-bit buses.

At multi-GB/s rates, CPU processing speed, software efficiency, and thermal throttling become bottlenecks. A 14 GB/s PCIe 5.0 SSD can deliver data faster than most applications can consume it. Decompression, parsing, and memory allocation in software often cannot keep up with raw storage bandwidth.