Terabyte per second to Gibibit per second

Large language models have billions of parameters that must be read from memory for every inference pass. An LLM with 70 billion parameters at 16-bit precision needs 140 GB of data read per forward pass. At 3 TB/s, the H100 can perform roughly 20 inference passes per second — bandwidth directly determines tokens-per-second output.

During LLM inference each token requires reading all model weights from memory. A 70-billion-parameter model at 16-bit precision means 140 GB read per forward pass. At 30 tokens per second, that is 4.2 TB/s of memory reads — right at the limit of an H100's HBM3. This is why AI inference is "memory-bound": the GPU's compute cores sit idle waiting for data. Quantising weights to 8-bit or 4-bit halves or quarters the bandwidth demand, directly increasing tokens per second.

The NVIDIA B200 GPU with HBM3e achieves approximately 8 TB/s of memory bandwidth as of 2025. Each generation roughly doubles bandwidth — from 2 TB/s (A100) to 3.35 TB/s (H100) to 4.8 TB/s (H200) to 8 TB/s (B200). The trajectory suggests 16+ TB/s within a few years.

About 16.7 minutes. A petabyte is 1,000 terabytes, so at 1 TB/s, the math is simple division. For context, the Library of Congress contains roughly 10–20 petabytes of data. Transferring it all at 1 TB/s would take about 3–6 hours.

Yes — petabytes per second (PB/s). Experimental optical interconnects and photonic computing architectures are pushing toward PB/s-class bandwidth. Some supercomputer storage systems already aggregate into the PB/s range when all nodes operate simultaneously. It is the next frontier for AI training clusters.

At gibibit speeds, 7.4% represents a substantial amount of data. The difference between 10 Gibps and 10 Gbps is 737 Mbps — enough bandwidth for several 4K video streams. When designing storage fabrics or HPC interconnects, misinterpreting the unit can lead to underprovisioned systems.

PCIe specifications are actually defined in GT/s (gigatransfers per second) with specific encoding overhead. PCIe 3.0 uses 128b/130b encoding at 8 GT/s, giving about 985 MB/s per lane — which is closer to binary GiB/s than decimal GB/s. The industry uses both units somewhat loosely.

InfiniBand specifications use decimal rates (HDR = 200 Gbps, NDR = 400 Gbps per port). However, some HPC benchmarks and documentation convert to binary units for consistency with memory bandwidth figures. Always check the document's unit convention to avoid the 7% discrepancy.

Ordering a 100 Gibps fabric when you needed 100 Gbps means overpaying for 7.4% more bandwidth than necessary. Conversely, provisioning 100 Gbps when your workload needs 100 Gibps leaves you 7.4% short, potentially causing congestion during peak loads. At data center scale, these margins translate to real money.

Unlikely. Networking is firmly decimal (Ethernet, fiber optics), while memory and storage have binary roots. The two worlds overlap in storage networking, causing permanent confusion. The best practice is to always explicitly state "decimal" or "binary" in specifications rather than hoping everyone agrees.