Why GPU Memory is the Real Bottleneck in AI Infrastructure

The conversation around AI infrastructure focuses on FLOPS and GPU count, but in practice memory is what determines what models you can run. A 70B parameter model needs at least 140GB of GPU memory in FP16, far exceeding what a single GPU offers — and this constraint shapes nearly every infrastructure decision.