Throughput

In Depth

Throughput measures how much work an AI system can handle, typically expressed as tokens per second, requests per second, or images per minute. High throughput is essential for serving many users simultaneously and for keeping costs low. Throughput and latency often involve trade-offs — batching more requests together increases throughput but may increase latency per request. For LLMs, throughput depends on model size, hardware, batch size, sequence length, and optimization techniques like continuous batching and PagedAttention (used in vLLM). A single H100 GPU might generate 1,000-3,000 tokens per second for a 7B model but only 100-300 for a 70B model. Throughput optimization is a major area of systems research, as it directly impacts the economics of running AI services at scale.

Throughput infrastructure underpins the AI industry, enabling training and deployment of models at scale. Major providers including NVIDIA, AWS, Google Cloud, and Azure offer specialized infrastructure optimized for Throughput workloads. Demand for infrastructure has driven a global chip shortage and billions of dollars in capital expenditure.

Understanding Throughput is essential for anyone working in artificial intelligence, whether as a researcher, engineer, investor, or business leader. As AI systems become more sophisticated and widely deployed, concepts like throughput increasingly influence product development decisions, investment theses, and regulatory frameworks. The rapid pace of innovation in this area means that today best practices may evolve significantly within months, making continuous learning a requirement for AI practitioners.

The continued evolution of Throughput reflects the broader trajectory of artificial intelligence from research curiosity to production-critical technology. Industry analysts project that investments in throughput capabilities and related infrastructure will accelerate as organizations across sectors recognize the competitive advantages offered by AI-native approaches to long-standing business challenges.