Search

Before building an AI factory, it’s important to understand the economics of inference — how to balance costs, energy efficiency and an increasing demand for AI. Throughput refers to the volume of tokens that a model can produce. Latency is the amount of tokens that the model can output in a specific amount of time, which is often measured in time to first token — how long it takes before the first output appears — and time per output token, or how fast each additional token comes out. Goodput is a newer metric, measuring how much useful output a system can deliver while hitting key latency ta

The Pareto frontier, represented in the figure below, helps visualize the most optimal ways to balance trade-offs between competing goals — like faster responses vs. serving more users simultaneously — when deploying AI at scale. The vertical axis represents throughput efficiency, measured in tokens per second (TPS), for a given amount of energy used. The higher this number, the more requests an AI factory can handle concurrently. The horizontal axis represents the TPS for a single user, representing how long it takes for a model to give a user the first answer to a prompt. The higher the valu

An AI factory is a system of components that come together to turn data into intelligence. It doesn’t necessarily take the form of a high-end, on-premises data center, but could be an AI-dedicated cloud or hybrid model running on accelerated compute infrastructure. Or it could be a telecom infrastructure that can both optimize the network and perform inference at the edge. Any dedicated accelerated computing infrastructure paired with software turning data into intelligence through AI is, in practice, an AI factory. The components include accelerated computing, networking, software, storage, s

An AI factory transforms AI from a series of isolated experiments into a scalable, repeatable and reliable engine for innovation and business value. NVIDIA provides all the components needed to build AI factories, including accelerated computing, high-performance GPUs, high-bandwidth networking and optimized software. NVIDIA Blackwell GPUs, for example, can be connected via networking, liquid-cooled for energy efficiency and orchestrated with AI software. The NVIDIA Dynamo open-source inference platform offers an operating system for AI factories. It’s built to accelerate and scale AI with max