Search

Inference drives revenue, so it is the key workload to optimize. When operators increase inference throughput per watt, they directly increase the number of tokens they can sell or insights they can create. This also translates to additional revenue per unit of time. At the hundred megawatt to gigawatt scale, even a few percentage points of throughput improvement per megawatt can translate into meaningful gains in profit. Model architecture is also important. Mixture-of-experts (MoE) models are typically more energy efficient per unit of intelligence compared to dense models with similar total

The ML.ENERGY Initiative has developed a leaderboard and benchmark for sharing observations from their measurements and a reasoning framework that explains why they observe certain energy behaviors. These benchmarks can be tied into energy aware operations- telemetry-driven systems that show how to run an AI factory under real deployment constraints, including power cost, carbon intensity, thermals, cooling capacity, and grid limits. NVIDIA DSX provides these energy-aware operations. The platform delivers a coordinated view across compute, racks, cooling, facility power, and workload schedulin

Agentic AI infrastructure must optimize more than individual GPUs; it must make effective use of power, cooling, networking, and rack-level resources across the entire AI factory. NVIDIA Vera Rubin NVL72 extends the GPU architecture into an integrated, resilient rack-scale execution domain. This section examines how NVIDIA approaches power efficiency, operational scalability, and system-level reliability. 

As models, context windows, and GPU domains grow, data movement becomes as important as computation. Rubin is designed to improve the flow of weights, activations, KV cache data, and communication traffic within the GPU and across scale-up systems. The following sections look at the memory and communication innovations that help sustain high-throughput inference.