Host offloading is most useful when GPU memory limits model size, sequence length, or batch size, and when selected tensors are large enough to reduce HBM pressure. It is especially useful when offloading can replace expensive activation rematerialization or make a larger batch configuration feasible. Performance depends on overlap. Host offloading works best when the workload has enough compute, communication, or other independent work to hide transfer latency. On NVIDIA GPUs, XLA helps create this overlap by using dedicated copy streams, scheduling transfers with LHS, and enabling pipelined
Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading | NVIDIA Technical BlogSpeculative decoding has two phases: drafting and verification. A smaller draft model proposes future tokens. The target model verifies those tokens in parallel and accepts the longest valid prefix. If the draft is correct, the system generates multiple tokens with one target-model verification pass. Traditional speculative decoding methods often use autoregressive draft models. These drafters still generate tokens sequentially, so drafting cost increases as the number of speculative tokens increases. This limits how far the method can push throughput. DFlash replaces the autoregressive drafte
Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding | NVIDIA Technical Blog… Jul 08, 2026 By Sean Lopp , Matthew Penn and Sukrit Rao Discuss 0 Discuss 0 L T F R E AI-Generated Summary Like Dislike Fine-tuning and harness profile engineering for agentic systems have enabled open-source models like NVIDIA Nemotron 3 Ultra to approach the accuracy of proprietary frontier model… …
… Key takeaway: Static fractions + bin packing: Predictable traffic, low-to-moderate concurrency, models with stable memory footprints Dynamic GPU fractions + bin packing: Variable traffic, high concurrency, models with significant KV-cache growth Dynamic GPU fractions eliminate the performance ceili… …
… Learn more Large language model LLM training workloads increasingly run into GPU memory limits before compute is fully used. Model weights, gradients, optimizer states, communication buffers, and intermediate activations all compete for GPU high-bandwidth memory HBM . …
Data Center / Cloud Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding Jun 23, 2026 By Amr Elmeleegy , Benjamin Chislett , Fernando Xiong , Michael Iovine , Omri Almog and Hao Zhang Discuss 0 Discuss 0 L T F R E AI-Generated Summary Like Dislike DFlash, an o… …
… Learn about how Mixture of Experts Powers the Most Intelligent Frontier AI Models, Runs 10x Faster on NVIDIA Blackwell NVL72 in this blog . View More Performance Data Training to Convergence Deploying AI in real-world applications requires training networks to convergence at a specified accuracy. …
… When to use Working with limited GPU resources Maintaining multiple specialized versions of a base model Requiring quick iterations and fast training cycles Limitations Retraining a subsection of the model weights limits the degree of change possible quality ceiling Direct Preference Optimization W… …
… Because attention now dominates that cost, how it is designed—not just how it is implemented—increasingly determines a model’s inference performance. Shaping model architecture around how GPUs execute it is the premise of AI model co-design. …
… Discuss 0 Discuss 0 Tags Data Center / Cloud | HPC / Scientific Computing | Blackwell | DSX | Dynamo | GB200 | TensorRT-LLM | Intermediate Technical | Best practice | AI Factory | AI Inference | Energy | Inference Performance | Megatron | Mixture of Experts MoE | NVFP4 | Training AI Models About th… …
… For OpenFold3, the practical workflow spans MSA generation, co-folding inference, deployment, and the memory limits that determine how large a biological assembly can be modeled. …
… They run power-dense training and inference workloads, increasingly support agentic and reasoning models, and must deliver predictable performance even as compute demand shifts rapidly. …
