To validate the practical impact of low-precision training for real-world large-model pretraining, the team evaluated both the training convergence and downstream task performance across two widely used dense transformer architectures: Llama 3 8B and an NVIDIA internal research 8B model (Research-8B with dense grouped query attention (GQA) architecture that is similar to Llama 3 8B). The models were trained on 1 trillion tokens.
Using NVFP4 Low-Precision Model Training for Higher Throughput Without Losing Accuracy | NVIDIA Technical Blog… To post-train on proprietary or domain data, follow the post-training guide . Explore NVIDIA Cosmos Cookbook for step-by-step workflows and technical recipes from Cosmos users. …
… For inference, responses must meet certain accuracy thresholds, and for training, the models must be trained to specific quality targets ie, the model training process must converge . …
… This post compares the following three low-precision training formats directly against established BF16 precision training across multi-hundred-billion token pretraining runs and downstream benchmarks: 8-bit floating point per-tensor current scaling FP8-CS Mixed precision training with FP8 MXFP8 NV… …
Agentic AI / Generative AI Run High-Throughput Reinforcement Learning Training with End-to-End FP8 Precision Apr 20, 2026 By Guyue Huang , Shuang Yu , Zhaopeng Qiu , Oleg Rybakov , Wenwen Gao and Sylendran Arunagiri Discuss 0 Discuss 0 L T F R E As LLMs transition from simple text generation to com… …
… Domain-specific pre-training data Building on a 10T token pre-training foundation, Nemotron 3 Ultra adds 212B new tokens targeting three high-value domain gaps: 4B tokens of synthetic legal data, increasing the proxy LegalBench average from 64.6% to 74.7% 35B tokens of synthesized Wiki-based data, … …
… When to use Working with accessible data for well-defined tasks with output examples Customizing a model for a low-resource domain where labelled examples are limited, and high-quality synthetic data can be generated to bootstrap the fine-tuning dataset Requiring the model to reliably produce speci… …
Agentic AI / Generative AI NVIDIA Ising Introduces AI-Powered Workflows to Build Fault-Tolerant Quantum Systems Apr 14, 2026 By Tom Lubowe , Christopher Chamberland , Shuxiang Cao , Ivan Basov and Jan Olle Discuss 3 Discuss 3 L T F R E AI-Generated Summary Like Dislike NVIDIA Ising, released in 202… …
… In addition to Muon, NVIDIA also supports many other optimizers for the research community to explore, including: The ultimate form of orthogonalized optimizer MOP Momentum Orthogonalized by Polar decomposition An advanced SOAP variant that updates eigen basis per step with eigen decomposition plus… …
… End-to-end training and evaluation recipes The complete pre-training , post-training , and evaluation recipe for Nemotron 3 Nano Omni is available, covering the full pipeline from pre-training through alignment. …
… The typical flow is: import, patch, optional validate, train as usual. lightning client.py import pytorch lightning as pl from pytorch lightning import Trainer import nvflare.client.lightning as flare Lightning Client API from model import LitNet from data import CIFAR10DataModule def main : model … …