QAD uses the original full-precision model (teacher) to teach the quantized model (student). First, create a quantized model by running PTQ on the full-precision model. Then distill the frozen BF16 model into the quantized model using a KL divergence loss comparing the teacher’s and student’s logits. Figure 1 shows the two-stage QAD process used to build the Nemotron 3.5 Lightning NVFP4 checkpoint. The full-precision BF16 model serves as the frozen teacher and is also the starting point for Stage 1, a PTQ pass that quantizes weights to W4A16 to produce the quantized student. In Stage 2, the st
Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer | NVIDIA Technical Blog… Use prequantized results to understand whether quantization overhead is the bottleneck, or to compare raw tensor core throughput across precisions independent of the quantization implementation. …
… First, clone the Model Optimizer repository for the quantization example: git clone https://github.com/NVIDIA/TensorRT-Model-Optimizer.git -b 0.37.0 Step 3: Quantize the model Next, execute the Hugging Face example script with the chosen model and quantization format—in this case, Llama 3.1 8B Inst… …
… Benchmarks on an NVIDIA RTX 6000 Ada GPU and profiling with Nsight Deep Learning Designer show that FP8 quantization provides 1.39x1.45x speedup in inference latency for CLIP encoders versus FP16, with the majority of the gains traced to optimized FP8 matmul and attention kernel execution. …
… We show that QAD recovers accuracy degradation from aggressive quantization. Even with more conservative configurations, QAD consistently outperforms PTQ on agentic benchmarks, ensuring high quality while reducing memory usage . What is quantization-aware distillation? …
I measured various Qwen3.8 27B quantizations by Unsloth on popular benchmarks: FPQA Diamond, IFBench, and Terminal-Bench-2.1. Q4_K_M is all you need. submitted by /u/pmigdal to r/LocalLLaMA [link] [comments]
Link to the article: KV Cache Quantization on Gemma 4 31B: Non-QAT vs QAT KLD benchmarks with BeeLlama.cpp v0.4.3, fork of llama.cpp with more KV cache quantization options, comparing Gemma Q4_0 non-QAT vs Gemma Q4_0 QAT…
Link to the article: KV Cache Quantization Benchmarks: KVarN, Precision Tail KLD benchmarks with BeeLlama.cpp v0.4.0, fork of llama.cpp with more KV cache quantization options. Models: Qwen 3.6 27B Q5_K_S 64k context, Ge…
LFM2.5-2.6B is a new tiny model by LiquidAI, with benchmarks that put it head to head with much larger models. I've run llama-perplexity on many model GGUF quants, crossed with many KV cache quants, to understand the mod…
I have various quants of this model and am curious how they perform. can anyone recommend which benchmark would be a good test case for quantization effects? Maybe that can be completed with about 1 million tokens? submi…
… Quantization recipe The following quantization recipe is used in this post as a step-by-step guide for running CLIP model quantization with ModelOpt to understand how the process works. …
… The second column shows the ratio of benchmarks that have improved post-quantization. …
… It then scores each layer’s sensitivity and searches for the per-layer format assignment that meets the budget at the best accuracy, keeping the most sensitive layers in the higher-precision format or skipping them entirely. import modelopt.torch.quantization as mtq model, search state = mtq.auto q… …
… See Footnote ZD-064 Accuracy vLLM-zentorch quantization preserves model accuracy within tight margins of the BF16 baseline. Quantized models are validated using the LM Evaluation Harness with 5-shot prompting across standard benchmarks including GSM8K and ChartQA. …
… KVarN establishes a new state-of-theart for KV-cache quantization on generative benchmarks, including MATH500, AIME24 and HumanEval, at 2-bit precision. …
… Large retrieval benchmarks are comprehensive but too heavy to rerun during development, and there is little infrastructure for comparing production settings-- dimensionality reduction , quantization , reranking --across many models under identical conditions. …
