… With a straight-through estimator and an auxiliary regularization loss , BEAM induces dynamic expert sparsity through end-to-end training while maintaining model capability. …
… In heterogeneous training systems , the total importance ratio should ideally be decomposed into two semantically distinct factors: a training--inference discrepancy term that aligns inference-side and training-side distributions at the same behavior-policy version, and a policy-staleness term that… …
… These approaches either fail to produce a unified model that can be jointly trained on all rewards or necessitates heavy manually tuned sequential training. We find that the failure stems from using a naive weighted-sum reward aggregation. …
… In this work, we systematically study MoE compression in large-scale pretraining, focusing on three key questions: whether pruning provides a better initialization than training from scratch, how expert compression choices affect the final model after continued training , and which training strateg… …
… We release model weights, training code, and complete training data. …
… It's collaborative intelligence , where multiple independent LLMs across device and cloud exchange natural language or structured messages at the task level. 🧭 Two axes, one taxonomy We organize the design space along two composable dimensions: 📡 Vertical device-cloud collaboration : heuristic rout… …
… To enable stable and efficient training under this architecture, we propose to train MELT using chunk-wise training in a two phase procedure: interpolated transition, followed by attention-aligned distillation, both from the LoopLM starting model to MELT. …
… Our study reveals three key findings: 1 even training with purely synthetic data can outperform non-scalable alternatives such as the widely used PrismLayersPro dataset, demonstrating its viability as a scalable and effective substitute; 2 performance consistently improves with increased training d… …
… To address the second challenge, we introduce a parameter-efficient FP8-aware quantization-aware training FP8-aware QAT strategy with partial attention distillation , which freezes the vast majority of pretrained backbone parameters and trains only lightweight linear-branch projection layers, enabl… …
… This report summarizes the main improvements behind GLM-5V-Turbo across model design, multimodal training, reinforcement learning, toolchain expansion, and integration with agent frameworks. …