Financial Services Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
Search
Related topics:
LLMs
All sources
xda-developers.com
37
developer.nvidia.com
29
blogs.nvidia.com
14
amd.com
7
theregister.com
3
intel.com
3
huggingface.co
3
phoronix.com
2
wccftech.com
2
press.asus.com
2
extremetech.com
1
cnx-software.com
1
Videos
More videosPublic Sector Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
Open Source Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
Fast, Low-Cost Inference Offers Key to Profitable AI
…To run state-of-the-art LLMs in real time, enterprises need multiple GPUs working in concert. Tools like the NVIDIA Collective Communication Library , or NCCL, enable multi-GPU systems to quickly…
Top stories
Trying to self-host LLMs made me realize local AI has a friction problem, not a quality problem
5d agoAMD just dropped a compact AI workstation that makes discrete GPUs look outdated for running LLMs
1w agoI added a second GPU just for local AI workloads, and it cost less than upgrading my main one
1w ago13 years later, the GTX Titan is still the most important GPU Nvidia ever made
1w agoIndustrial and Manufacturing Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
Healthcare and Life Sciences Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
I replaced my local LLM with a model half its size and got better results — And it wasn't about the parameters
…It ran smoothly on my setup through GPU offloading, even though it’s officially designed for 16GB of VRAM. For most things, it was fine. I would primarily prompt it for quick…
Discussions and forums
My first 10 inch rack with local LLM! No more Spotify, Google Home, Netflix, ChatGPT...
I'm pretty new to homelabbing and this is my first mini rack! Started with the Beelink ME Mini and then just kinda grew from there (it's always the way hey haha). It idles at 70 watts (not too shabby for how much is goin…
Computer build using Intel Optane Persistent Memory - Can run 1 trillion parameter model at over 4 tokens/sec
As the title states, my build is indeed able to run a 1 trillion parameter model (in this case Kimi K2.5) locally at ~4 tokens/second. I thought r/LocalLLaMA would be interested in the build due to that stat line, and al…
80 tok/sec and 128K context on 12GB VRAM with Qwen3.6 35B A3B and llama.cpp MTP
Just wanted to share my config in hopes of helping other 12GB GPU owners achieve what I see as very respectable token generation speeds with modest VRAM. Using the latest llama.cpp build + MTP PR, I got over 80 tok/sec w…
2.5x faster inference with Qwen 3.6 27B using MTP - Finally a viable option for local agentic coding - 262k context on 48GB - Fixed chat template - Drop-in OpenAI and Anthropic API endpoints
2026-05-07 edit: I have updated the hardware based recommendations with more focus on quality. I do not recommend q4_0 KV cache anymore beyond 64k context. After multiple rounds of testing with the different size quants,…
services with actually generous free tiers for open-source projects. my list, what would you add?
Been in the weeds shipping an OSS side project for the past few weeks (social media publishing API). Real launch post is coming, this isn't that. Along the way I kept a list of services that actually have usable free tie…
I tested the Geekom A9 Max 2026 Edition — and discovered a powerful AI mini PC with workstation-level performance
…The GPU came out at 34,145, not a bad result for an integrated GPU and more than enough for playing many games at HD resolution. Sure enough, when loading Indiana Jones…
Pruning and Distilling LLMs Using NVIDIA TensorRT Model Optimizer | NVIDIA Technical Blog
…samples. The script for this process is provided below, showing how to prune using a two-GPU pipeline parallel setup. torchrun --nproc_per_node 2 /opt/NeMo/scripts/llm/gpt_prune.py…
Nvidia slaps Groq into new LPX racks for faster AI response
…To be clear, LPUs won't replace Nvidia's GPUs but rather augment them. Speed for decode LLM inference encompasses two stages: the compute-heavy prefill phase in which the prompts are…