…GPU fractions with bin packing for multiple small models on a GPU Many NIM workloads, like embeddings, rerankers, and small LLMs, rarely need an entire GPU. When used with GPU fractions , NVIDIA…
…You’re at the mercy of the model Google chooses and wait for an update if it falls short. Running LLMs locally frees you from those constraints — of course, you still need…
…pay for the extra fps, but with GPU prices being what they are now, the gap is much smaller than it was at launch for the $10,000 Pro graphics card. Related…
…But I occasionally need to work with AI-accelerated workloads on my dev VM. Since I’ve already enabled GPU passthrough long ago (which is a lot easier than you think), I…
…Sign in to your XDA account I recently started integrating local LLMs with my arsenal of free and open-source tools, and they’ve been a game-changer for my productivity needs…
…The reason why the future MTIAs as well as the current MTIA 300, which has been deployed for R&R training workloads, need to look like GPUs and AI XPUs because they…
…Better GPU utilization: Separating stages lets each saturate its target resource (compute for prefill, memory bandwidth for decode) rather than alternating between both. Frameworks like NVIDIA Dynamo and llm-d , implement this…
I'm pretty new to homelabbing and this is my first mini rack! Started with the Beelink ME Mini and then just kinda grew from there (it's always the way hey haha). It idles at 70 watts (not too shabby for how much is goin…
As the title states, my build is indeed able to run a 1 trillion parameter model (in this case Kimi K2.5) locally at ~4 tokens/second. I thought r/LocalLLaMA would be interested in the build due to that stat line, and al…
Just wanted to share my config in hopes of helping other 12GB GPU owners achieve what I see as very respectable token generation speeds with modest VRAM. Using the latest llama.cpp build + MTP PR, I got over 80 tok/sec w…
2026-05-07 edit: I have updated the hardware based recommendations with more focus on quality. I do not recommend q4_0 KV cache anymore beyond 64k context. After multiple rounds of testing with the different size quants,…
Been in the weeds shipping an OSS side project for the past few weeks (social media publishing API). Real launch post is coming, this isn't that. Along the way I kept a list of services that actually have usable free tie…
…With PTQ, models can be served more efficiently using fewer GPUs. Summary This post showed you how to use PTQ in NeMo to build efficient TensorRT-LLM engines for LLM deployment. For…
…NVIDIA HGX H100 GPUs, totaling 1,024 GPU cards. Next-Gen Infrastructure Solutions ASUS is advancing digital transformation with HPC and AI-driven server systems for diverse enterprise needs, including the ASUS…
…Available VRAM on the GPU D Internet bandwidth Spot on! VRAM is the key bottleneck for local LLM inference. If a model fits entirely in your GPU's VRAM, it runs dramatically…
