… Offloading layers lets me run massive LLMs on weak GPUs That’s how I managed to deploy Qwen3.6-35B-A3B on 12GB of VRAM Although your GPU is the ideal component for providing extra processing oomph to your LLMs, it’s not the only device capable of running them. …
… Related Nvidia stopped supporting my GPU, so I started self-hosting LLMs with it I self-support my gpu now because Nvidia won't I went with the Vulkan variant of llama.cpp for this project Getting GPU passthrough working was the easy part Let me make this clear: Ollama is a fantastic local LLM prov… …
… The most notable feature is PagedAttention , which dynamically pages the KV cache in and out instead of allocating fixed GPU resources per request. vLLM also uses continuous batching to keep the GPU fully saturated rather than idle, which gets your work done faster. …
… The issue there is that RAM is actually really slow, at least compared to VRAM on a GPU or CPU cache. The former is the best choice for running LLMs with Nvidia GPUs and tool sets leading the way. But how much do you need to spend on a GPU to comfortably run an LLM with decent results? …
… Related Nvidia stopped supporting my GPU, so I started self-hosting LLMs with it I self-support my gpu now because Nvidia won't Passing a GPU to LXCs is fairly straightforward Especially compared to PCI passthrough on VMs Compared to configuring GPU passthrough on VMs manually, LXCs have a simple p… …
… Obsession with GPUs is real GPUs are important, but not everything When you first get into self-hosting LLMs , everything revolves around the GPU; and honestly, that makes sense. …
… This is why an older high-end GPU like the RTX 3090 is better than newer cards for local AI inference . It has 24GB of GDDR6X VRAM while being powerful enough for most modern LLMs. Even the value for money of a used RTX 3090 is hard to resist, considering the prices of GPUs in today's market . …
… It's a great entry-level option for home labs 12GB of VRAM gives you the headroom to actually run local LLMs If you've tried running local LLMs on an 8GB GPU like the RTX 4060 or 5060 , you already know where the problem starts. …
… Related I tried dozens of Photoshop alternatives and I keep coming back to this free tool The free Photoshop replacement I finally settled on You can run local LLMs on your GPU With tools like Ollama, your GPU can start acting like a local AI engine, running on billions of parameters, all locally s… …
… Related I started self-hosting LLMs and absolutely loved it Who needs OpenAI when your home lab can do the thinking for you? …