… Related Nvidia stopped supporting my GPU, so I started self-hosting LLMs with it I self-support my gpu now because Nvidia won't I went with the Vulkan variant of llama.cpp for this project Getting GPU passthrough working was the easy part Let me make this clear: Ollama is a fantastic local LLM prov… …
… Ollama does have experimental Vulkan support now, which can technically work with Arc GPUs on Windows and Linux, but it's not the polished experience you'd get with CUDA or even ROCm, and as we've shown, performance is significantly worse when comparing XPU with Vulkan. …
… ROCm 7.2.4 is the latest Linux-side quality release focused on inference performance and stability for Instinct GPUs, while AMD’s Radeon and Ryzen documentation separately lists ROCm 7.2.1 support for Radeon RX 9000, select RX 7000 cards, and Ryzen AI Max, AI 300, and select AI 400 APUs. …