… When enterprise AI demand gobbled almost all the available memory, GPU manufacturers could no longer continue to sell GPUs at the prices we saw until the end of last year. …
… Sign in to your XDA account In terms of running local LLMs, the RTX 3090 is the firm king of consumer GPUs used to run local AI on a budget with its massive 24 GB of VRAM. Above that, it's either the RTX 5090, or you start to venture into the enterprise realm, which can sound scary and proprietary. …
… Related This 5-year-old GPU handles local LLMs better than the newest from Nvidia The RTX 3090 has aged beautifully for local AI, thanks to its solid performance and VRAM Tesla cards are not for the faint of heart So am I never buying a Tesla P40? …
… Since they're trained on way bigger datasets with massive enterprise compute, they're in a whole different league in terms of performance. They're available in a browser window on any device, and aren't constrained by your computer's hardware the way local LLMs are. …
… In fact, pairing the right model with a handful of optimizations goes a long way in bolstering any outdated machine’s LLM-hosting capabilities. And I’m not just talking about bulky enterprise-tier hardware, either. …