…an Nvidia RTX graphics card. Paperless-ngx doesn't demand too many resources or GPU power, but I combined it with a local LLM and WSL’s GPU pass-through, which helps…
…The multi-stage pipeline support is particularly valuable for enterprise use cases where you need that SFT → DPO → RLHF workflow for safety and alignment. This could democratize custom model development for mid…
…Keep PyTorch as the model definition while relying on TensorRT LLM for runtime integration AutoDeploy currently supports more than 100 text‑to‑text LLMs and offers early support for VLMs and SSMs…
…for inference, avoiding the need to purchase GPUs. In general, AMD EPYC server CPUs deliver enough performance for models up to 20 billion parameters. This includes many popular large language models (LLMs…
…Sign in to your XDA account As much as I adore my local LLMs, they can’t hold a candle to the reasoning capabilities of their cloud counterparts, and for good reason…
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
I'm pretty new to homelabbing and this is my first mini rack! Started with the Beelink ME Mini and then just kinda grew from there (it's always the way hey haha). It idles at 70 watts (not too shabby for how much is goin…
As the title states, my build is indeed able to run a 1 trillion parameter model (in this case Kimi K2.5) locally at ~4 tokens/second. I thought r/LocalLLaMA would be interested in the build due to that stat line, and al…
Just wanted to share my config in hopes of helping other 12GB GPU owners achieve what I see as very respectable token generation speeds with modest VRAM. Using the latest llama.cpp build + MTP PR, I got over 80 tok/sec w…
2026-05-07 edit: I have updated the hardware based recommendations with more focus on quality. I do not recommend q4_0 KV cache anymore beyond 64k context. After multiple rounds of testing with the different size quants,…
Been in the weeds shipping an OSS side project for the past few weeks (social media publishing API). Real launch post is coming, this isn't that. Along the way I kept a list of services that actually have usable free tie…
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
