If you’re a homelab purist like me, you probably care a lot about self-hosting being actually self-hosted. That means no surprise external fetches, no hidden dependencies on third-party CDNs, and no weird packaging choices that undermine the whole point of running software on your own machine.
T...
In this post I want to explore the pitfalls and working settings for running LLMs locally on an RDNA2 GPU. It mainly focuses on cache quantization but also offers more settings to improve performance and some rambling as well.