@snonux
14.04.26 • 19:07 UTC
I've been playing around with LLMs runnable on local hardware lately to check whether if it is worth to invest 4k in it. So far, the results have been promising, but I'm still expecting a bit better performance and utility for my specific use cases.
- I'd like to have at least 256GB of VRAM (or unified memory) for inference (not like most home-solutions which max out at 128GB)
- I'd like to have at least the same speed as Nvidia's A100
- I'd prefer a Linux based OS, but a Mac would not be a disaster here. Mac Ultras are way too expensive still (especially due to the memory prices).
I don't think there is anything below 5 grand available for now. So for now I'll keep renting my GPUs from Hyperstack for further experimentation and will revisit bying local LLM hardware in a year from now or so!
#llm #inference #hardware #local
