What is the cheapest setup I can have for running LLMs locally? I do not want to train, just want to do inference. Also won't be running heaviest models, I would be experimenting with smaller versions of models. Fine with used components/laptop/mini pc, whatever sails the boat.
Graphic Cards Setup for running LLMs locally
- Thread starter dank.it
- Start date
thetechguy
Contributor
ollama is the simplest.
ollama pull mistral
For ui,
docker run -p 8080:8080 open-webui/open-webui
ollama pull mistral
For ui,
docker run -p 8080:8080 open-webui/open-webui
MostlyHarmless
Beginner
you can run it on just CPU and RAM although it will be slow. I get 4t/s running deepseek 14b on 32gb of RAM with a 9800x3d
That's helpful. So just good cpu and ram can make it happen? No need for discrete GPU if I can be little patient?
CasualGamer91
Forerunner
MostlyHarmless
Beginner
Yes, you can use LMStudio, that is the most user friendly way to run local llms. And smaller models will be faster
MostlyHarmless
Beginner
For llms RAM is the most important thing so a 4gb card will mean you can only run the smallest models. The model should be able to fit in RAM
So that rules out all the budget GPUs , I'll get a good cpu with hefty RAM instead
MostlyHarmless
Beginner
A Mac mini is also a great option. The unified memory architecture allows the gpu to share system RAM so you can have 32GB of memory with the GPU that will give much better perf.
Another thing people do is get one of those mini PCs with expandable RAM and put 2x48GB sticks.
But a GPU with enough vram will beat any of these options
Another thing people do is get one of those mini PCs with expandable RAM and put 2x48GB sticks.
But a GPU with enough vram will beat any of these options
My base version Mac Mini M4, with an external SSD is doing really well. I can definitely recommend it.
roadrash99
Galvanizer
Watch out for Ryzen AI Max 385/395 based mini PCs, like the framework desktop. There will most likely be more of these launching soon. These also have unified memory and will likely be cheaper than Mac Mini for higher RAM versions.