Graphic Cards Setup for running LLMs locally

[dank.it]

What is the cheapest setup I can have for running LLMs locally? I do not want to train, just want to do inference. Also won't be running heaviest models, I would be experimenting with smaller versions of models. Fine with used components/laptop/mini pc, whatever sails the boat.

 

[thetechguy]

ollama is the simplest.

ollama pull mistral

For ui,
docker run -p 8080:8080 open-webui/open-webui

 

[dank.it]

Yes but wouldn't that require local computation? I am looking for a budget setup to make it happen.

 

[MostlyHarmless]

you can run it on just CPU and RAM although it will be slow. I get 4t/s running deepseek 14b on 32gb of RAM with a 9800x3d

 

[dank.it]

you can run it on just CPU and RAM although it will be slow. I get 4t/s running deepseek 14b on 32gb of RAM with a 9800x3d

That's helpful. So just good cpu and ram can make it happen? No need for discrete GPU if I can be little patient?

 

[CasualGamer91]

Rtx 3060 12gb

 

[dank.it]

I see laptops around with with 4GB GTX 1650, how helpful can that be for this ?

 

[MostlyHarmless]

That's helpful. So just good cpu and ram can make it happen? No need for discrete GPU if I can be little patient?

Yes, you can use LMStudio, that is the most user friendly way to run local llms. And smaller models will be faster

 

[dank.it]

Yes, you can use LMStudio, that is the most user friendly way to run local llms. And smaller models will be faster

Thanks mate. That was very helpful.

 

[MostlyHarmless]

I see laptops around with with 4GB GTX 1650, how helpful can that be for this ?

For llms RAM is the most important thing so a 4gb card will mean you can only run the smallest models. The model should be able to fit in RAM

 

[dank.it]

For llms RAM is the most important thing so a 4gb card will mean you can only run the smallest models. The model should be able to fit in RAM

So that rules out all the budget GPUs , I'll get a good cpu with hefty RAM instead

 

[MostlyHarmless]

A Mac mini is also a great option. The unified memory architecture allows the gpu to share system RAM so you can have 32GB of memory with the GPU that will give much better perf.
Another thing people do is get one of those mini PCs with expandable RAM and put 2x48GB sticks.
But a GPU with enough vram will beat any of these options

 

[soulweaver]

My base version Mac Mini M4, with an external SSD is doing really well. I can definitely recommend it.

 

[roadrash99]

Watch out for Ryzen AI Max 385/395 based mini PCs, like the framework desktop. There will most likely be more of these launching soon. These also have unified memory and will likely be cheaper than Mac Mini for higher RAM versions.