Budget 90k+ Dual GPU build for running local LLMs

kazstar101

kazstar101

Patron
I am thinking about getting two 3090 used , if i don't get that buy two 4060 ti 16gb . I have 7700x and am5 build already and will the cpu and ddr5 ram from it. I am confused about motherboard, that i would require x670 or b650 and the power supply requirement and the initial setup for motherboard to use dual gpu. Any suggestion in this regard would be helpful.
 
If you're planning to hook 2x 3090, then it's important to choose a proper PSU as well. Choose one which is atleast 1500w. Regarding motherboard, there's no reason to not opt for a b650 board. Just make sure that the VRMs are solid and won't melt away. Lastly, don't fall for the PCIe 5.0 marketing gimmicks. Even the 4090s are not yet fully capable of utilizing the PCIe 5 lane's bandwidth. You could save a lot by choosing a proper motherboard.

Also, could you be a bit more specific about your use case?
1. Will you be training models? If so, what will be your workflow?
2. If you're just gonna try out pre-trained LLMs, what'll be the average size of the model (ex., 13B) you'll be running?
3. Just to be clear, you've already mentioned that it's for local LLMs. But, do you have any plan to self host the models and expose APIs (either to your local network or the internet) so that multiple people can have access to the models at the same time?

It would help if you could post your complete build information of your current setup.
 
I do a lot of offline LLM finetuning and inferencing.
Doing it offline always has limits.
Don't ignore fast and large quantities of RAM, it makes CPU offloading bearable and also is cheaper per Gb than GPU.
avoid AMD as Rocm hasn't gone that far yet.

Stick to nvidia.
If you want pure performance per dollar go for 3090.
If you want the cheapest option with good usability, go for dual 3060 16gb.

Keep the number of pcie lanes in mind.
Ryzen 7000 has only 28 lanes.
Running multiple gpus with lesser pcie lanes has diminishing returns.
 
Back
Top