12tb Nvme + 48tb hdd + loads of vm NAS

[salman8506]

Hi All, Recently upgraded to am5 leaving am4 parts to tinker with, Always wanted a Nvme nas and has built a smaller one in a m920q,

Specs

5600g
X570 prime pro
128gb ram(64gb kit in order, currently 96gb)
2tb*6 nvme
8tb*6 hdd
2tb+1tb+500gb(proxmox disk) sata ssd
msi expander dual ssd card *2,

Swapping 5700x into my itx rig which is giving 5600g for the nas,

Re-used proxmox from earlier nvme nas which was running in m920q,

Nvme nas built waiting for data to be backed up into backup nas about 16tb data,

Setting up pcie x1 to 4x sata adapter for total 10 sata ports, wanted to install my hba here however all pcie slots were full, will be reconfiguring everything once i manage to get a hold of quad nvme card,

All hdd's are in now awaiting data restore to bring everything back together,

Halfway through the transfer i realized the way proxmox handles zfs i will be loosing close to 2hdd space even with raid z1, it blocks last 15% of space so you will be blocked, i had created a OMV with a single 30tb hdd with all shares configured,

Killed everything, 1 day of data restore wasted, passed all hdd's to truenas with 16gb ram and started the data restore again,

With 3 of my system including backup nas now supporting 2.5gbe i decided i needed to upgrade, 2.5gbe switch procured from @aasimenator bought 2 to upgrade both my current and backup nas,

Had to get creative to cool everything now that my little fan on sata controller had to be removed to make space for the 2.5gbe card,

With networking upgraded the data transfer is close to completion, the cabling needs some work along with some truly needed cleaning, I will get around it once construction work is done at my place.

Any suggestions or advise are welcome, this system is now running a slew of publicly exposed services including various of my personal cloud services, The snappy feeling of 8tb of storage on the fly backed by nvme raid is priceless

 

[presdoge]

Nice, i am also planning something similar but can't find reasonably priced high-capacity HDD 's . I am just waiting for aasimenator to put a new listing for them. Where did you got hard disk's from?

 

[belictony]

What do you do that needs 16tb or higher NAS?.
What are the end utility for your NAS?

 

[salman8506]

Nice, i am also planning something similar but can't find reasonably priced high-capacity HDD 's . I am just waiting for aasimenator to put a new listing for them. Where did you got hard disk's from?

All new bought enterprise drives, i paid around 16-18k per drive and bought 4 of those in 2020 and 2 in 2021, I will be upgrading to 6x18tb during the next year or so. its crazy high so dont need that right now,

What do you do that needs 16tb or higher NAS?.
What are the end utility for your NAS?

I have around 12tb media library and around 7-8tb of other misc storage. I have a long list of services i am using in the nas, These are some services i need to get around to updating the dashboard, Once i fix my temp monitoring and auto patcher will fix below.

 

[ishanjain28]

Looks good OP. I have 30tb hdd nas and 1.5tb ssd nas mostly for proxmox. Home network is largely still 1g so didn't see a benefit in switching to nvme nas(and don't see a benefit in doing that until 2.5g)

Hope you also have a pbs outside of your network for disaster recovery situations.

 

[salman8506]

Looks good OP. I have 30tb hdd nas and 1.5tb ssd nas mostly for proxmox. Home network is largely still 1g so didn't see a benefit in switching to nvme nas(and don't see a benefit in doing that until 2.5g)

Hope you also have a pbs outside of your network for disaster recovery situations.

I have a nice 3-2-1 strategy for backup, pbs has been setup for proxmox server backups as well.

 

[rsaeon]

Cool setup, definitely understand the mess. If it works, it works! haha

I would think you'd use the first and second x16 slots for the NVME drives since your motherboard will split the lanes x8 + x8 and has bifurcation support so you can have four drives across both slots as x4 x4 + x4 x4. Coupled with the cpu m.2 slot, the last x16 slot which is x4 from the chipset and the m.2 from the chipset, you could have 7 m.2 drives running off the motherboard. And you'd still have 3 x1 slots leftover for extra-curricular activities!

View attachment 220431

You're supposed to remove the blue protective film from the heatsink of the MSI M.2 adapter cards – I have a few of those.

The heatsink on the M.2 adapter isn't strictly necessary, I've removed mine to access the neighbouring slots.

 

[salman8506]

Cool setup, definitely understand the mess. If it works, it works! haha

I would think you'd use the first and second x16 slots for the NVME drives since your motherboard will split the lanes x8 + x8 and has bifurcation support so you can have four drives across both slots as x4 x4 + x4 x4. Coupled with the cpu m.2 slot, the last x16 slot which is x4 from the chipset and the m.2 from the chipset, you could have 7 m.2 drives running off the motherboard. And you'd still have 3 x1 slots leftover for extra-curricular activities!



You're supposed to remove the blue protective film from the heatsink of the MSI M.2 adapter cards – I have a few of those.

The heatsink on the M.2 adapter isn't strictly necessary, I've removed mine to access the neighbouring slots.

So on my board the bifurcation is happening on any one slot hence the first slot has that x16 single nvme, second slot has 2, and last slot has dual nvme card with single nvme installed, once I find a quad nvme card will use the first slot freeing up the remaining board. I did spend a good 4 hours to get the 6 to work hence the odd combination. Once I get a quad nvme card it should be fixed hopefully. The heatsink yes not necessary but the nvmes are staying under 42-43c under load so i kept them on. I am in need of another x1 slot for wifi card hence might Remove the heatsink.

The protective film has been cut till the point the nvme is installed have put the blue film on the remaining thermal pads to ensure the adhesive over the excessive thermal pad doesn't spoil.

 

[rsaeon]

Ah, I see now, that's a limitation of the 5600G processor. It can only do x8 + x4 + x4, the first 8 lanes can't be bifurcated regardless of motherboard/chipset. I did not know this despite owning a few of them, haha.

The last two x4 + x4 lanes can be either bifurcated in the first slot, or second slot like you have it.

So if you do get a Quad NVMe card, it'll only allow for three drives to be detected when plugged into the first slot (and only two in the second slot).

If you had a non-G 5000 series or 3000 series processor then the x16 slot can be bifurcated to x4 + x4 + x4 + x4 with the last two being either in the first slot or second slot.

For most motherboards, If there's a card in the second slot, the first slot is reduced to 8 lanes (if both are connected to the cpu like most ATX boards).

There's a nice table here specifically for Asus motherboards: https://www.asus.com/support/faq/1037507/ I haven't been able to find similar documentation by any another manufacturer.

Gigabyte's cheapest B550 motherboard allows for x4 + x4 + x4 + x4 birfurcation the first slot so it's the least expensive way to get 6 nvme drives on an AM4 platform (with a quad nvme card).

@tech.monk discovered that a seller named SaiRetail.com on Amazon is selling dual and quad nvme cards for an affordable price but they're lower on their own website.

For your current setup, the max is 6 drives, cpu m.2, chipset m.2, chipset pcie slot and three drives total for the two cpu pcie slots (3+0 or 1+2). Looks like you're already at your ideal setup for the hardware you have, nice!

What I did was to use a x1 slot for a graphics card with a non-G processor so that the main slots can be used for nvme drives.

 

[salman8506]

Ah, I see now, that's a limitation of the 5600G processor. It can only do x8 + x4 + x4, the first 8 lanes can't be bifurcated regardless of motherboard/chipset. I did not know this despite owning a few of them, haha.

The last two x4 + x4 lanes can be either bifurcated in the first slot, or second slot like you have it.

So if you do get a Quad NVMe card, it'll only allow for three drives to be detected when plugged into the first slot (and only two in the second slot).

If you had a non-G 5000 series or 3000 series processor then the x16 slot can be bifurcated to x4 + x4 + x4 + x4 with the last two being either in the first slot or second slot.

For most motherboards, If there's a card in the second slot, the first slot is reduced to 8 lanes (if both are connected to the cpu like most ATX boards).

There's a nice table here specifically for Asus motherboards: https://www.asus.com/support/faq/1037507/ I haven't been able to find similar documentation by any another manufacturer.

Gigabyte's cheapest B550 motherboard allows for x4 + x4 + x4 + x4 birfurcation the first slot so it's the least expensive way to get 6 nvme drives on an AM4 platform (with a quad nvme card).

@tech.monk discovered that a seller named SaiRetail.com on Amazon is selling dual and quad nvme cards for an affordable price but they're lower on their own website.

For your current setup, the max is 6 drives, cpu m.2, chipset m.2, chipset pcie slot and three drives total for the two cpu pcie slots (3+0 or 1+2). Looks like you're already at your ideal setup for the hardware you have, nice!

What I did was to use a x1 slot for a graphics card with a non-G processor so that the main slots can be used for nvme drives.

i did not know this limitation of 5600g, I can always put my 5700x back in and do some testing although it will be only on next weekend now.

I did find the table when i discussing the possibility of doing this with @tech.monk and yes its very detailed that was the basis of me ordering the 2 dual nvme cards i did suspect the limitation of 5600g when the first slot simply did not detect any drives from the dual card however everywhere i read it said it has 16+4+4 pcie lanes which lead me to believe it will work, I will play around with this a bit and refine it further as i had stopped after spending hours as soon as i saw 6 drives listed in bios For now it just works perfectly.

I did order the sairetail card from amazon locally post extensive testing it did not work the way its supposed to, @tech.monk has same card imported from US and it works fine. I sent the card back.

Yes i am having a pciex1 to x16 riser i had got with another package deal i can use that probably for display out or the last x16 slot can also be used for a gpu. Thanks for pointing it out, although i am happy with current setup including the power consumption of the 5600g but let me see if i can make it look cleaner, i would love to put my hba in the last x4 slot so i can enjoy power saving features of truenas by passing the hba to it.

Thanks for the detailed explanation, Helps me a lot in understanding why my system is behaving the way it is.