Storage Solutions Enterprise SSDs for Proxmox OS on ZFS

theGloaming

theGloaming

Disciple
Am looking to set up a Proxmox system at home with the Proxmox OS on mirrored SATA SSDs using ZFS (cannot use NVMe).
Along with the Proxmox OS, I also intend to use these SSDs for host images for VMs and such (storage will be on different devices) - as such looking for a 480GB/500GB SSD.

I have read that one should use enterprise SSDs for Proxmox OS when using ZFS (due to super high writes).

1. Where best to order these from? Whatever few places I have seen, these are super expensive (2nd hand are okay, as long as they are reliable).
2. Or is there a possibility to use regular consumer SSDs? If so, which ones are recommended for this use case?
 
Last edited:
I have been using dram-less consumer SSD's for my hosts and apart from a second hand m.2 WD Green one failing prematurely, the ones purchased new are still working fine after almost two years.

Here is a 2.5" WD Green at 14000+ power on hours:

Screen Shot 2022-02-20 at 11.44.39 PM.png

And a 2.5" SIlicon Power S55 at 15000+ hours:

Screen Shot 2022-02-20 at 11.46.20 PM.png

I haven't done anything to reduce disk writes, these drives are running with default Proxmox settings. But none of my drives are under ZFS.

There are refurbishing dealers on OLX that sell server pulls of Micron and Intel drives, with 1TB going for around 6k and up and 1.6tb for 13k and up.

For new, PrimeABGB has a 480GB Micron 5300 Pro for under 12k: https://www.primeabgb.com/online-pr...0-gb-sata-3d-tlc-ssd-mtfddak480tds-1aw1zabyy/
 
rsaeon said:
I have been using dram-less consumer SSD's for my hosts and apart from a second hand m.2 WD Green one failing prematurely, the ones purchased new are still working fine after almost two years.

Here is a 2.5" WD Green at 14000+ power on hours:

View attachment 127334

And a 2.5" SIlicon Power S55 at 15000+ hours:

View attachment 127336

I haven't done anything to reduce disk writes, these drives are running with default Proxmox settings. But none of my drives are under ZFS.

There are refurbishing dealers on OLX that sell server pulls of Micron and Intel drives, with 1TB going for around 6k and up and 1.6tb for 13k and up.

For new, PrimeABGB has a 480GB Micron 5300 Pro for under 12k: https://www.primeabgb.com/online-pr...0-gb-sata-3d-tlc-ssd-mtfddak480tds-1aw1zabyy/
Click to expand...
Thank you, this gives me much to think about.

Your drives are M.2 drives right? How do they compare with SATA SSDs (which I am looking to use)?

The prices which I am seeing for enterprise SSDs (like the Micron one you linked to) are much too high for me. (Am not able to yet see any feasible ones on OLX.)
I am thus considering, for the Proxmox OS drives, to either go without ZFS or to use consumer grade SSDs with ZFS.

Would you know:
1. If I do not use ZFS, then would consumer drives like Samsung 870 EVO or Crucial MX500 be fine for Proxmox OS + VM images? I would also install the Truenas image on this.
2. If I do go for ZFS with consumer SSDs, are there a grade of consumer SSDs which may be better for e.g. the Samsung 860 Pro?
 
I'm using 2.5" SATA drives, most of my hosts are DDR3 based so they lack m.2 slots. m.2 is great when you're able to use it natively in an m.2 slot. In my experience, using a m.2 to SATA adapter can be iffy even though it's just an electrical adapter with no active processing. You may notice drive disconnects with some adapters.

I do not have any experience with ZFS on Proxmox, I went with a single drive setup for my hosts because I needed to have the lowest cost of entry possible for each node. I got a 10 pack of the 120GB Silicon Power S55 for 11k a few days before the lockdown. These days, KLEVV ssds from Amazon are my preferred choice for no actual reason other than I like their branding.

The MX500 and 870 EVO are on the higher end of the sata drive spectrum, so they should be fine inside of a Proxmox host. I have many, many VM's running off these dramless SSD's and I never encountered any i/o delays, even when migrating VM's across different nodes for arbitrary reasons.

Single drive configurations should be fine in a homelab environment for less than a handful of users with no write intensive VM's. We have a Proxmox thread here somewhere and other users have far more advanced configurations than I do so they might be able to chime in with their experiences.
 
theGloaming said:
Am looking to set up a Proxmox system at home with the Proxmox OS on mirrored SATA SSDs using ZFS (cannot use NVMe).
Along with the Proxmox OS, I also intend to use these SSDs for host images for VMs and such (storage will be on different devices) - as such looking for a 480GB/500GB SSD.

I have read that one should use enterprise SSDs for Proxmox OS when using ZFS (due to super high writes).

1. Where best to order these from? Whatever few places I have seen, these are super expensive (2nd hand are okay, as long as they are reliable).
2. Or is there a possibility to use regular consumer SSDs? If so, which ones are recommended for this use case?
Click to expand...
Just curious - why do you want to use zfs on a home server in the first place?
 
I have been running a homelab for 8 years now. It went through so many iterations, but the raidz2 pool has been constant since when I created it in the beginning. It contains loads of family media we've collected over the past decade, and I wouldn't do it any other way.

What purpose do you have in mind for this server? While there's nothing wrong with a pool with mirrored SSD, it's probably overkill. The true power of ZFS is leveraged with hard drives.

You also should be keeping your main storage pool separate from your OS, which can be installed on a single, smaller SSD. This way, your storage pool is portable to any machine and gives you flexibility to switch OS. Over time, I've used the same disks/pool in MacOS, Joyent Smartos, RancherOS, Alpine Linux, Proxmox, and I'm currently running Ubuntu server.
 
rsaeon said:
I'm using 2.5" SATA drives, most of my hosts are DDR3 based so they lack m.2 slots. m.2 is great when you're able to use it natively in an m.2 slot. In my experience, using a m.2 to SATA adapter can be iffy even though it's just an electrical adapter with no active processing. You may notice drive disconnects with some adapters.

I do not have any experience with ZFS on Proxmox, I went with a single drive setup for my hosts because I needed to have the lowest cost of entry possible for each node. I got a 10 pack of the 120GB Silicon Power S55 for 11k a few days before the lockdown. These days, KLEVV ssds from Amazon are my preferred choice for no actual reason other than I like their branding.

The MX500 and 870 EVO are on the higher end of the sata drive spectrum, so they should be fine inside of a Proxmox host. I have many, many VM's running off these dramless SSD's and I never encountered any i/o delays, even when migrating VM's across different nodes for arbitrary reasons.

Single drive configurations should be fine in a homelab environment for less than a handful of users with no write intensive VM's. We have a Proxmox thread here somewhere and other users have far more advanced configurations than I do so they might be able to chime in with their experiences.
Click to expand...
Basis comments here and further reading, I am dropping the plan of using ZFS for my OS and VM image SSDs and will use regular consumer grade SSDs. Your comments have helped me get to this - thanks for that!

By chance, would you know if I can install the TrueNAS image on the same SSD as I use for the Proxmox OS (I read that when installing the TrueNAS image it writes over the full SSD). I guess it is possible, but I would need to first partition the SSD for a specific TrueNAS partition?

And thanks for the headsup on the Proxmox thread; shall go seek it out this weekend.
 
superczar said:
Just curious - why do you want to use zfs on a home server in the first place?
Click to expand...
From reading and videos online, there were a number of advantages for ZFS for Proxmox OS and thus I started looking at that option. However, now that I know that it is best to do this with enterprise SSDs, which are way too costly, I am dropping this ZFS idea.

However, TrueNAS storage needs ZFS, so I will need to set up the HDDs which I pass to TrueNAS as ZFS.
 
deltasquare4 said:
I have been running a homelab for 8 years now. It went through so many iterations, but the raidz2 pool has been constant since when I created it in the beginning. It contains loads of family media we've collected over the past decade, and I wouldn't do it any other way.

What purpose do you have in mind for this server? While there's nothing wrong with a pool with mirrored SSD, it's probably overkill. The true power of ZFS is leveraged with hard drives.

You also should be keeping your main storage pool separate from your OS, which can be installed on a single, smaller SSD. This way, your storage pool is portable to any machine and gives you flexibility to switch OS. Over time, I've used the same disks/pool in MacOS, Joyent Smartos, RancherOS, Alpine Linux, Proxmox, and I'm currently running Ubuntu server.
Click to expand...
8 years, wow! And a lot of OS's and the like, impressive!
What all do in your homelab? Any tips for a noob? :)

My server is going to be used for the following initially:
  1. TrueNAS based NAS setup. For this I will need ZFS storage (still looking for a HBA card!)
  2. Ubuntu VMs to run a bunch of software on (mostly as Docker containers) - some for regular usage (Git server, Bitwarden) and some for experimenting with new tools and such (e.g. email server, Prometheus, Grafana, Elastic, etc.). Thinking of setting a separate LVM storage pool for this (and for #3 below).
  3. Deploy and test some programs that I write
And over time, intend to do home automation and OpenVPN on this.

After more online research and comments from you and others here (and looking at cost of Enterprise SSDs), I am dropping ZFS for OS and VM images.
I will instead go with consumer SSDs for that (currently looking at Crucial MX500 which is going for ~Rs.4700 for 500GB on Amazon).

By chance, would you know if one can install the TrueNAS image on the same SSD as Proxmox as is?
Or would I first have to create a TrueNAS OS specific partition? (read that TrueNAS installation writes over the entire disk that it is given).
 
Well, the most important thing is to document everything you do with your server somewhere because you can understand/change/reproduce what you did after a while.

My primary use case for the lab is media - Plex, but I run other services like VaultWarden/BitWarden, Gitea, Ghost blog, Resilio Sync, Matrix Synapse Chat server, Outline Wiki and Netdata to monitor everything. I have a large collection of family media, setup to sync to the server from everyone's phones through Resilio Sync, and shared back to the phones + TV through Plex. Some of my family use Vaultwarden, while I am the sole user of all of the other services. Setup matrix for integration with other chat services through various bridges, but it's not yet functioning at the level I want.

For your use case, if I'm understanding it correctly, you'd like to run proxmox on bare hardware and have TrueNAS run on top of Proxmox as a VM with passthrough physical SSD disk?

This is a bit redundant indeed. I'd pick one of Proxmox and TrueNAS and stick to that. If you were to use both for experimentation, install TrueNAS in a VM that has a file in proxmox as disk, that way you avoid the complexity of having two OSes in different partitions. I'd keep the data ZFS disk/pool separate from the OS disk.

Hope it helps.
 
theGloaming said:
By chance, would you know if I can install the TrueNAS image on the same SSD as I use for the Proxmox OS (I read that when installing the TrueNAS image it writes over the full SSD).
Click to expand...

Instead of installing alongside Proxmox, install it as a VM inside of Proxmox. And have TrueNAS manage drives you intend to use for network storage by passing the drives directly to TrueNAS to manage.

 
Back
Top