Hello everyone! I’m new here and looking to build a NAS that better fits my needs. I’ve done some homework on the potential setup and would like to know your thoughts on it.
A bit of context
So I’ve been running a tiny storage setup with a raspberry pi 4B hooked up to a 4TB 3.5” HDD and a 120GB SSD for a while now. It acts as my primary backup and media server, supporting 1-2 users, mostly over my local network.
Recently, my 4TB drive decided to give up, taking four years’ worth of memories and backups with it. I am reluctant to proceed with a data recovery service, as they quoted me 50k+ for recovery, while only vaguely explaining what exactly went wrong with the drive.
On the other hand, while my rpi does most of its duties without complaint - I think I’ve also maxed out its capabilities over the course of 4 years.
I’m saying this because it struggles with all the services I’ve crammed into it with just 4GB ram. The overall experience, reliability and ease-of-use could be better/snappier/faster.
The Build
I’ve now decided to switch to a proper setup, with redundancy and backup built into it by design. Although I’m a sucker for small footprints and minimal power consumption, I’ve realised that this corners me into a limited upgrade path. So goodbye SBCs, NUCs and tinyPCs.
Hardware
Lenovo P330 Tower (i3-9100, 16GB, 128GB NVMe SSD) - might get a good deal for ~15k
2x 8TB, 3.5” drives - recertified, lightly used or refurbished (in that order) - ~20-22k
The raspberry pi stays on as the “helpful assistant” - running adguard, reverse proxy and other misc. services, possibly with a cold drive. More on this later.
The P330 and the rpi will be connected to a gigabit router via ethernet, and all my devices access the P330 and rpi over wifi.
Software
Proxmox, with TrueNAS running in a VM.
Another lightweight VM for Docker, which my current services on the rpi will be migrated to.
Open to your suggestions on how to design the layout for this; please also include the advantages over the planned setup.
Going deeper…
TrueNAS sees the 2x 8TB drives as a mirrored vdev - so it can handle at a drive failure.
I’m drawn to this layout because of its simplicity and ease of future upgrades, as opposed to having more than two drives in Raidz/z2. Easier on the wallet too.
Given the current growth rate of my data, 8TB of usable space should last me a good 3-4 years, unless I encounter another drive failure.
Passthrough the onboard SATA controller so TrueNAS has complete access to the ZFS pool, and the NVMe drive hosts Proxmox and its VMs.
Don’t want an HBA card as of now in order to keep the energy usage down. I’ve also read that these cards don’t let the CPU go into deep sleep. Thoughts?
Docker services access the ZFS pool via smb/nfs
In the future, I would like to use a “cold” drive with the rpi, so it can receive snapshots of critical data from the ZFS pool on the P330.
Other questions & challenges
Safe shutdowns: I do have a small UPS that I can use with this setup, but it doesn’t have a comms port for NUT. Possible to use a smartplug or something similar to trigger a safe shutdown? This is quite important; I believe my 4TB drive failed due all the powercuts.
Transfer Speeds: While this setup runs on a gigabit network, are there any optimizations I can make? Any suggestions on how to improve random read/write speeds? I used to get ~50MB/s writes using my rpi, but I’m sure this can be improved.
How long have your used, refurbished or recertified drives lasted? I’m aware of the benefits of CMR drives, but is there anything else I should look out for, other than SMART data?
Since this setup is considerably larger than my previous setup, are there any tweaks I can do to reduce energy consumption? Scheduled sleep, HDD spin-downs, etc?
Lastly, for those who have moved away from storage and entertainment subscriptions: how has your experience been so far with your own setup?
As you can see, I’m trying to keep this setup as lean and mean as possible Would love to read your thoughts and suggestions on this build! Thank you for your time.
@yugaaa For basic fileserving + current docker stack, I agree a Pentium would work just fine. However I do plan on running a lot more services or at least having the headroom to, and the extra cost for the i3 isn’t that much more.
Any particular reason you suggest going with 3 drives?
Sorry if I am missing something but why the need for TrueNAS when you are already running proxmox? Is TrueNAS only for network sharing or any other purpose does it serve?
I think TrueNAS has a better and easier UI for managing ZFS, backups, shares etc., which works for me because I like having it visually laid out, without having to deal with CLIs and stuff.
Another alternative to the GUI, which I just discovered, could be to run an LXC with Cockpit. This can manage the ZFS pool created on Proxmox itself, I think. I’m still learning about the differences between TrueNAS and Cockpit, apart from the GUI itself. Will update if I find anything interesting.
To follow up in-detail about why TrueNAS over Proxmox for ZFS:
From what I’ve read, TrueNAS is built around ZFS, unlike Proxmox that is built as a hypervisor. This makes TrueNAS more “appliance” like, with snapshots, dataset permissions, SMB/NFS, and pool health monitoring all in one place.
With Proxmox, I believe it would be more of a bolt-on approach.
Data-first workflows: Proxmox does VM backups well, whereas TrueNAS simplifies the backup process of your data itself (eg. snapshot certain datasets → send it to rpi’s cold drive → trim fat from main pool). I plan on using both snapshots and VM backups, so IMO both Proxmox and TrueNAS are able to do what they are designed for.
Better portability: In the event of some failure, I think migrating VMs would be simpler. So I’m just planning to let Proxmox do its job as a hypervisor and use VMs for different purposes. So one VM for ZFS, another for all my docker services.
This is based on my research alone, so take it with a pinch of salt. Looking fwd to your thoughts @intruder16 !
With 3 you will effectively get double the storage .
IN general it is always better to use trunas as a standalone device or as an appliance . It gives the flexibility of adding drives , connecting to different VM servers and in case of an issue , you need not shut down the whole box . Your storage can still be accessed .
You could technically install the proxmox on a ssd and use the true nas drives for storage . It would be faster .
Two things: hypervisor and managing ZFS. Truenas and proxmox can do both. ZFS is same for both TrueNAS and proxmox. Installing another hypervisor as a VM makes no sense to me.
Again, creating snapshots is a single command same as zfs send/recv. Using app like sanoid, syncoid one can set automatic snapshots and sending them incrementally to remote location.
The way I have setup NAS is,
Connected NAS disk to proxmox server and mounted to a mount point in proxmox directly.
This mount point can then be shared to LXC’s/VM’s directly and it is all done without a network. So your lxc’s/vm’s can access specific content directly but cannot access other data on the drive.
For network share, I have turnkeyfs as a container, which provides a very nice webmin gui frontend to manage shares, security and permissions.
Setting up this way, I see very less resource usage, since no VM’s are used, no network usage for all usual containers/vm’s/nas backup except when sending to remote location.
It should be either proxmox or TrueNAS. Using both same time in a VM doesn’t makes sense.
Of course this is all subjected to your personal preference and the features you need or most comfortable with. I just saw too many people using same multiple hypervisors together, I thought maybe I am missing something.
Completely agree! However since this is a budget, all-in-one kinda box, I’m not sure your advice applies in this case. I would love to have separate machines for storage and services, some time in the future.
Yep, planning to install Proxmox on a small NVMe SSD. So the 2x 3.5” drives, connected to the onboard SATA controller can be passed through completely to TrueNAS, incl. SMART data and such. So Proxmox won’t be able to “see” the drives directly; it can access them only using the virtual network share from TrueNAS.
TrueNAS’s “hypervising” isn’t as good as Proxmox’s, from what I’ve read. I’ll definitely test it out, can’t know until then. Have you tried using docker services inside TrueNAS?
Gotcha. I’d like to stay away from CLIs or separate services for backups, so leaning towards TrueNAS. However I will try out the Proxmox + Cockpit approach as it looks very interesting. Might just need to run sanoid for this approach There’s just so many ways to achieve this lol.
Nice setup you got there. Architecturally, it looks really lean! You have a lot more experience with Linux and ZFS, don’t you
Couple of questions:
How has your experience been with the “administrative” side of things, like managing ZFS + SMB permissions for a dataset?
What hardware are you running?
Nah, just different pathways to the same end result I think. One favors architectural simplicity, the other favors ease-of-use to a beginner.
HBA cards are necessary only when you run out of sata ports on motherboard.
I don’t think it is possible without some DIY hardware solution. @rsaeon
Random read write speeds of hdd are ~1MB/s while those of ssd start from ~25MB/s.
Any hdd or ssd can fail at any time irrespective of its status/model. If SMART data is showing all good then it all depends on your luck but typically I would suggest not to get HDDs with more than 50k of power on hours unless getting a very good deal & data on it is not not going to be very important.
A typical 7200rpm CMR hdd at max uses 10W during intensive read/write operations so not worth it just for 2HDDs (everything else will already be very power efficient).
If storage main purpose is media then there are paid options like seedbox+pvt trackers or even usenet (more expensive as well as complicated).
+ To pass through complete disk access to TrueNAS? Anyway, it’s overkill for this setup so let’s skip that.
Correct. Does anyone know if I can configure an rpi to check when a smartplug loses power?
I was thinking: if I use a smartplug on a regular wall socket, the rpi could see it via homeassistant. And if I lose power (=smartplug disconnects), the rpi triggers a safe shutdown for the P330 and itself. The UPS should sustain the setup long enough to shut down safely.
I think this is doable, just need to verify with fellow members, if anyone has tried it.
Good question haha. I think there’s a BIOS setting to turn on automatically after power is restored? But since it was a safe shutdown, I’m not sure if this bios setting is the right way.
Might just use the rpi’s GPIO pins to “press” the P330’s power switch on power restore. I like where this is going…
That can work , but that is a hit or miss . I have enabled the bios to restore on ac switch on . Sometimes it works , sometimes it does not . Better option would be an IPMI
My mini PC and an external hard drive enclosure are connected to an APC BX600C UPS.
I’m using my old phone with the tasker app to detect power cuts and send a shutdown signal to the mini PC. A script running on the mini PC listens for a POST signal and shuts down the system if a valid signal is received. The phone’s battery charging is capped at 70% for precaution, since it’s plugged in 24/7.
Once power is restored, the tasker app sends a WOL signal (via Wake On Lan app) to start the mini PC.
I’ve configured a few conditional statements so that the shutdown delay depends on the last power cut. For e.g If the last powercut was b/w 4-6 hrs, shutdown the mini pc after 45 mins.
I recently replaced the UPS battery, so your backup time may vary. Pls configure delay accordingly
Would love to read your thoughts and suggestions on this build! Thank you for your time.
There’s just so many ways to achieve this lol.