Home Server/lab guide

[burntwingzZz]

Requirements

PFsense or any open source router ,Media center services and storage ,network cameras to dump their recordings and bit of automation and some nested vms.

HW would be, Ryzen based system ,intel nic and may be LSI HBA (for increasing sata ports).

Require help on the Hypervisor front ,vm approach ,docker approach and storage , basically what approach is suitable.

Storage part :Wont be using raid , there are multiple hdd that i use on the windows machine which i would like to move to this server. I have multiple sized HDD which i have scavenged over the years which i use to store the data and would keep on upgrading in size ,remove hdd introduce hdd etc which shouldn`t break the file services . so what approach or or file system would suit me freenas ,zfs and lot of it is out there ? . the network shared drive should be accessible over windows, linux ,android etc

Few other major concern is how do Linux system or Hypr visors behave , when you change the underlying processor board ,do i need to always start from scratch , or just change the vcpu allocation after replacing the processor etc .

And how do vcpu allocation work ? for supposed 6c/12 thread ,could i oversubscribe/over allocate vcpu ? my eg would be 12 vms with one cpu each ,could it run more than 12 vms ?

Would it survive power outages , i dont face any but , the old stories is that linux based system are bit sensitive.

i will take snapshopt of all the critical vm and store them in some other drive just incase some thing crashes i could just get the whole thing running again.


it would be a server with i say hustled hardware
@booo

any view on the whole topic or the storage section ?

Bump

 

[booo]

too many questions which have too many answers.

it would be a server with i say hustled hardware

I dont know how I should feel about helping with hustled hardware. but being that said; enterprise hardware usually becomes useless if you want to combine with consumer hardware. for example, you wont be able to use SAS hdds even though they look like regular hdds.

Storage part :Wont be using raid , there are multiple hdd that i use on the windows machine which i would like to move to this server. I have multiple sized HDD which i have scavenged over the years which i use to store the data and would keep on upgrading in size ,remove hdd introduce hdd etc which shouldn`t break the file services . so what approach or or file system would suit me freenas ,zfs and lot of it is out there ? . the network shared drive should be accessible over windows, linux ,android etc

I am hearing that you will need both nfs and samba on the same storage. nfs for linux exports and samba for windows. expect samba to give you some problems.

Few other major concern is how do Linux system or Hypr visors behave , when you change the underlying processor board ,do i need to always start from scratch , or just change the vcpu allocation after replacing the processor etc .

And how do vcpu allocation work ? for supposed 6c/12 thread ,could i oversubscribe/over allocate vcpu ? my eg would be 12 vms with one cpu each ,could it run more than 12 vms ?

you can oversubscribe vcpus but it will end up in slowing down every vm. but being that said, you can hard allocate the cpu. ps: vmware will start asking more money for more features. I never used hyperv extensively. so linux kvm/docker is the only thing I can help you with if you want.

Would it survive power outages , i dont face any but , the old stories is that linux based system are bit sensitive.

"linux cant survive power outages." is 1990s talk. I pull the power cords on a regular basis. file systems like ext4 and zfs are way too stable to get corrupted by power outages.

Now, coming to the storage part, there are two things I would recommend. word of caution.... both come with a huge learning curve.
1. ceph: I would suggest ceph if you want scale the servers in future. not just hard drives. a setup which requires kubernetes and clusters etc... you first setup a kubernetes cluster and then use rook-ceph for installing ceph.
2. minio: heard a lot of good stuff about it and people recommend it.
Both require good knowledge on docker.
3. simple zfs/btrfs based storage config with nfs and samba servers... (you will configure zfs/btrfs with a combination of striping and mirroring to create fs. and hot plugging wont be straight forward)

 

[superczar]

any hypervisor will fulfill your needs.
i have been using proxmox for the scenarios you listed (+some) as a home server for the past many years.

network bulk storage is on a ubuntu vm
a couple of debian based VMs for home automation and other needs
1 windows vm
and a router os that was subsequently retired and replaced with an edgerouter- the router os was pfsense followed by sophos with 3x nic for load balancing

also, you can take a running snapshot of any VM , put it in cold storage and replicate/ replace on existing or o any new hardware

PS: FWIW, a recent full hardware upgrade involved zero downtime
added new hardware on network and just migrated the VMs over

 

[burntwingzZz]

@superczar any particular reason to move away from pfsense and migrating to edge router ?

 

[superczar]

no particular reason other than convenience.
in any case, i was using sophos which i found to be better than pfsense.

pfsense is pretty good too but you need to know why you want to use it /have a clear usage requirement in mind else it’s not going to yield you anything more than what any mid range router will offer

 

[Chaos]

@superczar any particular reason to move away from pfsense and migrating to edge router ?

I use an edge router 4. The damn thing is so stable even when bombarded with hundreds of devices at the same time. It has never rebooted/hung ever since the day I deployed it and load balances multiple connections like a champ. I rebooted it the first time last week for a fw upgrade after over 500+ days of uptime. Also it is really configurable especially from the command line and can handle upto 3.3 million packets a second without slowing down. Very few devices have that kind of sustained throughput at under 30w power consumption.

Tried pfsense before and it was a pain to make it work the way I wanted.

 

[superczar]

I use an edge router 4. The damn thing is so stable even when bombarded with hundreds of devices at the same time. It has never rebooted/hung ever since the day I deployed it and load balances multiple connections like a champ. I rebooted it the first time last week for a fw upgrade after over 500+ days of uptime. Also it is really configurable especially from the command line and can handle upto 3.3 million packets a second without slowing down. Very few devices have that kind of sustained throughput at under 30w power consumption.

Tried pfsense before and it was a pain to make it work the way I wanted.

To be fair, Edge OS is also kind of esoteric .
I have had my fair share of issues with it incl but not limited to:
- LB/ Failover for 3 WAN connections (instead of the usual 2) - Anything other than the usual use-cases can be a PITA to setup
- PPPOE on primary would sometimes not come back on after a drop - and with failover, you don't even realise it for a long time .. I had to stick a manual script (to check PPPOE status and reconnect if dropped ) and add it to cron it to run every 60 mins or so

Having said that, once done, it has been perfectly stable with practically no downtime / reboot required after the initial hassles

pfsenses/ sophos can also be similar - a Bit of a pain to setup initially but very stable once done.

The added flexibility/ features can also be pretty useful.
I setup a virtualised pfsense install running for my building WAN - because I needed to add hotspot with authentication for residents
It did take a fair bit longer than ER to setup - but has been going strong from 4+ years now

From a home install, ER has a slight edge - esp given it's a whole lot more cost effective than adding a x64 machine (unless its a shared usage scenario like this one)
Unfortunately for some reason, Ubnt routing equipment has become difficult to source of late.

 

[burntwingzZz]

i wont be doing multiple WAN , as by the the grace of god i have just used one ISP throughout my life at my hame and has been rock solid

wanted storage sorted out ,as that would be biggest paint , all of my important documents etc are sorted just my fathers media and moms plays are difficult to handle around

 

[booo]

wanted storage sorted out ,as that would be biggest paint , all of my important documents etc are sorted just my fathers media and moms plays are difficult to handle around

MinIO | Learn more about MinIO's Erasure Code Implementation

The MinIO Erasure Code Quickstart Guide offers examples, code and insight on this powerful data protection approach

docs.minio.io

I think for your scenario i would keep it simple. kubernetes and ceph would be bit of overkill imo.

so here is what i would do.
1. keep it to just docker since it is a single node and not many services are involved.
2. the advantage with docker is that as long as the service doesn’t persist any data, it can be destroyed and recreated again and again.
3. install portainer.io for esay management of docker containers on the server. create one volume with mount point., say /mnt/docker_data on the drive that is used to install the os and then use it to hold all the docker services data. e.g., /mnt/docker/portainer for all the portainer data /mnt/docker/minio for minio data /mnt/docker/egdeos for all the edgeos data.
4. use the minio erasure coding tutorial above to create a storage pool of all the remaining drives to be used for data storage. right now since you have only one server this will suffice. in future you can convince your best friend to get same type setup and then setup your server’s replica to your friends’ and vice varsa so that you both have backups for each other.
5. finally setup nfs, samba and s3 services on the minio storage pool.
6. not sure how you can mount nfs volume on an android phone but if it is rooted it is a simple shell command. on windows you can mount samba volume and easily automate mount at logon.
7. backup /mnt/docker so that you cam retain all the containers’ persistent data. not really necessary but good to have.
8. needless to say that all the other services like edge router etc have to be installed in docker containers. i haven’t experimented much with docker’s network stack much but you should be able to create multiple private and public networks using multiple nics and sriov stuff like that.

 

[aasimenator]

I am in the same boat right now, have been building my home lab slowly in my new house for the past few months, this is the configuration that I have setup / setting up.

1. Router / Firewall - pfSense/OPNSense on Dell Precision T1700 SFF (i5 4th Gen, 8GB DDR3, Quad Port 1Gb Intel Pro, HP NC523SFP 10Gig Dual Port, 120GB SSD) - I have experience with pfSense but testing OPNSense cause I've heard its easier to use and configure than pfSense.
2. Switch - Dell PowerConnect 6224 24 Port Gigabit switch
3. HP Prodesk 600G2 Tiny - Pentium G3900, 8GB DDR4, 128GB NVMe SSD, 2Tb 2.5inch HDD - Debian with Docker for HomeAssistant, Portainer and AdGuard Home & Recording software for the cameras (yet to finalize)
4. HP EliteDesk 800 G1 Tiny - i5 4th Gen, 16GB DDR3, 500GB SSD, Apple WiFi Card, etc - Big Sur Hackintosh
5. HP Z240 Workstation - Xeon 1225 v5, 16GB DDR4, 128GB NVMe, 3 x 4TB Seagate Enterprise SATA HDD, 250GB SSD(Caching) - TrueNas Core
6. NZXT Manta Mini ITX case, i5 9600K, Gigabyte Z390I Aorus Pro Wifi, 500GB + 250GB NVMe, 4TB WD RED, Asus RX 580, 16GB RAM - My current and main system.
7. TP-LINK PoE 8 Port switch for IP Camera through-out the house.
8. 3 x TP-LINK Deco M5 Wireless Mesh.
9. 6 x Hikvision 5MP IP Camera
10. APC 1000KvA UPS
11. 17U ValRack Network Rack

For Hypervisor, if you want open-source you can't go wrong with Proxmox, but there is a bit of a learning curve with it if you are not used to it. if you are ok with closed-source then VMware ESXi is the best one, you shouldn't need anything other than the free version.
I prefer multiple systems for specific tasks rather than an all-in-one, the best part of building a home lab is that you can go as crazy as enterprise-grade everything or desktop-grade everything or even mix and match.

Ceph is not required at all, it's super difficult to set up and get it going, and even after that if you turn off a node and turn it back on, most of the time it won't be added back to the cluster, and its again hard to figure out why and fix it, so don't use Ceph unless you are a pro at it.

If you have experience with pfSense, then go with it, I've deployed it for my clients and I provide support for those for the past 8 years, but if you want an alternative look at OPNSense it's far more user intuitive than the former. You don't need anything else if you have either one of these setups. they can do a lot more than any closed source solution from big and small brands, and it is much better than anything that is available in the Indian market.