Party Monger
Skilled
I've been tinkering with Proxmox lately on my NUC. Im wondering if anyone else here is also into it or working on other platforms like Unraid, VMware etc.
Proxmox Thread - Home Lab / Virtualization
- Thread starter Party Monger
- Start date
Party Monger
Skilled
My setup right now is - NUC8i5BEH - 8th gen NUC with i5 processor. 32GB RAM. Samsung 970 Evo 1TB NVME SSD.
Few of my main concerns for Proxmox -
1) Write Amplification - I got a brand new 1TB Samsung Evo 970 plus NVME SSD. It has 1GB Dram Cache. Two days with Proxmox and its already written 250GB as per SMART data. This is after I have reduced Swappiness value to 1. Basically it means that it will use disk swap only if there is an urgent requirement. I've read many threads about how SSDs conk off very fast. Is there a way to reduce the writes? I read that if you switch off Cluster and HA it reduces the writes a lot. But something like a ram disk of 5-8gb would be great that writes data periodically.
2) Backup - Is the inbuilt backup system good? I want to backup all VMs to a network drive at night so that in case of issues I can just switch to a different server and start the same machines again. I read that if you're restoring the same machine with original one running then you have to change the machine ID etc. Has anyone found a good guide in general?
Im looking for more advanced guides. But most I find are on clustering etc. Want to go with 1 single server to save electricity costs. But is there a way we can have 2nd servers but switch it on just for an hour so that it can sync machines or something to the other.
Few of my main concerns for Proxmox -
1) Write Amplification - I got a brand new 1TB Samsung Evo 970 plus NVME SSD. It has 1GB Dram Cache. Two days with Proxmox and its already written 250GB as per SMART data. This is after I have reduced Swappiness value to 1. Basically it means that it will use disk swap only if there is an urgent requirement. I've read many threads about how SSDs conk off very fast. Is there a way to reduce the writes? I read that if you switch off Cluster and HA it reduces the writes a lot. But something like a ram disk of 5-8gb would be great that writes data periodically.
2) Backup - Is the inbuilt backup system good? I want to backup all VMs to a network drive at night so that in case of issues I can just switch to a different server and start the same machines again. I read that if you're restoring the same machine with original one running then you have to change the machine ID etc. Has anyone found a good guide in general?
Im looking for more advanced guides. But most I find are on clustering etc. Want to go with 1 single server to save electricity costs. But is there a way we can have 2nd servers but switch it on just for an hour so that it can sync machines or something to the other.
rsaeon
Patron
I have a few Proxmox clusters at home, the highlight of Proxmox is in it's ability to cluster separate boxes for redundancy and high availability.
The homelab cluster has two nodes, an HP T610 Thin Client with 4GB of memory and a Pentium G4400 system with 8GB of memory. These run two DietPi VM's with pihole as well as a general purpose Windows VM since my main computer is a 15" 2012 Macbook Pro. Eventually I'll add in an HomeAssistant VM along with a few NAS ones, whenever I discover how to squeeze more hours in a day.
I have two other clusters, one I cannot share as it's Chia (XCH) farming cluster that I maintain on behalf of clients.
The other is a passive income cluster that's basically an extremely overbuilt and overspecc'ed phone farm:
On my todo list, I need to setup a Proxmox Backup Server. Everyone who uses Proxmox highly recommends it. I have no backup system in place, but the VM's are spread across enough nodes that I'm alright with the resiliency that that brings.
I have 32TB of writes over 150 days with the same drive as yours, but that's on a node with ~190 VM's. This ends up being an average of ~200GB per day. I try to keep the SWAP usage below 1GB on this particular node.
On another node with a 120GB dram-less SATA ssd, I have 8TB written over 450 days. This appears to be about average as the other nodes with the same storage configuration. SWAP is at under 1% for all of them.
Proxmox really doesn't like it when a node is offline. I've been at times unable to create VM's or even start VM's on other nodes because another node was offline.
I can't imagine there to be much cost savings on your scale. A 41.6W load would consume 1 KWh per day, that's less than the cost of a cup of chai?
Generally, when I do need to restore a VM from backup, I first destroy/remove the VM it is replacing. When I remove a node from a cluster, I find it's best not to reuse the hostname or IP address. It can be done, but the ssh keys become a mess and you'll need to reset all of them on all nodes and it's just easier to pick a different hostname and IP for the new node.
The homelab cluster has two nodes, an HP T610 Thin Client with 4GB of memory and a Pentium G4400 system with 8GB of memory. These run two DietPi VM's with pihole as well as a general purpose Windows VM since my main computer is a 15" 2012 Macbook Pro. Eventually I'll add in an HomeAssistant VM along with a few NAS ones, whenever I discover how to squeeze more hours in a day.
I have two other clusters, one I cannot share as it's Chia (XCH) farming cluster that I maintain on behalf of clients.
The other is a passive income cluster that's basically an extremely overbuilt and overspecc'ed phone farm:
On my todo list, I need to setup a Proxmox Backup Server. Everyone who uses Proxmox highly recommends it. I have no backup system in place, but the VM's are spread across enough nodes that I'm alright with the resiliency that that brings.
I have 32TB of writes over 150 days with the same drive as yours, but that's on a node with ~190 VM's. This ends up being an average of ~200GB per day. I try to keep the SWAP usage below 1GB on this particular node.
On another node with a 120GB dram-less SATA ssd, I have 8TB written over 450 days. This appears to be about average as the other nodes with the same storage configuration. SWAP is at under 1% for all of them.
Proxmox really doesn't like it when a node is offline. I've been at times unable to create VM's or even start VM's on other nodes because another node was offline.
I can't imagine there to be much cost savings on your scale. A 41.6W load would consume 1 KWh per day, that's less than the cost of a cup of chai?
Generally, when I do need to restore a VM from backup, I first destroy/remove the VM it is replacing. When I remove a node from a cluster, I find it's best not to reuse the hostname or IP address. It can be done, but the ssh keys become a mess and you'll need to reset all of them on all nodes and it's just easier to pick a different hostname and IP for the new node.
superczar
Skilled
I have been using proxmox in production for about 5 years now in my home.
It runs 5 VMs (4 *nix, 1 windows) and it has performed without a glitch all these years.
The old rig I had it on was an i3 4th gen which was upgraded last year to an i3 10th gen. Moving the VMs from old node to new was trivial and too 30 -40 mins tops.
For home usage, single node is enough.
For backup purposes , I take a nfs backup once a month or two for each of the VMs.
The VMs are on a nvme SSD which is at about 98% health after about a year post the upgrade- At this rate, the SSD would anyway not be on the expected failure path for the overall system - hence I wouldn't worry about SSD writes / TBW at all for normal usage
Party Monger
Skilled
Can you post screenshot of the Smart data ? Cause I've read multiple threads that said get enterprise SSD or atleast premium ones like samsung evo plus with dram.
Have you shut down any parts of Proxmox or is this a default install.
Any videos or anything that show how to backup and move machines to other servers? Also any good bookmarks that you have that helped you understand it better? Cause im a linux noob. And while I can do basic stuff, anything complex and I have to rely on guides completely.
Im just testing various windows 7, 10 & 11 VMs right now. Will start checking out some containers too. If you have any interesting uses or apps etc please do let me know.
Once Im done testing. I will have one Win7 VM with my office apps servers. Then a VM to gather RSTP or OVIF streams from multiple cameras and storing it to USB HDD.
Rest will be linux and other experimenting.
Please do take some time and share more about your setup. and what you have learnt..
superczar
Skilled
nothing fancy - just a regular WD SSD.
and its a default install.
By and large, I used the documentation on the proxmox site/wiki. 99% of what you would want to do is already available there.
Having said that, I have been using linux for 2 decades now so some of the items that may have seemed easy to me may get a bit difficult for someone new to it - but given the reasonably good documentation available, it should be stil not too difficult.
Most things, including backups and node migration are available via GUI
Some things (e.g. assigning SATA disk to a specific VM) will require CLI
This node currently runs
- 1 instance of Win 10
- 1 instance of Ubuntu (for NAS, downloads, plex etc,)
- 1 X home assistant (home automation server)
- 1x debian for Omada controller (network and mesh roaming controller)
- 1 x debian for pihole
- 1 x debian for emoncms (data logging of various sensors)
I could probably combine a couple into a single instance - but why bother to do more work.
FWIW, this machine also used to run pfsense (and later sophos) as a load balncer/ router (and worked very well at that) - but I later switched to omada
Attachments
Party Monger
Skilled
Damn how are you using a dram-less SSD I've read reports of 1tb kingston drives going kaput in weeks. Check this thread -> https://forum.proxmox.com/threads/proxmox-crashing-and-killing-nvme-ssds.79626/
Oh that doesnt sound very good if it hates offline nodes
Its not so much the cost, Im just interested in efficiency man. What's the point of idling resources. Also with mesh wifi, Rpi4s and all the other stuff, the bill adds up quickly. And every thing extra goes in the highest unit rate.
Your drive has a 300 TB endurance so that makes sense. I guess my writes are also high cause I'm installing a lot of stuff right now?
How complex are these firewalls? And any real use apart from just tinkering? Like I want to know if there's any real world usage for these since I see so many people using them.
You've got a great setup, with a lot of headroom. Are you using any containers?
Im also planning to eventually get one of those google coral accelerator usb devices and run Frigate. Had seen a video on reddit of it alerting an owner about possible thieves touching the car. lol
GitHub - blakeblackshear/frigate: NVR with realtime local object detection for IP cameras
NVR with realtime local object detection for IP cameras - blakeblackshear/frigate
github.com
Last edited:
rsaeon
Patron
That would be pretty cool. We have a similar problem here with petrol thieves for cars parked outside.
Party Monger
Skilled
It is very cool. Its basically object detection coupled with recording streams from dumb cameras.
bobbyprajan
Adept
Recently built an LXD cluster on couple of Pi 4. Not yet decided what to run on it
superczar
Skilled
300TB endurance is the minimum you will see these days I guess..Damn how are you using a dram-less SSD I've read reports of 1tb kingston drives going kaput in weeks. Check this thread -> https://forum.proxmox.com/threads/proxmox-crashing-and-killing-nvme-ssds.79626/
Oh that doesnt sound very good if it hates offline nodes
Its not so much the cost, Im just interested in efficiency man. What's the point of idling resources. Also with mesh wifi, Rpi4s and all the other stuff, the bill adds up quickly. And every thing extra goes in the highest unit rate.
Your drive has a 300 TB endurance so that makes sense. I guess my writes are also high cause I'm installing a lot of stuff right now?
How complex are these firewalls? And any real use apart from just tinkering? Like I want to know if there's any real world usage for these since I see so many people using them.
You've got a great setup, with a lot of headroom. Are you using any containers?
Im also planning to eventually get one of those google coral accelerator usb devices and run Frigate. Had seen a video on reddit of it alerting an owner about possible thieves touching the car. lol
GitHub - blakeblackshear/frigate: NVR with realtime local object detection for IP cameras
NVR with realtime local object detection for IP cameras - blakeblackshear/frigate
The SSD on my MBA is at 1% after 25TBW so probably looking at something like 2000-2500TB endurance
AT a minimum, this WD should last 15 years or more which is way more than I can imagine using it for.
Regarding firewalls, most end users like I set them up for failover/loadbalancing 2 or more WANs
Other features like UTM and stats are a cool side bonus
Party Monger
Skilled
Guys what would you say is the ideal Proxmox setup?
1 SSD for Proxmox OS and 1 for VMs? Another for storage?
1 SSD for Proxmox OS and 1 for VMs? Another for storage?
fastapifastdisks
Adept
anyone tried running proxmos on a pendrive and the actual vm's in the ssd ? how long does your proxmox last,with esxi i have had pendrives killed in 13-14 months
Party Monger
Skilled
Wont last long. There are huge amounts of writes for logs etc
tech.monk
Adept
IIRC there is an module called log2RAM - not sure if it works with Proxmoxx
Party Monger
Skilled
I came across this thread -https://www.reddit.com/r/Proxmox/comments/ncg2xo/minimizing_ssd_wear_through_pve_configuration/
If you find more, do share. Log2ram I think only does it for system logs. Other Proxmox processes continue to do the normal stuff.