Proxmox Thread - Home Lab / Virtualization

[rsaeon]

My various Proxmox clusters are pulling 1500 watts at the wall so now I've a reason to pursue power efficiency over everything else.

I've a few Dell Optiplex SFF's and my plan was to get some more but I first need to see if the numbers make sense, for efficiency.

A 9020 configured with an i5 4690 processor, 32GB of memory and a 240GB ssd pulls 40w at nominal load. Which doesn't sound too bad but my thinking was that maybe a modern system with more memory would win out in the long run, because my workloads are entirely memory limited.

A Ryzen 2700X system with a MSI Tomahawk B450 MAX II motherboard, 128GB of DDR4 and a 512GB NVME drive with 4x of that same nominal load pulls 130w. So while this replaces 4 of those Optiplexes, it also reduces nominal power draw by 30w overall. Which is... fine, I guess? I expected it to be greater. I guess those Optiplexes are really well tuned in their power draw, which makes sense since they're meant for an office environment with little to no downtime.

But 30w power savings translates to a mere Rs 216 over the course of a month. For a power bill of 216 less, I'll be giving up the safeguard of having the workload divided over four systems and having the downside of 4x the number of VM's go offline when something goes wrong and waiting far longer for 4x vm's to spin up sequentially on the single host after a power failure.

My VM's are overprovisioned on memory so I need to stagger them to allow for KSM — Kernel Samepage Merging — to kick in 'naturally': https://pve.proxmox.com/wiki/Kernel_Samepage_Merging_(KSM)

I don't think I'm convinced if the power savings is worth it. Sure there's added infrastructure cost (more ethernet connections), but that's not much at all.

As for up-front cost, we're looking at 4.5k for the barebone, 4k for the processor, 2k for the ssd and 8k for the memory = 18.5k for the Optiplex.

For a modern system, it'll be maybe 12k for maybe a 3600, 7k motherboard, 3k ssd, 3k power supply, 2k case and 45k for the memory = 72k which is not that cheaper than 74k for 4 optiplexes.

So I'm in the odd position where it doesn't matter if I have 6 or 8 core nodes with 128GB of memory or 4 core nodes with 32GB, it's all mostly the same efficiency-wise.

That is very unsettling. How could something from 2015 be just as power efficient as something from 2018?

The 256GB memory density of X299/Threadripper is out of the question since the processor+motherboard cost would make the platform cost prohibitive — it'll have to be under 40k for both, which is not happening anytime soon for 12+ cores.

I feel like I'm missing something here but I don't even know what it could be.

 

[Mann]

I had a similar predicament and now going for a 12900t with a cheap motherboard. It has the best performance/watt (1/3 of a 5900x) with as many threads and supports an off the shelf motherboard and ram. It's 55k when imported from US but totally worth it.

 

[rsaeon]

I feel like I'm missing something here but I don't even know what it could be.

performance/watt

Ryzen 2700X

So this took an embarrassing amount of time to connect together but the 2700X is really not a power efficient processor. I mean it is, relatively, but there are far better options.

I tested the Ryzen 5 4500 which is very much capable of handling the same workload that the 2700X does and that system pulls a nominal load of 85w. This is far more in line with what I was expecting to see.

And so the power bill is down Rs 540 a month with a 128GB node that has a 65w processor. That's far more substantial over the course of a year than the Rs 216 I was getting with the 2700X's power draw of 130w (Rs 6480 vs 2592).

Liquidating four of the 32GB nodes for 12k each to replace with a single 128GB node requires an investment of 24k. For the 6k in electricity savings per year, it's not the smartest decision to make in our current times (inflation, weak rupee, crypto crash).

For now though, I'll experiment with undervolting the 2700X system.

 

[theGloaming]

Was referred to this thread months ago when I started looking at setting up Proxmox for a homelab in some detail (via this other thread - thanks rsaeon!).
Unfortunately things came up and I had to step away - but I have some time again now and am hoping to get the system up over the next 3-4 weekends.

I am still trying to work out storage. Current thinking is to go for:

• ext4 on Sata SSD for OS and VMs (not ZFS)
• HDDs for VM and other storage

For the OS and VM SSD, I have a couple of new 120GB Samsung 850 EVO disks at home.
This brings up the following questions:

1. Would these SSDs be sufficient (usage would be 2-3 linux VMs for now)? Considering I am not using ZFS, assume write amplification will not be such an issue.
   1. If so, should I set it up as a single SSD or a mirrored RAID1 SSD setup with 2 disks?
   2. If mirrored/RAID1, is it advisable on motherboard-RAID or should I look for a HW Raid PCI-e card?
2. If not, which SSDs would you suggest?

 

[30fps101]

For now though, I'll experiment with undervolting the 2700X system.

i also want to move my 2700x to my pve cluster, @rsaeon can please leme know what power draw you saw after the uv. also i have heard that ryzen has soft lockup in C6 state on linux, have you faced this issue ?

 

[Mann]

I think this can do a lot of things we have been talking about

 

[vishalrao]

I think this can do a lot of things we have been talking about

Omg omg omg! This is exactly what I want but preferably without windows and also barebones...

Are they reliable???
.

I mean is the seller reliable for delivery to Pune?

I plan to play around with opnsense, openwrt, linux/openbsd etc on this.
.

Update... I ordered it ₹25k flat. Shipping from Bangalore by Saturday. Apparently no windows tax.

Will let you know how it turns out after I get it.

 

[superczar]

Omg omg omg! This is exactly what I want but preferably without windows and also barebones...

Are they reliable???
.

I mean is the seller reliable for delivery to Pune?

I plan to play around with opnsense, openwrt, linux/openbsd etc on this.
.

Update... I ordered it ₹25k flat. Shipping from Bangalore by Saturday. Apparently no windows tax.

Will let you know how it turns out after I get it.

Are you planning to run multiple VMs on this?
The CPU may fall woefully short on that front.
This one will be excellent as a dedicated pure opnsense machine but the price is kinda high just for that .

FWIW, I spent just a tad over this amount for a larger form factor traditional machine .. opnsense is one of the 7 VMs running on it and does a perfect job in virtualised mode

 

[Mann]

Generally speaking, it is recommended to router/firewall software on dedicated hardware. I mean no one can stop you from it, but it's not recommended. In which case this hardware will be great, not cheap but almost perfect.

 

[vishalrao]

@superczar oh I'm not going to bother with a hypervisor, at least not long run, maybe just temporarily for learning and playing around with various vendors.

I mainly want to play around bare metal installs of stuff like opnsense, openwrt, linux and maybe openbsd for load balancing multiple ftth links...

Basically imitate the use case of my current tplink er605 router.

I like that it's a 11th gen n5105 4 core CPU without the e core p core hybrid nonsense.
.

IMHO the price is acceptable even if it turns out to be totally barebones without the 8gb and 256gb nvme lol. Just the CPU and quad 2.5gbe NIC makes it acceptable for me versus me going DIY for similar configuration.

 

[superczar]

@superczar oh I'm not going to bother with a hypervisor, at least not long run, maybe just temporarily for learning and playing around with various vendors.

I mainly want to play around bare metal installs of stuff like opnsense, openwrt, linux and maybe openbsd for load balancing multiple ftth links...

Basically imitate the use case of my current tplink er605 router.

I like that it's a 11th gen n5105 4 core CPU without the e core p core hybrid nonsense.
.

IMHO the price is acceptable even if it turns out to be totally barebones without the 8gb and 256gb nvme lol. Just the CPU and quad 2.5gbe NIC makes it acceptable for me versus me going DIY for similar configuration.

On a side note, FWIW, I was on pfsense and then sophos for a while but had to switch to opnsense after airtel upgraded to dual stack ipv4/6
sophos and pfsense have broken/ incomplete support for ipv6 still
neither does er605

OpenBSDis probably the best possible option but despite the GUI and large user base, i had to struggle quite a bit to get opnsense right. I suppose configuring openbsd as a router will probably be even more effort intensive
If loadbalancing is your primary goal, opnsense has excellent support for that- failover switching takes just a few seconds at most if configured per your preference
Give taht a try first

Generally speaking, it is recommended to router/firewall software on dedicated hardware. I mean no one can stop you from it, but it's not recommended. In which case this hardware will be great, not cheap but almost perfect.

Yes, however depending on the use case, it may be overkill (a dedicated machine, that is)
I have been using virtualized routers for some time now - performance is same as bare metal for all intents and purposes.

Since I anyway needed a couple of home servers and it would have been cost/ power intensive to have separate instances for each, I had them virtualized already.

So when it was time to move swap physical router with a router /FW, I decided to reuse the same box - TBH, i thought it would be stop-gap because I wasn't able to find a good PC like the one linked in this post.
Turned out it's fine any which way.. The NICs are PCI passthrough-ed to the VM running the router OS so there is no perf degradation.
In my case, I had to use 4 NIC + mobo port : 3 for WAN1-WAN3 (pass through) , 1 for LAN (pass through) and 1 for the hypervisor bridge itself

Just to be on the safe side, I have a replicated config (same ISP config, same IP range, same static IP config etc ) on an er605 in standby - just in case the server goes down some day , it wil be a simple matter of unplugging the RJ45s and plugging them into the ER605.
Although even if it does, redeploying the backup from periodic snapshots will not take long.

 

[rsaeon]

i also want to move my 2700x to my pve cluster, @rsaeon can please leme know what power draw you saw after the uv. also i have heard that ryzen has soft lockup in C6 state on linux, have you faced this issue ?

I've had system halts due to USB3 XHCI issues on 1st gen Ryzen — XHCI handoff needed to be disabled. Some more information about that here:

https://www.reddit.com/r/linuxquestions/comments/l3ab9w/_/ick3sg9

I used to have power related system lockups, but they went away after I changed "Power Supply Idle Control" from "Low Current Idle" to "Typical Current Idle". This needed to be done on both 1st and 2nd gen Ryzen. Some talk about this here:

208201 – AMD Ryzen 7 1800X soft lockup

bugzilla.kernel.org

I actually hadn't had the downtime to try undervolting, I'm running it stock. The work I do keeps the CPU pinned above 50% constantly so I never need the power savings of C6 states.

However, the 2700X is drawing about 75% power more than another system with a Ryzen 4500 at stock, with the exact same workload across both:



This translates to an electricity bill that is higher than necessary — about 500 per month, or about 6k per year.

I'll be upgrading my nodes to 5600G's by the end of the year, if finances allow.

 

[30fps101]

However, the 2700X is drawing about 75% power more than another system with a Ryzen 4500 at stock, with the exact same workload across both:

View attachment 173230View attachment 173231

This translates to an electricity bill that is higher than necessary — about 500 per month, or about 6k per year.

I'll be upgrading my nodes to 5600G's by the end of the year, if finances allow.

Thanks for your reply, i guess I will sell off 2700x and stick to coffee lake, most of the time my boxes idels so lower power states are important for me. One more thing, what are you using to measure the power draw over the period of time some kind of smart plug or ups ?

 

[rsaeon]

One more thing, what are you using to measure the power draw over the period of time some kind of smart plug ?

I'm using Polycab's Smart Plugs, these are the 10 amp versions. They're running Tasmota, a custom firmware that eliminates the dependency of a cloud service for automation: https://tasmota.github.io/docs/

They're one of the more difficult smart plugs to convert to Tasmota, the process for the 10 amp is almost exactly the same as the 16 amp version guide I posted here:

User Guides - Teardown and preparation of Polycab's 16A Wi-Fi Smart Plug for Tasmota

This is a teardown and a guide about preparing a Polycab Hohm Lanre 16A Smart Wi-Fi plug model SLV1910001 for a Tasmota installation. A heavy vise can be used to clamp down opposing corners, the compression will cause the casing to separate. The tip of a box cutter can then be run around...

techenclave.com

I haven't set up any historical logging or graphing yet, other members have. The most I've done is to automate power up by power cycling the smart plug, this is usually done after a safe shutdown when there's a power cut. The plug stays powered on since its on an inverter, so when power returns it is turned off and on again to get the server to start up since the server is configured to power on after a power cut in the bios.



The automated power up is done after monitoring AC power for 10 mins for stability, notifications are sent on Telegram with a bot.

 

[Mann]

Yes, however depending on the use case, it may be overkill (a dedicated machine, that is)
I have been using virtualized routers for some time now - performance is same as bare metal for all intents and purposes.

Ok, I assumed people would know but just to clarify, running it bare metal was for security reasons not for resource optimization. All this software is designed security in mind (I'm talking about a firewall). And running it on a VM may open security vulnerability.

 

[MarioBros]

Just seeking help from experts here - I am noob

How do I power a ES8266 board with batteries ? If my DIY project needs 5V power supply

 

[superczar]

Ok, I assumed people would know but just to clarify, running it bare metal was for security reasons not for resource optimization. All this software is designed security in mind (I'm talking about a firewall). And running it on a VM may open security vulnerability.

That makes sense for an office firewall where you can get targeted attacks - also for a office setup, no one will use a vitualized firewall

But for a home router - where consumer routers do not even have a firewall for ipv6 to begin with, a virtualized instance is already a pretty strong padlock where there was none to begin with.

Additionally, breaking through Linux KVM virtualization layer to gain access to the system kernel is far from a trivial task even for a targeted attack on a corporate, let alone someone attempting it on a home server

Just seeking help from experts here - I am noob

How do I power a ES8266 board with batteries ? If my DIY project needs 5V power supply

use 2S lipo batteries and a buck converter like LM2596 to set output to 5V

 

[Mann]

If not for security, Why use it in the first place??
Just to convince yourself there is a firewall protecting you?

Additionally, breaking through Linux KVM virtualization layer to gain access to the system kernel is far from a trivial task even for a targeted attack on a corporate, let alone someone attempting it on a home server

How do you not know there exists a button to hack through it all?

 

[superczar]

If not for security, Why use it in the first place??
Just to convince yourself there is a firewall protecting you?

How do you not know there exists a button to hack through it all?

I think you are missing the point altogether

With ipv4, all your devices were sitting behind a NAT layer. unless you explictly enabled port forwards, an external actor had no way to establish a connection with any device behind your router . - at least not unless there was a rogue device sitting within your network establishign outward connections.

With ipv6, every single device sitting behind your router has its own externally addressable address on the web. There is zero protection - literally... not even a semblance of a door left unlocked.

Any ipv6 firewall that you setup is a big layer of protection on something thats entirely unprotected -

Now we can get into the weeds and argue that a bare metal firewall is a reinforced titanium door with impenetrable locks
vs
a hypervisor firewall being a vanilla titanium door with potentially penetrable locks

or we could argue that the latter is also largely impenetrable.
But it would be silly to say that KVM virtualization is so weak so as to bebypassed by a click-of-a-button attack by any Tom DIck or Harry who wants to call himself/herself a hacker


PS: Want to test this out? If you are on an ipv6 connection, note down the ip6 address of your windows rig (for example) and then use the remote desktop client from your phone (on LTE/5G) to log into it.. see what happens

 

[napstersquest]

I have a large PC with 5600X, 32GB RAM, 2x nvme, 4x Optane 16GB, 6x HDDs. Also contains 1060 3GB.
It takes a lot of space as it is built in a CM Storm Scout 2 Advanced case.

It is also underutilized, the compute is overkill and storage is 30% used (around 5TB out of 16TB - drives are in RAIDZ2).

I have a network rack with around 15 gigabit ports on a switch unoccupied. It has some space which I can use for some tiny PCs like the P330 I bought from you.

Is there a configuration like Esxi/ProxMox where I can cluster these multiple tiny PCs to have redundancy (redundancy in compute isn't really necessary, just for storage)?

Currently, in Truenas, if a drive fails I can just replace it and rebuild the array. In case any drive fails, I would like the recovery to be quick and easy.
I don't need a discrete GPU as intel iGPU should be enough to do the job currently the 1060 does - transcoding for Jellyfin which is also rarely useful.

Thanks in advance!