deusExMachina
Level E
Pretty much everything is in the title lol. I'm curious how much ram it tends to consume or if the snapshots degrade performance over time etc.
cp --reflink=always file1 copy1
Why do you take a snapshot though instead of just an incremental backup? seems like the same thing while not depending on a filesystem.1. Snapshots : I take all backups of a read-only snapshot. This way, backups are always consistent. Also, a snapshot itself is a poor man's backup, but it doesn't protect against hard disk failure. I tried very few snapshots with ZFS, but with Btrfs I have daily snapshots enabled and my script to delete old snapshots is not too reliable . Still I don't see any performance degradation with hundreds of snapshots. My disk usage is less than 70% for that filesystem, though.
I can see how this would be useful! So, you're saying that the reflink option only works in Btrfs?2. Copy-on-write : this is great. Especially large videos to edit / large virtual machine images to boot from. The below command on Btrfs (with ZFS on linux we need to clone a dataset, but the idea is the same)
Bash:cp --reflink=always file1 copy1
takes negligible extra space and finishes immediately, but creates an independent copy in file "copy1". Now small edits to copy1 will take very little space.
Never done this but I can see how this could be useful3. Multiple devices for a single filesystem : I have not made advanced use of this, but I did combine a partition each from 2 different hard drives to make a single Btrfs filesystem.
Do you know if ZFS auto checks the checksum? or would it detect issues only after a scrub? I personally liked how FreeBSD was put together but man they seem to have issues with hibernate and can't run a bunch of tools (that work on Linux).4. Checksum is an internal feature, giving us the peace of mind that data won't be corrupted by certain hardware errors.
Btrfs did screw up my filesystem once initially about 9 years ago, but it was partially my fault. I opened same virtual hard disk in 2 virtual machines simultaneously. I could not recover the data after trying very hard, some btrfs-progs tools were not available those days. Even fsck was new, if I recall correctly.
ZFS is difficult to manage on Linux because it needs to be installed separately after installing Linux, so cannot easily make it your one and only system drive, but this is not a problem on BSD.
I do incremental backup of snapshots. This is very important for consistent backups, which are necessary to avoid trouble after restoring. Taking an example that is simple and manual, but in reality it may be complex and files automatically created through programs :Why do you take a snapshot though instead of just an incremental backup? seems like the same thing while not depending on a filesystem.
#!/bin/bash
/home/deus/liftTheGodInCrane.sh
GODCOLOR=brown
#!/bin/bash
/home/deus/liftTheGodInCraneNew.sh
GODCOLOR=white
#!/bin/bash
/home/deus/liftTheGodInCrane.sh
GODCOLOR=white
No extra space, because both ZFS and Btrfs are copy on write filesystems. But more complicated to use on ZFS Linux than simply cp --reflink of Btrfs.I can see how this would be useful! So, you're saying that the reflink option only works in Btrfs?
Also, what does a dataset clone do on ZFS? do you need extra space/disk to make the clone? It's somethign I've never tried.
Yes, that is nice, though this is achieved with LVM also.Never done this but I can see how this could be useful
One thing I liked about ZFS is that the actual disk space seems to be shared between partitions in the dataset rather be prealloted like in a regular setup.
There are different kinds of checks at different times. I have forgotten some details. But it does store checksum. There is error correction in some circumstances (I think when there is replicating RAIDZ like 1, 5, 6 or 10) and only error detection in absence of RAIDZ, or RAIDZ 0.Do you know if ZFS auto checks the checksum? or would it detect issues only after a scrub? I personally liked how FreeBSD was put together but man they seem to have issues with hibernate and can't run a bunch of tools (that work on Linux).
I don't know about unraid or proxmox, seem to be virtualisation plateform. If so, try taking consistent backup rather than just copying VM image : copy is not backup. E.g. virtualbox and VMware support snapshots of VMs.Thanks for the detailed explanation. I have been researching Unraid, proxmox etc and has been ignoring this part. What do you think is the best for a proxmox server where I will ne backing up the VM itself everyday.
Also what's the effect of these fancy file systems on disk usage since ssds have a write limit. I had read that just in proxmox the write amplification can be 3-80times more than actual write in the VM. Is there any ram cache etc that can be programmed for these..
Thanks, you made it perfectly clear! the example was on pointInstead, if you took a snapshot of the filesystem and backup the snapshot, you get perfect consistent backup.
Are there any other advantages of a CoW filesystem? Also, doesn't journaling help avoid data loss during abrupt shutdowns? how does btrfs / ZFS handle this?No extra space, because both ZFS and Btrfs are copy on write filesystems. But more complicated to use on ZFS Linux than simply cp --reflink of Btrfs.
Really? the last time I experimented with LVM, the space usage / reporting was similar to traditional partitions.Yes, that is nice, though this is achieved with LVM also.
Btrfs and ZFS also do journaling and abrupt shutdowns have not been a data loss problem for ext3, JFS, ext4 and XFS also. Btrfs and ZFS protect from other problems - e.g. hard disk hardware problem when it writes 1 and reads 0. I've personally faced a problem with defective hynix RAM in 2007 which was silently corrupting my data. Btrfs and ZFS, due to checksums while saving and reading back, save us from those errors corrupting our data. Traditional journaling file systems don't do that.Thanks, you made it perfectly clear! the example was on point
Are there any other advantages of a CoW filesystem? Also, doesn't journaling help avoid data loss during abrupt shutdowns? how does btrfs / ZFS handle this?
Yes, you are right, I exaggerated LVM's functionality quite a bit . LVM does a very small part of what ZFS does in this case. LVM just lets us freely resize the "partitions" (volumes). But the filesystems contained in those volumes still pose problems.Really? the last time I experimented with LVM, the space usage / reporting was similar to traditional partitions.
I mean say you split a 1TB disk into four partitions and mounted say /home /usr /tmp /var/logs. df would show % usage with respect to the corresponding partition size. On LVM also, I saw that it was reporting as a % of the respective volume size.
On ZFS on the other hand, all of them would report freespace as a % with respect the the entire dataset size (1TB). This seemed neat to me because I wouldn't have to over allot space.