Windows Made a script to do backups and perform checksums, could use help testing it

deusExMachina

deusExMachina

Disciple
Hi,
This is essentially a follow up on my previous discussions here and here. The primary goal is to protect important data using backups. Linux has a lot of options for this purpose and file systems like ZFS/btrfs helps a lot.

On Windows, the options are limited to several paid software for backups. Also since ntfs has no scrub feature (this helped me detect silent corruption in the past), an alternative approach was needed.

I've made a python script to both do recursive checksumming and verification using sha256 (7z) to detect data corruption and also perform incremental/differential backups. It was validated on a bunch of test directories, which also additional error conditions BUT it would be awesome if you guys could try it out and let me know if there are any issues.

The script is all cli and is meant to be run periodically using the windows scheduler. It can be downloaded from here, the readme contains the usage instructions

@ibose , @vishalrao @kiran6680
 
Nice. I don't have Python on my Win system but can offer to test it on my Rpi over the weekend. Not sure if that will help though as this appears be Windows oriented.
 
@ibose Thanks! It should work on Linux too, just need to update the path to 7z. It's quite useful on Windows when compared to Linux (because of the stuff you get there anyway), however I've tried to keep it mostly OS agnostic.
 
For Windows, I take manual partition snapshots periodically using Macrium Reflect which are saved to my Xpenelogy NAS. My backups are on the NAS itself with copies on offline external USB drives. No backups on any of my Windows systems.
 
Wouldn't that generate a LOT of data? also why Xpenelogy instead of something like FreeNAS?

My plan is to keep the periodic backups on a different drive on the same PC. These would be synced to onedrive every now and then. Will also keep copies frequently on external drives.
 
deusExMachina said:
Wouldn't that generate a LOT of data? also why Xpenelogy instead of something like FreeNAS?

My plan is to keep the periodic backups on a different drive on the same PC. These would be synced to onedrive every now and then. Will also keep copies frequently on external drives.
Click to expand...
No, Macrium Reflect only saves the data instead of a bit-by-bit replica of the partition. And it allows you to set a number of backup policies based on time and space so you can rotate partition backups. The free version is pretty powerful. For files and folders, I use the free SyncFolders to push to the NAS on periodic basis. I do not have any automated or scheduled backups nor do any cloud backups. I only have preconfigured versions of both the partition and file/folder backup jobs which I run as and when I need.
Xpenelogy, because I fell in love with the Synology interface ever since I have their router. Its the best damn remote interface for a NAS on the planet. Unfortunately their NAS boxes are pretty pricey but when the option exists to roll your own, then so be it. It has been working pretty well and you get almost all the features of a Synology NAS on your own hardware (as long as its compatible).
 
Last edited:
I have a windows free household :), I'll try to adapt it to Linux some day. The github page looks really nice, with very thoughtful information given upfront.

To address one of your concerns raised here : " 1. The encryption key is stored in the script as plaintext for convenience. I generally dislike using passwords"

For this, encryption using public key cryptography works wonders. We only need the public key to encrypt, but even if this data including keys is seen by someone, they can't decrypt anything.

Code: 
gpg -e

FTW
 
Thank you! Parts of the readme were notes for my own use while I considered options and eventually made the script.

I haven't used gpg much, let me check what's involved in using it on windows. The thought did cross my mind but given how 7z handled most stuff including the encryption, just went with that.
 
1639033937896.png

FYI this is my GitHub handle https://github.com/naveenpothana
 
Last edited:
I've added you as a collaborator to that repository. Please accept the invite, it should be possible to do pull requests after that.

Thanks for the help!
 
Back
Top