To zip, iso or store plain!

nRiTeCh

nRiTeCh

Level N
I have TBs of data pertaining to many memories of family, tours etc. captured in images and videos and stored across few hard drives.

Since 3 week s I have undertaken the job of complete data consolidation across all hard drives. (sort and categorize data, delete duplicates and dump the needless one)

During this course, I came across few pics which were captured crisp clear yet they now appear corrupted or choppy or completely greyed out from one corners or side.

This is a worry factor as all hard drives are in perfect healthy state with no issues and this is happening.

Now, thinking to either zip/rar the files/folder-wise or create a single iso year-wise.

Looking for suggestions to preserve the data offline for years to come.
 
If you use any of those techniques maybe provide PAR files for recovery? You decide on how much redundancy of the PAR file
 
Videos, and audio don't compress well in any of the those form. And ZIP and ISO are the worst format to store them.

Photos taken in JPEG format do compress a lot in RAR format but even then it varies from picture to picture.

Our digital cameras have advanced by leaps and bounds in the last few years. The photos and videos are exactly what it was like a few years ago. You just have fond memories of them.

Just like in video games, contra and Mario were always pixelated but our childhood memories makes them uber high fidelity ones. But in reality they are exactly as they were.

In my opinion it's better to save them just as is. But if you really want to save space on pictures then you can use WINRAR to make them into RAR files. One RAR file for one year is good enough. And make sure to select "solid archive" , "best compression" and also add recovery health. That was in case your RAR is somehow corrupted , then you can use that recovery file to restore the RAR file.

Another thing which you can do is to save your most critical data onto a Blu Ray disc and save the disc in a proper casing. I know it's not really possible for TBs of data to store in this way but i have some 10-15 year old DVD which still work fine till date. VERBATIM was the brand of DVD. I don't know if they even make Blu ray discs anymore.
 
This is old school stuff that i used to do with CD's and DVD's

QuickPar - Verifying and Repairing

QuickPar is a PAR 2.0 recovery volume creation tool. Use it to protect and repair any data file.
www.quickpar.org.uk

Sometimes the disc would develop a problem and after recovering the file it would miss bytes here and there. So the PAR file was like a backup file you made along with the disk and if there was a problem offered a way to recover data in case of corruption.

QuickPar tutorial | Binaries4all Usenet Tutorials

You can use PAR2 to check whether your downloaded files were received completely and correctly and you can repair any missing or corrupt files if necessary!
www.binaries4all.com

Should not need it these days but it is an option if you want.

en.wikipedia.org

QuickPar - Wikipedia

en.wikipedia.org en.wikipedia.org

No development since the last 18 years. It works so nothing more required but it means there is no window 64bit client
 
Once you've kept things however you have wanted, it would be a good idea to catalogue them using VVV. We are using this to keep tracking of data in our lab, very handy, esp when each drive is labelled (Both when mounted and externally labelled).

Store your hard drives safely!
 
1674212691469.png

This is how less than 1% pics have gone bad. Unsure if this is corruption or what.

Also, 6yrs ago when primary pic storage hdds partitions vanished all of a sudden, I was able to recover 98% data but some files were zero bytes, some showed proper file size but very small images. Need to understand this behavior as well.
 
nRiTeCh said:
View attachment 157981
This is how less than 1% pics have gone bad. Unsure if this is corruption or what.
Click to expand...
Yeah, that's corruption. A few bytes got changed. If you prepared a par file with say 10% redundancy you would be able to recover it.
nRiTeCh said:
Also, 6yrs ago when primary pic storage hdds partitions vanished all of a sudden, I was able to recover 98% data but some files were zero bytes, some showed proper file size but very small images. Need to understand this behavior as well.
Click to expand...
Same thing, the sectors where the files were located on the HDD could not be read so the recovery software you used put 00 in place. Or where larger sections were gone it left them blank which is why the file size is small.

I don't know whether this kind of problem can happen with the present generation HDDs. They swap sectors around if they sense things are going bad.
 
How foolproof and trustworthy is Rar recovery record thing?

If I zipped 1gb worth images in a single rar and if 40% of the rar gets corrupted/damaged, will there be a 100% data recovery?
 
You can simulate it in a hex editor with a smaller amount.

Rar up some small file, open in hex editor, replace x% bytes with zero here and there and see if it works
 
nRiTeCh said:
How foolproof and trustworthy is Rar recovery record thing?

If I zipped 1gb worth images in a single rar and if 40% of the rar gets corrupted/damaged, will there be a 100% data recovery?
Click to expand...
If 40% of rar is corrupted then you'll lose 40% of the data. Provided header of the file is fine.

Zipping photos and videos doesn't make any sense. You are adding headache without any possible plus sides. It has zero use for the data protection. If the header of the zip file gets corrupted then you will lose 100% of the data even if rest 99% is fine.

Compress your videos and photos using media compressors like irfanview and handbrake. There's no reason an old photo should be 5mb. Compress that photo to 300kb. And put your multiple hdds into some RAID to protect your data.
 
lockhrt999 said:
If 40% of rar is corrupted then you'll lose 40% of the data. Provided header of the file is fine.

Zipping photos and videos doesn't make any sense. You are adding headache without any possible plus sides. It has zero use for the data protection. If the header of the zip file gets corrupted then you will lose 100% of the data even if rest 99% is fine.
Click to expand...
Agree but if you add parity info you can recover that much in corrupted data.
lockhrt999 said:
Compress your videos and photos using media compressors like irfanview and handbrake. There's no reason an old photo should be 5mb. Compress that photo to 300kb.
Click to expand...
Disagree. I's always better to preserve the data of old photos in their entirety instead of throwing away data. If you ever need to photoshop them in the future, you will be glad you did.
lockhrt999 said:
And put your multiple hdds into some RAID to protect your data.
Click to expand...
fine
 
blr_p said:
Agree but if you add parity info you can recover that much in corrupted data.
Click to expand...
I dunno how parity info is saved with zipped files but I don't think it can recover 40% of data. Few percentage maybe at best. To create parity data of zip file which can recover from 40% data loss, would inflate zip file much above the original file size.

blr_p said:
Disagree. I's always better to preserve the data of old photos in their entirety instead of throwing away data. If you ever need to photoshop them in the future, you will be glad you did.
Click to expand...
Old photos taken with old devices had weak processors. They lacked the processing power to compress the photos. That's why you get big file sizes. Today on computers you can compress photos to just 10% without losing any visual details.
 
lockhrt999 said:
I dunno how parity info is saved with zipped files but I don't think it can recover 40% of data. Few percentage maybe at best. To create parity data of zip file which can recover from 40% data loss, would inflate zip file much above the original file size.
Click to expand...
40% is a lot, you can use a lower ratio. And that is recovery from any random loss that does not exceed that amount.
lockhrt999 said:
Old photos taken with old devices had weak processors. They lacked the processing power to compress the photos. That's why you get big file sizes. Today on computers you can compress photos to just 10% without losing any visual details.
Click to expand...
This is more true with video than images I think.
 
6pack said:
Does raid really protect data? I've always heard raid is not backup.

Click to expand...
You heard correct. RAID helps you create a resilient system in case of disk failure. But that's not good enough if you want proper redundancy. You need to setup a separate storage to keep another copy of your data.
 
I was always against raid as always required a sure-shot backup solution and some known experts who ever relied on raid truly as a backup ended up trying to rebuild their disks with little success coupled with horrifying nights and no straight forward solution. (data was lost)
lockhrt999 said:
Old photos taken with old devices had weak processors. They lacked the processing power to compress the photos. That's why you get big file sizes. Today on computers you can compress photos to just 10% without losing any visual details.
Click to expand...
Not true. I have been working with images since very long time, a compression itself means 1% compromise on quality and more the C more the later C!
Once you compress there's no straight forward aka simple way to retrieve the original form unless some complex post-process for hd/4k thing which again contributes hugely on the in eased file size!
 
Okay! I will try to put my expert hat on…
What you described is called as “silent data corruption” (google it.)
Now coming to your question about whether to zip it or raid it other anything else…
There are two approaches to safeguarding your data. One is to checksum it and other is to recover it when something goes wrong.
Take an example of zipping… zip along with compression also has checksums so you will know if the data is good or bad. But is compute intensive ( takes hours and burns coal hence, money )

Other is raid, where your data is sharded and put into multiple drives and then checksummed so you can retrieve upto on damaged drive (20%) and rebuilding raid is also time consuming and will not guarantee if more than one drive fails. (Lets talk about raid 60 and erasure coding some other day)

For data replication you not only want to safeguard the data but also want to recover it and journal it ( go back in time, like a Time Machine) for this you would need a little bit more advanced system.

Because i am an expert :p I am also too lazy, also the storage is getting cheaper by the day. Instead of spending money on electricity for running complex algorithms for checksums etc, i simply buy an extra big hdd and make one more extra copy. All my photos are stored in multiple external hdds and thumb drives strategically thrown around the house.

Being all that said, if you insist on raid like system, i would use minio and erasure coded setup to store data on a xfs/zfs/btrfs(butterfs) like file system which allows cow snapshots at regular time intervals on a home kubernetes cluster built using multiple raspberry pis which provides s3 storage and exported through router configured to dyndns on google domains exposed on my personal domain.
 
Back
Top