To zip, iso or store plain!

@nRiTeCh I'm a bit confused - is compression a requirement for you or are you just looking to prevent bitrot? Because I don't see any reason to archive the data if it's the latter.
 
Yup. Even zip files go bad. It might open fine and you only get half the contents back instead of everything. Happened to me when i was saving things compressed to save disk space instead of buying new hdds. Rar+par is not worth the effort if you have constantly updating data or folders. Was saving to DVDs a lot but got fed up and just dumped to hdd. Some DVDs have gone bad in these 20 years and all the old hdds were replaced by new ones every 3-6 years i think.
 
nRiTeCh said:
View attachment 157981
This is how less than 1% pics have gone bad. Unsure if this is corruption or what.

Also, 6yrs ago when primary pic storage hdds partitions vanished all of a sudden, I was able to recover 98% data but some files were zero bytes, some showed proper file size but very small images. Need to understand this behavior as well.
Click to expand...
This is data corruption caused by either bit rot/hdd or ram bit flip errors(possible if using xmpp in bios/ram overclocking & using typical non-ecc ram as ecc ram is sold as server memory & much costlier). Bitrot can be prevented by using multiple storage devices & using checksums of known good data but ram bit flip errors are theoretically impossible to prevent when using non-ecc ram. That is why I always spend at least 8-10 hours stress testing my any newly installed ram using xmpp profile by running a prime95 customized test especially for ram testing(default prime95 testing stress test only processor). For videos, bit flip ram errors don't matter in majority scenarios(except for video editing scenarios) as it may result in just few corrupted pixels out of billions in a hour long videos which won't even be noticeable while watching but for pictures it is quite noticeable if you are unfortunate enough to encounter it. The ram bit flip error basically corrupt data when something is loaded in ram & then written back to hdd/ssd(like all photo/video editing software) or you use "system memory" as cache while transferring data(like all the good dram less ssd like wd sn570/samsung 980 do) or you download something over internet/intranet & windows system caching is being used by default.
@blr_p @lockhrt999 @booo @Mr.J @ibose @6pack @SirVer
 
guest_999 said:
This is data corruption caused by either bit rot/hdd or ram bit flip errors(possible if using xmpp in bios/ram overclocking & using typical non-ecc ram as ecc ram is sold as server memory & much costlier). Bitrot can be prevented by using multiple storage devices & using checksums of known good data but ram bit flip errors are theoretically impossible to prevent when using non-ecc ram. That is why I always spend at least 8-10 hours stress testing my any newly installed ram using xmpp profile by running a prime95 customized test especially for ram testing(default prime95 testing stress test only processor). For videos, bit flip ram errors don't matter in majority scenarios(except for video editing scenarios) as it may result in just few corrupted pixels out of billions in a hour long videos which won't even be noticeable while watching but for pictures it is quite noticeable if you are unfortunate enough to encounter it. The ram bit flip error basically corrupt data when something is loaded in ram & then written back to hdd/ssd(like all photo/video editing software) or you use "system memory" as cache while transferring data(like all the good dram less ssd like wd sn570/samsung 980 do) or you download something over internet/intranet & windows system caching is being used by default.
@blr_p @lockhrt999 @booo @Mr.J @ibose @6pack @SirVer
Click to expand...
But then there are the cosmic rays -
superuser.com

How come bit flips aren't destroying my computer?

Tech YouTuber ThioJoe made a video about ECC memory, in which he explained how bits in memory can randomly change and can lead to data corruption if you don't have ECC memory. Here's what an article
superuser.com superuser.com
Me, I would not worry that much as long as periodic multi-level backups are taken.
 
SirVer said:
@nRiTeCh I'm a bit confused - is compression a requirement for you or are you just looking to prevent bitrot? Because I don't see any reason to archive the data if it's the latter.
Click to expand...
No compression!
guest_999 said:
This is data corruption caused by either bit rot/hdd or ram bit flip errors(possible if using xmpp in bios/ram overclocking & using typical non-ecc ram as ecc ram is sold as server memory & much costlier). Bitrot can be prevented by using multiple storage devices & using checksums of known good data but ram bit flip errors are theoretically impossible to prevent when using non-ecc ram. That is why I always spend at least 8-10 hours stress testing my any newly installed ram using xmpp profile by running a prime95 customized test especially for ram testing(default prime95 testing stress test only processor). For videos, bit flip ram errors don't matter in majority scenarios(except for video editing scenarios) as it may result in just few corrupted pixels out of billions in a hour long videos which won't even be noticeable while watching but for pictures it is quite noticeable if you are unfortunate enough to encounter it. The ram bit flip error basically corrupt data when something is loaded in ram & then written back to hdd/ssd(like all photo/video editing software) or you use "system memory" as cache while transferring data(like all the good dram less ssd like wd sn570/samsung 980 do) or you download something over internet/intranet & windows system caching is being used by default.
@blr_p @lockhrt999 @booo @Mr.J @ibose @6pack @SirVer
Click to expand...
I wont agree at all with the ecc/non-ecc/xmp/flip-flops etc. and yeah not even stress-tests. I used to OC but never stressed my pc for 8-10hrs like some anxious person willing to launch some satellite on Pluto! A 10-15min test was always sufficient for me even if I OCed to more than 50% of existing capabilities!

Coming back to data corruption part, why only images get affected why not other file formats?
BTW the images which got corrupted were actually recovered from the then image storing hard drive on which they were stored primarily and whose partitions vanished overnight.
I recovered the partitions and then the entire data. I still have that hdd but now used for storing games/movies/torrents etc. data and zero corruption since 5+ years!
 
Last edited:
ibose said:
Me, I would not worry that much as long as periodic multi-level backups are taken.
Click to expand...

nRiTeCh said:
I wont agree at all with the ecc/non-ecc/xmp/flip-flops etc. and yeah not even stress-tests. I used to OC but never stressed my pc for 8-10hrs like some anxious person willing to launch some satellite on Pluto! A 10-15min test was always sufficient for me even if I OCed to more than 50% of existing capabilities!

Coming back to data corruption part, why only images get affected why not other file formats?
BTW the images which got corrupted were actually recovered from the then image storing hard drive on which they were stored primarily and whose partitions vanished overnight.
I recovered the partitions and then the entire data. I still have that hdd but now used for storing games/movies/torrents etc. data and zero corruption since 5+ years!
Click to expand...

I used to think the same way until someone I know lost many pics because of ram bit flip errors(same horizontal wide line across top/bottom of pic though in this case it happened most likely due to hdd data corruption). Anyway, running a test for 8-10 hours once a year is no big deal for me as my luck has always been on the bad side most of the time(aka that 1 unlucky guy among 100 to face that particular issue). It is most likely that partition recovery was not 100% correct(that is why professional data recovery ppl first create an image of target hdd & then do recovery operations on the clone of that image to see if the required data is recovered correctly 100% & if not then try again with other tools/techniques with another cloned image).
 
nRiTeCh said:
Coming back to data corruption part, why only images get affected why not other file formats?
Click to expand...
It affects any file and is down to where on the HDD they reside.

What percentage do Image files make up compared to other file formats on your storage
 
Although all the systems at work were servers. Ecc memory becomes somewhat irrelevant. Read what is known as T10DIF
 
6pack said:
I searched for dif and apparently the Linux kernel has supported this type of data integrity checks.
Interesting read about it in the docs.

Click to expand...
Just be careful; although they say its part of mainline there could be gotchas. No one uses sata drives in enterprise setting. So these features would be well tested for sas drives and such. I remember we had so many regressions on our storage array when t10dif was enabled. But that was long time ago. I just realized that this was mainline after clicking the link you provided.
6pack said:
Does raid really protect data? I've always heard raid is not backup.

Click to expand...
Funny this guy completely fumbled. Got confused between checksum and parity. And you can definitely recover raid when controller dies, there is a common practice of labeling the drives with stickers for this exact reason. His explanation of raid 5 is totally wrong. (raid 5 stores parity not checksum)

This could be good guide to raid interview questions… but…
0 - stripe
1 - mirror
5 - parity
10 - mirror while striping
50 - mirror, stripe, parity
60 i think is double parity. There are more distributed raid types.
Now coming to spinning disks to wards the end of the video… we are currently working on something like raid on enterprise nvme drives. So raid only for spinning disks is bollocks.

Also, there is one more technology. Its called erasure coding. Very popular in the cloud because it requires 1.4 storage to achieve something like raid 10. So money… if you want ec at home; use ceph-fs on a raspberry pi for home nas systems.

Also, modern systems not only raid but also create storage tiers. Like using nvme for cache etc…you can try all of that on ceph. Just create a kubernetes instance of ceph and ceph dashboard will let you configure all of that.
 
Last edited:
Back
Top