Storage Solutions Need help with diy nas

Status
Not open for further replies.
6

6pack

Mastermind
All my 3 - 9+ year old hdd's are dying now one by one. I have around

2 x 500GB hdd [1 TB]
+ 5 x 1TB hdd [5TB]
+ 2 x 1.5 TB [3 TB]
+ 2 x 2TB hdd [4TB]
+ 2 x 3TB hdd [6TB]
+ 1 x 4TB hdd. [4TB]
Total : 23TB

All of them are completely full. There are some folders duplicated in 1 or two drives but most have just one backup. Probably around 15TB has no backup at all.

I have an AMD 8320 system with 8 Gb ram and support for 6 sata ports. I think the PSU in it - VX450 is going bad so i will change it soon. This system is housed in a cabinet with 4 hdd trays only. 5.25 drive cage can hold another 3 hdd's i think. Fifth slot is already occupied by dvd drive.

I'm thinking of using this machine as a nas now. it won't be on always, just for taking backups.

I'm thinking of using ubuntu or debian server and keep data os on 32GB ext4 formatted usb 3 flash drive. Rest hdd will be formatted btrfs. Using flash drive since it will free up one 3.5" hdd slot.

I've seen 2.5 x 4 bay hdd trays that fit in 5.25" slots. Are these good and will drives last or fail in these tiny boxes?
Example: Icydock

I guess I need sata port pcie cards too.

Can't think anything more than this because I'm not a networking guy. Budget is limited as in I will keep adding things when I get money. So I don't want to install server rack in my home where space is limited.

What would be the cheapest way for me to take backup of entire 15+ TB data? No raid please. I don't want to double up hdd drives too when they cost so much money now.
 
6pack said:
All my 3 - 9+ year old hdd's are dying now one by one. I have around

2 x 500GB hdd [1 TB]
+ 5 x 1TB hdd [5TB]
+ 2 x 1.5 TB [3 TB]
+ 2 x 2TB hdd [4TB]
+ 2 x 3TB hdd [6TB]
+ 1 x 4TB hdd. [4TB]
Total : 23TB
Click to expand...

That is 14 drives in all. Do you plan to stick with them or remove some?

All of them are completely full. There are some folders duplicated in 1 or two drives but most have just one backup. Probably around 15TB has no backup at all.
Click to expand...

Do you want to retain some backup such that the data is on 2 or more drives?

I have an AMD 8320 system with 8 Gb ram and support for 6 sata ports. I think the PSU in it - VX450 is going bad so i will change it soon. This system is housed in a cabinet with 4 hdd trays only. 5.25 drive cage can hold another 3 hdd's i think. Fifth slot is already occupied by dvd drive.

I'm thinking of using this machine as a nas now. it won't be on always, just for taking backups.
Click to expand...

Should work. You would need more SATA ports though.

I'm thinking of using ubuntu or debian server and keep data os on 32GB ext4 formatted usb 3 flash drive. Rest hdd will be formatted btrfs. Using flash drive since it will free up one 3.5" hdd slot.
Click to expand...

OMV/FreeNAS/Rockstor/UnRAID - the last one is paid.

I've seen 2.5 x 4 bay hdd trays that fit in 5.25" slots. Are these good and will drives last or fail in these tiny boxes?
Example: Icydock
Click to expand...
https://www.amazon.in/ICY-DOCK-FLEX-FIT-MB344SP-External/dp/B01IE37ZRU/

Yes, they should have a fan for cooling as well. HDDs will survive as long as they remain cool and vibration free.

I guess I need sata port pcie cards too.
Click to expand...

Depends on number of HDD. They are not cheap - good ones, so I would suggest to sell this machine and buy one with more ports, or reduce HDDs.

Can't think anything more than this because I'm not a networking guy. Budget is limited as in I will keep adding things when I get money. So I don't want to install server rack in my home where space is limited.

What would be the cheapest way for me to take backup of entire 15+ TB data? No raid please. I don't want to double up hdd drives too when they cost so much money now.
Click to expand...

Server rack will be too expensive - you cannot go that route.

I can lend you a SATA raid controller card, since I myself am not using my NAS system at the moment.

There is no cheap way out. You need to have tradeoffs - for example, if you want to save to cloud, uploading and downloading will take weeks and months. I think you will not prefer the DVD route. And HDDs are expensive but local access. Costwise - Cloud>HDDs.

I would suggest to segregate into 2 parts - important and non important data and then decide as needed.
 
  • Like
Reactions: 6pack and tommy_vercetti
vivek.krishnan said:
That is 14 drives in all. Do you plan to stick with them or remove some?

Do you want to retain some backup such that the data is on 2 or more drives?

Yes. First I thought only one backup is enough. But then it will be similar to what i have now. only one copy on failing hdd and no bck. I think at least 4 hdd's have bad secors in them and losing data on them is terrible.

Should work. You would need more SATA ports though.

OMV/FreeNAS/Rockstor/UnRAID - the last one is paid.

https://www.amazon.in/ICY-DOCK-FLEX-FIT-MB344SP-External/dp/B01IE37ZRU/

Yes, they should have a fan for cooling as well. HDDs will survive as long as they remain cool and vibration free.

Depends on number of HDD. They are not cheap - good ones, so I would suggest to sell this machine and buy one with more ports, or reduce HDDs.

Server rack will be too expensive - you cannot go that route.

I can lend you a SATA raid controller card, since I myself am not using my NAS system at the moment.

I'm thinking of getting 6 x 4TB WD Red's when i save the money. Selling the machine is out of question since it will mean more cost specially when ram prices have gone up a lot. I don't see a reason to buy new cpu / ram / mobo when the same ones i bought are costing more now than what i bought them for. I will have to get a psu though. Plus the amount i put into buying a new rig will be above 40k which could easily be used to buy new hdd's.

There is no cheap way out. You need to have tradeoffs - for example, if you want to save to cloud, uploading and downloading will take weeks and months. I think you will not prefer the DVD route. And HDDs are expensive but local access. Costwise - Cloud>HDDs.

I would suggest to segregate into 2 parts - important and non important data and then decide as needed.
Click to expand...

Cloud is out of the question. don't know when companies will suddenly stop the service. what is not with me is not mine.
I'd rather keep the data with me only. All data is important now since i don't know what will be important later. Its better to backup everything now than to lament later when i can't access the disks.

Is there some setting such that I copy all data once and then the system keeps 3-4 TB free so that in case one of the new hdd's fail, it can copy data to free space?
 
6pack said:
Cloud is out of the question. don't know when companies will suddenly stop the service. what is not with me is not mine.
I'd rather keep the data with me only. All data is important now since i don't know what will be important later. Its better to backup everything now than to lament later when i can't access the disks.

Is there some setting such that I copy all data once and then the system keeps 3-4 TB free so that in case one of the new hdd's fail, it can copy data to free space?
Click to expand...

What data is movies and TV shows and the likes which you can get later? That is not important.

Secondly, what data is going to be one time access and not even on monthly basis which you need to store away? That is also not required to be kept locally.

You are right - cloud services can go away anytime, but its not like it will suddenly shutdown. You will get ample time, especially if you go with the larger vendors like Microsoft and Google.

I would suggest to get the largest HDDs you can get, 4TB at the very least. Setup a Windows Server and Storage Spaces. If you want Linux - your call.
 
vivek.krishnan said:
What data is movies and TV shows and the likes which you can get later? That is not important.

Secondly, what data is going to be one time access and not even on monthly basis which you need to store away? That is also not required to be kept locally.

You are right - cloud services can go away anytime, but its not like it will suddenly shutdown. You will get ample time, especially if you go with the larger vendors like Microsoft and Google.

I would suggest to get the largest HDDs you can get, 4TB at the very least. Setup a Windows Server and Storage Spaces. If you want Linux - your call.
Click to expand...

none of the tv/movies i have can be gotten later. last time i lost 3TB data, i hardly got 70% of it back after waiting for close to a year. whats gone is gone. i know it seems silly to save them, but i wasted hours downloading them and wasted money on bandwidth. not saving them would be weird since i have yet to watch 50% of what i downloaded.

the extremely important data which is not anime or movies is around 200 GB only. I will get a 1TB 2.5" hdd to back it up asap.

i've had really bad experience with windows every time i used it. If an ntfs disk, even if its not the disk on which windows is installed fails then windows wont start properly. if any hdd has a few bad sectors windows will run like a snail etc. plus the random windows updates. lost important data because windows 10 decided to update randomly when i was shutting dsown the pc and then it wouldn't start and crashed - twice. even with a clean install, windows 10 crashes after updates and messes with hardware. i'm frankly fed up with windows now. linux is harder to set up but it does not do such things like download hundred's of updates as soon as net is connected and hog entire download bandwidth etc.
 
  • Like
Reactions: letmein and tommy_vercetti
6pack said:
none of the tv/movies i have can be gotten later. last time i lost 3TB data, i hardly got 70% of it back after waiting for close to a year. whats gone is gone. i know it seems silly to save them, but i wasted hours downloading them and wasted money on bandwidth. not saving them would be weird since i have yet to watch 50% of what i downloaded.

the extremely important data which is not anime or movies is around 200 GB only. I will get a 1TB 2.5" hdd to back it up asap.

i've had really bad experience with windows every time i used it. If an ntfs disk, even if its not the disk on which windows is installed fails then windows wont start properly. if any hdd has a few bad sectors windows will run like a snail etc. plus the random windows updates. lost important data because windows 10 decided to update randomly when i was shutting dsown the pc and then it wouldn't start and crashed - twice. even with a clean install, windows 10 crashes after updates and messes with hardware. i'm frankly fed up with windows now. linux is harder to set up but it does not do such things like download hundred's of updates as soon as net is connected and hog entire download bandwidth etc.
Click to expand...

Ok, in that case keep them. PMing you about another possibility.

Keep 2 copies of that 200GB data - external HDD and cloud.

Weird, that speed issue will happen irrespective of your OS, if you are accessing that sector which is corrupted. As for the data loss, that is something I have not seen yet, but i would suggest to check if your hardware was fine. Coming to updates, that is because of how Windows is, vs Linux.

Since you plan to use Linux, you will need to use mdadm. Make a RAID5 array - cheapest cost wise, and assuming you do have the needed backup power. Keep adding it over time.

If you do want to go with FreeNAS - I would suggest to get in touch with @cyberwarfare

@cyberwarfare Your points.[DOUBLEPOST=1515832659][/DOUBLEPOST]
mehargags said:
FreeNas with ZFS... FTW
Click to expand...

Do you run FreeNAS? Specs please.
 
Hi,

I just saw this thread (for some reason the earlier tag didnt reach me).

Ill be better able to help you if you could provide me with these details:

1) Whats your total budget for the current phase (upper and lower limits)
2) Seems like you want to store important data which cannot be obtained from other sources at a later point of time. Am i right?
3) Are you only considering local backups, or are you open to cloud storage also?
 
cyberwarfare said:
Hi,

I just saw this thread (for some reason the earlier tag didnt reach me).

Ill be better able to help you if you could provide me with these details:

1) Whats your total budget for the current phase (upper and lower limits)
2) Seems like you want to store important data which cannot be obtained from other sources at a later point of time. Am i right?
3) Are you only considering local backups, or are you open to cloud storage also?
Click to expand...

1 - have money for just 1TB hdd now. will buy additional stuff as and when i get the money to buy them. i wanted to make a plan which i can stick to.
2 - Only one drive is in really bad shape and its just 1TB. This is the only drive which has most imp data on it. The other stuff is less important and i only want to back them up when i have enough resources. You are right when you say the data cannot be obtained in a later point of time.
3 - local only. Don't have high speed broadband for upload / downloads. Just have a 2 Mbps data connection. upload speed is 1 Mbps. cloud storage is out of question.

This thread is just building a plan to meet my nas requirements. I want a NAS that is scalable on low budget and keeps 1 copy for safety in case of drive failure.
 
Do a Zfs setup of 1TB mirrored vdevs. My preference is FreeBSD (more scalable than Freenas). But you can use ZFS on linux as well. Works quite well with gentoo but other distributions may have it as well.
 
rajil.s said:
Do a Zfs setup of 1TB mirrored vdevs. My preference is FreeBSD (more scalable than Freenas). But you can use ZFS on linux as well. Works quite well with gentoo but other distributions may have it as well.
Click to expand...

Seems like Ubuntu has it from 16.04. I was thinking of using freenas or xubuntu runing from a usb 3 pendrive.

https://wiki.ubuntu.com/Kernel/Reference/ZFS

How do I go about doing this from scratch? Won't adding a disk to a new zfs pool format the original ntfs or ext4 partiton?

Edit: How true is this?
Code: 
For a small NAS where small addition of single drives is wanted to expand it BTRFS would be my pick, ZFS simply is not as flexible as BTRFS in that regard and not everyone is going to be able to add disks to a ZFS pool in groups of 4 or 8 etc. Then the other issue is that if you add vdevs to a pool they are striped so if a vdev dies the pool dies.

I don't think i can afford 4 new drives every time 1 want to upgrade my setup nor can i afford data loss if a pool dies dues to some bad config (which could happen eventually if a noob like me is running it)
 
Last edited:
My honest opinion here is that you save up some money and move to UnRaid.
It best seems to fit your needs (of flexibility of drives and not needing any special HW to run on).

Apart from that, you may choose to run RockStor.

Note: BTRFS is not stable, and i would be EXTREMELY cautious using it.
 
6pack said:
Seems like Ubuntu has it from 16.04. I was thinking of using freenas or xubuntu runing from a usb 3 pendrive.

https://wiki.ubuntu.com/Kernel/Reference/ZFS

How do I go about doing this from scratch? Won't adding a disk to a new zfs pool format the original ntfs or ext4 partiton?

Edit: How true is this?
Code: 
For a small NAS where small addition of single drives is wanted to expand it BTRFS would be my pick, ZFS simply is not as flexible as BTRFS in that regard and not everyone is going to be able to add disks to a ZFS pool in groups of 4 or 8 etc. Then the other issue is that if you add vdevs to a pool they are striped so if a vdev dies the pool dies.

I don't think i can afford 4 new drives every time 1 want to upgrade my setup nor can i afford data loss if a pool dies dues to some bad config (which could happen eventually if a noob like me is running it)
Click to expand...

I would stay away from BTRFS. ZFS is the way to go if you value your data. Setting up a mirror of 2 drives is very performant and safe. The only downside if that you will only be able to use 50% of diskspace (i.e. 1TB of 2TB). The rest of the disks you can set up as UFS, and migrate them over when you are able to buy more disks.
 
  • Like
Reactions: vivek.krishnan and cyberwarfare
yes, i think zfs is the right file system. thinking of installing openmediavault since it fits my needs of zfs plus storing media.
 
6pack said:
What blue print have you thought of for your NAS?
I don't think odroid is a good idea. All your data will be at the mercy of some USB power adaptor.
Click to expand...
Ok so cutting out odroid!
Im think a G4400 with other needed parts but my budget is mere 20k
Im using an OMV on an 4 year old pc since 5 days for testing on RAID 0 with sonarr,radarr,couchpotato,transmission, constatnly running 24*7. Will be testing DOCKER today. any help on build will be much apperciated
 
  • Like
Reactions: vivek.krishnan
Status
Not open for further replies.
Top Bottom