Managing ernormous personal Photos/Videos and automating on PC/Android/NAS

Naveen_Reloaded

Naveen_Reloaded

Adept
Source of Media :

1. My phone
2. My sony a6400 Camera
3. Mom Phone
4. My wife phone
5. Online downloads for personal and for office works.

Storage :

1. Individuals phones
2. Me and my mom have Google one Subscriptions
3. NAS( UNRAID )
4. PC

The problem :

1. Our phones are getting full rapidly
2. I need one place to pour all contents and manage into (possibly into Unraid)
3. sync on other cloud service (blackblaze ? or other similar ? AWS ?) as final backup
4. Managing , sorting and syncing

first and foremost , what should i do to first sync all these files to local NAS ? All have android
I want individual person to have their own space on UNRAID , but they could share a part to common pool , say for example the camera uploads . Whatsapp and other media can remain in thier own personal space on unraid.
How to sort and manage these medias ? If i am going to automate , what should be the folder structure ?
I work with lightroom / PS etc , but still struggling to use LR to manage my photos.. when i start syncing all photos onto Unraid , how can i used LR with unraid ? Should i allow LR to manage the media or some docker app on unraid to manage ?
How to sync all uploaded files to other cloud service for storage alone.
I want the common folders to be either visible on all devices or even sync with all devices( for eg official document PDF`s like PAN , driving license etc..)

I think you guys can get the picture here , rather than me going into further details.
 
You can try syncthing for one part of your requirement.

Install syncthing on mobiles and the backup server. Each phone has its own share for media and deletions on mobile are handled with one way sync and a custom versioning script copying deleted pics to another main media directory. This way even when a pic is deleted, there's still a copy on the server.

Consider these directories as live copies of mobile and do not modify them directly. Only work/organize the dedicated copies in the main media folder.

For shared documents create a common syncthing share and add it to all devices that need these files.

For backup to cloud you can use rclone.

Don't have any suggestions on how to organize though. For sharing the media locally you can use any of the local network file share like smb or nfs etc (better not to work on the originals but local copies).

You can give it a try and see if it fits your requirement(with some changes may be).

Don't have much inputs on automatic media organization.
 
It may not be exactly what you are looking for but you could host your own nextcloud on your unraid/nas/rpi. It comes with the mobile clients. And sort of works like google photos (but less sophisticated). Last time I tested it, it even had face recognition on the photos. It's a poor man's G-suite replacement.
 
Sorry this will probably be a word soup

Use unraid, SMB and any decent app that supports SMB. (FolderSync for syncing, FX file explorer, Solid explorer, built in SMB support in the File managers on android/ios/windows)

In your unraid server, create a container/vm to run the SMB server and mount your storage in this container.

Create a SMB user for each user that needs access to this machine

Create a shared folder that'll be accessible with rw permissions to every one

Create 1 folder for each user that only the user will have access to.

The downside of this is, People who have access to the server can see data for all users. I have not been able to find a reliable/standard way to do full e2e encrypted setup in this configuration.

I do not recommend apps that store data in their own format(like seafile) because if some thing breaks, you'll have a difficult time recovering data and you'll be dependent on their apps on all platforms.


What kind of backups are you looking for? Is this cold backups or will you be accessing it frequently?

There are services like wasabi that are cheaper than S3 that can be accessed frequently and there are services like AWS s3 glacier which charges you 1$/TB/mo for storage and 92$/TB for retrieval.


Use rclone for syncing files from your unraid container to the cloud service.

Use SMB on windows. You'll be able to use LR and most other apps with the SMB mount like any normal folder.

A small section from my SMB configuration
(fruit stuff is for improving compatibility on apple devices. you can skip that)
[global] workgroup = WORKGROUP server string = Samba Server %v netbios name = nas security = user map to guest = bad user server min protocol = SMB2_10 client max protocol = SMB3 client min protocol = SMB2_10 min protocol = SMB2 protocol = SMB3 encrypt passwords = true dns proxy = no ntlm auth = true log file = /var/log/samba/smb.log max log size = 10000 ea support = yes vfs objects = fruit streams_xattr fruit:metadata = stream fruit:model = MacSamba fruit:posix_rename = yes fruit:veto_appledouble = no fruit:nfs_aces = no fruit:wipe_intentionally_left_blank_rfork = yes fruit:delete_empty_adfiles = yes [ishan] path = /main/ishan valid users = ishan browsable = yes writable = yes guest ok = no read only = no hide unreadable = yes access based share enum = yes [user2] path = /main/user2 valid users = user2 browsable = yes writable = yes guest ok = no read only = no hide unreadable = yes access based share enum = yes [cameras] path = /mnt/cameras valid users = ishan user2 browsable = yes writable = no guest ok = no read only = yes hide unreadable = yes access based share enum = yes [user3] path = /main/user3 valid users = user3 browsable = yes writable = yes guest ok = no read only = no hide unreadable = yes access based share enum = yes
 
ishanjain28 said:
...FolderSync for syncing, ... Solid explorer, built in SMB support in the File managers on android/ios/windows
Click to expand...
I've been using this combo forever and can attest to their smooth and fast operation.

FolderSync will copy Pics + Videos to PC over WiFi (can be configure for mobile data as well) to a device specific folder (separate folder per device)
Then it will move RAW images which are a memory hog
 
lockhrt999 said:
It may not be exactly what you are looking for but you could host your own nextcloud on your unraid/nas/rpi. It comes with the mobile clients. And sort of works like google photos (but less sophisticated). Last time I tested it, it even had face recognition on the photos. It's a poor man's G-suite replacement.
Click to expand...
I'm in search for Google photos replacement for my family, my free google storage if over and don't want to pay for extra space. Nextcloud looks promising.
Does nextcloud work over internet with dynamic ip?
How fast is backup and retrieval?
I don't have a NAS yet, but looking to setup one for plex, private cloud, pi-hole etc.
 
nitin_g3 said:
I'm in search for Google photos replacement for my family, my free google storage if over and don't want to pay for extra space. Nextcloud looks promising.
Does nextcloud work over internet with dynamic ip?
How fast is backup and retrieval?
Click to expand...
It can work over dynamic IP with the help of tailscale but still it's better if you can arrange static IP. For dynamic IP + tailscale solution, mobiles will need to be configured. And tailscale itself isn't very reliable.

Backup retrieval depends upon your device. Do test the nextcloud before making it your permanent arrangement. Its docker images are available.
 
Ok from the replies :

1. @pa1 , Synthing app for syncing all the media from mobile to unraid and then manage from there. Rclone to sync from unraid to cloud.
2. @lockhrt999 , i have had it (Nextcloud) installed and yet to configure it. Will try to work on along with aforementioned apps
3. @ishanjain28 , idea is very basic and SMB is actually pretty fast. But mounting SMB server on unraid.. ? I always have trouble pointing the right folders from and to docker. Howmuch i try , each app within docker behaves like sh!t and have to create multiple similar named folders.. I like your idea of using basic tech. Will see. I still dont know how to configure LR to properly use the folder and create its database , which again i want it on Unraid , so that even if LR is reinstalled , i will have all the database and edits.(I properly have to google on it to learn further i suppose)
4. @n1r0 Will try foldersync..

Overall i have got the picture that i need to sync all them , but how to properly organize them ?

Lets say we three go out for tour , i take pics on my phone and also on my a6400 , when we come back , all phones sync to unraid and uploads pics. Now how to properly go about this sync so , that they are synced and organized properly instead of one folder with 1000`s of pics and folders. Or should i allow the media to reside on one folder without sub directories and allow the app like NEXTCLOUD to organize in its database and show accordingly ?
 
lockhrt999 said:
It can work over dynamic IP with the help of tailscale but still it's better if you can arrange static IP. For dynamic IP + tailscale solution, mobiles will need to be configured. And tailscale itself isn't very reliable.

Backup retrieval depends upon your device. Do test the nextcloud before making it your permanent arrangement. Its docker images are available.
Click to expand...
I'm trying to setup nextcloud on my personal windows 11 laptop using docker. It asks for domain at initial stage, got a cheap domain from ho hostinger as well.

Will try the next steps, this is a new territory for me. Will take some time to learn.
Did you also setup using a domain or some other way?
 
Last edited:
Naveen_Reloaded said:
Lets say we three go out for tour , i take pics on my phone and also on my a6400 , when we come back , all phones sync to unraid and uploads pics. Now how to properly go about this sync so , that they are synced and organized properly instead of one folder with 1000`s of pics and folders. Or should i allow the media to reside on one folder without sub directories and allow the app like NEXTCLOUD to organize in its database and show accordingly ?
Click to expand...
Split that task into 2 parts:
  1. Archive pics from everyone into their own subfolder
  2. Use hardlinks to rename files chronologically, with a suitable suffix, and have them show up in a single album folder
Eg: Person1's pics are IMG_YYYYMMDD_HHMMSS.jpg
Person2's pics are DCSxxxx.jpg

What I do is have the following folder structure for archiving:
YYYY/Trip_name/Person1
YYYY/Trip_name/Person2

This way the original filenames and ownership are available unaltered.

Now, you could create an album folder for that trip which will have all the files hardlinked, but renamed to YYYYMMDD_HHMMSSp1.jpg, YYYYMMDD_HHMMSSp2.mp4, etc using exiftool. This way, to reminisce a trip, just open the album folder and you will see everyone's media shown chronologically.
 
n1r0 said:
Split that task into 2 parts:
  1. Archive pics from everyone into their own subfolder
  2. Use hardlinks to rename files chronologically, with a suitable suffix, and have them show up in a single album folder
Eg: Person1's pics are IMG_YYYYMMDD_HHMMSS.jpg
Person2's pics are DCSxxxx.jpg

What I do is have the following folder structure for archiving:
YYYY/Trip_name/Person1
YYYY/Trip_name/Person2

This way the original filenames and ownership are available unaltered.

Now, you could create an album folder for that trip which will have all the files hardlinked, but renamed to YYYYMMDD_HHMMSSp1.jpg, YYYYMMDD_HHMMSSp2.mp4, etc using exiftool. This way, to reminisce a trip, just open the album folder and you will see everyone's media shown chronologically.
Click to expand...

how to do the second part on unraid ? Hardlinking ?

Your idea is excellent. But i need to find a way to make them share the pics to one folder which should be separate from the auto syncing i am planning to do from thier devices in thier own folders on unraid. I need to find to take the synced files or move the "Tour" files without duplicating in multiple folders. I think you get the point. For me , i dont have issues , i can do it manually.

Should i sync the folders and should i manage the incoming photos from their devices ? or should i ask them to send X number of "tour" pics to Y folder ?


Update : Installed and configured nextcloud on unraid and set my phone (trial) to sync my camera folder to nextcloud. Lets see how it goes. NExtcloud is like google drive. Need to find something which can sort photos like tagging , etc.
 
Never used Unraid, so can't comment. Every OS should support hardlinks on some file systems.

Auto sync will sync all files in their phone's camera folder. So create a folder like Backup/Person1 which only they can access to backup everything via Auto Sync.

If it's more of a select few pics and upload only those, then you don't need something like FolderSync. Solid Explorer can do it. Create YYYY/Trip_name/Person1 folder and have them upload those selected pics.

Using hardlinks is not necessary if the viewer you're using can read the EXIF date taken field and sort on it automatically. Google Photos for eg. does this.

If you need to tag them, you're probably better off with something like Lightroom. It can do the renaming as well.
 
I think my issue should be a universal one which everyone will go through if they are planning to backup the real images , rather than the compressed one from google photos(selecting compressed uploads) or whatsapp images. I wish there was an app which can backup all pics/videos to one folder(structured) from each device / person and upon users preference / choice , share or hardlink the images to the common folder for everyones view and even providing option to work on the photos via mobile app editor or LR.

If not , I should concentrate for now to backup the media from these devices to UNRAID and from there to cloud (cold) and then worry about on how to get everything sorted. In this scenario , i will be able to view all the medias which are going to get uploaded and should put up a public folder for them to view those selected media alone.

Lets see...
 
I just remembered JellyFin works with photos as well and gave it a quick try. Seems to work alright. It lets you add Tags too. Give it a try. This way you can use FolderSync for upload and JF for viewing. You can even set which libraries are visible to which people.

Only hitch I can think of is that you might need Static IP or a VPN to your home network to be able to access it from the internet.

Most people take pics to instantly share on WA/Insta, very few actually archive the originals. If you're good with scripting, you can write one to do what you want specifically.
 
n1r0 said:
I just remembered JellyFin works with photos as well and gave it a quick try. Seems to work alright. It lets you add Tags too. Give it a try. This way you can use FolderSync for upload and JF for viewing. You can even set which libraries are visible to which people.

Only hitch I can think of is that you might need Static IP or a VPN to your home network to be able to access it from the internet.

Most people take pics to instantly share on WA/Insta, very few actually archive the originals. If you're good with scripting, you can write one to do what you want specifically.
Click to expand...
As of now , i have no plans to open up the NAS to internet. But will have to do sooner or later for better access and connectivity.

I saw jellyfin too.

What you say is true regarding pics taken. But nowadays i need to upload most of the pics during such trip which mostly have my cousins also , to google drive and share them. So i am looking into something which will have online portal which they can also view and download. And wish they could also upload to common folder.
 
For organization, my advice is to keep it low-tech initially.

Maintain original media in simple per folder per device and within that keep the same structure as what the device uses.

Think hard about your preferred access patterns and make a note. Then create (hard or soft depending on access) links in another directory per month/year/person/whatever org schema you need. If you are comfortable with some coding/scripting you can make use of exif data to do this with standard linux tools and automate(may not work with whatsapp or any apps that resize and strip metadata).

With links its cheaper to make changes or creating new way of structuring.

If you prefer fancy features or easy access through browser or web(or over lan), look into the hosted media library options that others have suggested.
 
Back
Top