Linux Batch Move Job

L

LaatSahab

Level G
I have setup my Pi as a Seedbox and NAS. It downloads to an external HDD connected through USB. I want to move the completed downloads from the hard drive to another external one which I use as backup. Now instead of issuing multiple mv commands manually and waiting for one to finish before issuing another, how can I batch move multiple folders from the hard drive to multiple locations in another hard drive ? Googled a lot regarding this but couldn't make any headway in this regard.
 
psyph3r said:
Isn't there an option in the torrent client to move completed downloads to another folder ?
Click to expand...
This isn't about moving downloads to another folder on same drive. I'm trying to move completed downloads from the download drive(permanent mount external 2TB HDD) to another external HDD (temporary mounted 6TB). Raspberry Pi forum hints at using ; separated values for multiple mv jobs.
My source organised as below:
  • Downloads
    • Movies
    • Songs
    • Games
My intended destination is as below:
  • Media
    • Movies
      • Bollywood
      • Hollywood
    • Songs
      • Lossy
      • Lossless
 
Last edited:
Ofc you can move to another folder in another drive too, you'll have to label them manually when you add the torrent.

µTorrent Help: Moving Downloads Into Category Folders

Move µTorrent Downloads Into Category Folders Introduction I have recently seen a number of µTorrent users requesting help in moving their downloaded content to different folders, based upon categories, and continue seeding. This guide wil help in achieving this by either having µTorrent do this...
www.techsupportalert.com www.techsupportalert.com
 
As I said in my post above, another drive isn't permanently mounted, It's a backup store for overflow of 2TB one. Anyway the semicolon separation of multiple mv commands did the trick. Bit tedious but gets the job done.
 
First don't temporarily connect another hdd. if it is on a windows machine, use share and mount it using mount.cifs and if it lives on other linux machine, just share it using nfs.
Second, Add rsync commands as cron job between local drive and mounted folder.
rsync -avzh ~/Downloads /mnt/cifsmount
Click to expand...
second method is to chain different commands, that is using $() structures or old style back tics ``
example:
for file in $(ls ~/Downloads); do mv $file /mnt/backupdrive/; done
Click to expand...
for file in `ls ~/Downloads`; do mv $file /mnt/backupdrive; done
Click to expand...
you can get a lot more complex chaining using grep, awk, sed and many other commands.
 
Last edited:
booo said:
First don't temporarily connect another hdd. if it is on a windows machine, use share and mount it using mount.cifs and if it lives on other linux machine, just share it using nfs.
Second, Add rsync commands as cron job between local drive and mounted folder.

second method is to chain different commands, that is using $() structures or old style back tics ``
example:


you can get a lot more complex chaining using grep, awk, sed and many other commands.
Click to expand...
The download drive isn't temporarily connected. It has it's fstab entry and shared through smb.conf
Backup drive is the one that was temporarily mounted.
Will try the rsync method next time around.

Was able to get it done through mv command. Used something like this:-
mv -v <source location 1> <destination location 1>; mv -v <Source location 2> <Destination location 2>

Verbose output was helpful in monitoring. If ever felt a need to check the progress, used "Progress" command utility. While it showed the progress of the ongoing move job, it limited itself to the current file being moved. There should be something that gives a progress bar like of graphical information and feedback at command line level. The "pv" requires piping which couldn't be the case in this situation. As the mv commands executed serially, it went by smoothly as both the devices, source and destination, were connected through USB. Executed the command late in the morning, was finished by evening. Amount of data moved was 1TB. Source was a USB 2.0 device connected to a USB 2.0 interface of Pi, destination was USB 3.0 device connected to USB 3.0 interface of Pi
If there's a way to improve transfer speeds, let it be known.
 
Last edited:
1G ethernet is faster than usb 2, so if possible connect the hdd to another machine and mount it over nfs. if both rip and second machine are connected using ethernet cable then you can get better transfer rates.
another possibility is to connect both hdds using powered usb 3 hub.
 
@booo Tried using Rsync. Works great when moving files among folders on the Pi itself but I'm not able to issue Pull or Push requests from Pi to my Windows machine. I suspect that my syntax is wrong. The Windows machine has SSH enabled but not any kind of Rsync package installed, which I presume would be irrelevant as Pi's Rsync package over SSH is handling it all.
Push syntax->rsync -avzhp /home/pi/Documents/file1 <Username>@<DestinationIP>:C\Users\<Profile name>\Documents\
Alternatively I have tried this also: rsync -avzhp /home/pi/Documents/file1 \\<DestinationIP>\C\Users\<Profile name>\Documents\
Doing this way copies the file that I'm trying to Push to the /home/pi folder but with name which starts with Username@...
I think my syntax is wrong somewhere. Please correct me.
 
okay....
step 1: mount target folder on pi. ex: mount -t cifs -o username=booo //ipaddress/C$/Users/booo/Downloads /mnt/pc_mount
step2: rsync -avzhp /home/pi/Documents /mnt/pc_mount/

remember this is linux and all forward slashes’/‘ and not backslashes’\’

to use cifs you will need to ‘sudo apt install cifs-utils’ before mount command
 
booo said:
okay....
step 1: mount target folder on pi. ex: mount -t cifs -o username=booo //ipaddress/C$/Users/booo/Downloads /mnt/pc_mount
step2: rsync -avzhp /home/pi/Documents /mnt/pc_mount/

remember this is linux and all forward slashes’/‘ and not backslashes’\’

to use cifs you will need to ‘sudo apt install cifs-utils’ before mount command
Click to expand...
Have the cifs utils installed and copying after mounting isn't an issue. As I understand, mounting a drive creates local references for it, akin to any folder in the system's filesystem. I am trying to do it over the network, without mounting.
What I'm hoping to achieve is to have Machine 1(Pi) Push or Pull data to and from Machine 2 (Win 10) over SSH through Rsync only, without any extra steps. If this requires anything more than a SSH server running on Windows then maybe I have got the wrong idea about the functionality of Rsync, which AFAIU, is designed for Syncing and incremental backup of servers across network.

Edit:- Inputs from Pi forum made it clear that in order to achieve what I was hoping to, I need to have Rsync utility on Windows machine too as Rsync is a utility, not a native command of SSH. Without that, I would have to go the mounting/creating network shares route to transfer data across.
So my next question then is this; which way would be better to go, Cygwin on Windows or the Linux distribution for Windows from Windows store? In case of latter, which one to get?
 
Last edited:
you have to mount remote file systems to use them that's how linux works. now, cifs uses cifs ports(udp 137 and 138 and tcp 139 and 445) if you don't want to use those ports you can use ssh-fs which uses ssh(port22) but then you will have to mount too. think of mounting as mapping a network drive in windows.

once you mount win10 share over cifs or sshfs, you can run rsync in either directions, just switch the folder names in the argument and run it on pi. i.e., rsync -azvhp /mnt/win10 /pi/backup (win10 to pi)

if you want basic linux utils on windows, you could try installing gitbash for windows from https://git-scm.com/download/win it installs a portable version of mingw. but if you do the above method, you don't need anything on windows.
 
Back
Top