Mass file actions with bash
Bash yields its secrets grudgingly. For newbies, the man page -- at 4500 lines -- is terrifying to say the least, and not much help unless you already know what you are looking for. I thought I'd write up a common bash pattern which I learned a few years ago, and have found very useful.
The pattern is to doing some kind of action -- consisting of a command or a sequence of commands -- on a large group of files.
The way I solved this in the early years was to do get the list of files into vim, and then use vim's record/repeat functionality to transform each filename into the command(s) I needed to perform. That worked, but was often awkward and time-consuming. Now I do it all on the command line using bash.
For an example, I recently wanted to transcode a bunch of videos (.avi files) to reduce the file sizes. The command to transcode one of these videos was:
transcode -i video1.avi -o video1.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
To carry out this operation on a whole bunch of files, I use a bash for-loop and wildcards (globbing) to match the files to operate on. First I test to see that I'm picking exactly the files I want:
$ for i in *.avi; do echo $i; done video1.avi video2.avi video3.avi video4.avi video5.avi video6.avi
Now I decide how I will name the output files. In the example above, the original file was video1.avi and the corresponding output is video1.lowquality.avi. In the following command, we do a string subsitution on the variable i to produce the output filename from each input filename (for reference, look in the bash manpage under "parameter expansion")
$ for i in *.avi; do echo $i ${i/.avi/.lowquality.avi}; done
video1.avi video1.lowquality.avi
video2.avi video2.lowquality.avi
video3.avi video3.lowquality.avi
video4.avi video4.lowquality.avi
video5.avi video5.lowquality.avi
video6.avi video6.lowquality.avi
Ok, looks good.
Now we can put in the full command. Note that we still use echo so that the command doesn't actually get executed -- it's always safer to first verify that the commands are correct:
$ for i in *.avi; do echo transcode -i $i -o ${i/.avi/.lowquality.avi} -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4; done
transcode -i video2.avi -o video2.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
transcode -i video3.avi -o video3.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
transcode -i video4.avi -o video4.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
transcode -i video5.avi -o video5.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
transcode -i video6.avi -o video6.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
Note that if you need to perform multiple commands on each files, you can simply add more commands, separated by semicolons. Say we want to also copy each downgraded video into a different directory:
$ for i in *.avi; do out=${i/.avi/.lowquality.avi} ; echo transcode -i $i -o $out -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4; echo cp $out /somewhereelse; done
transcode -i video1.avi -o video1.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
cp video1.lowquality.avi /somewhereelse
transcode -i video2.avi -o video2.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
cp video2.lowquality.avi /somewhereelse
transcode -i video3.avi -o video3.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
cp video3.lowquality.avi /somewhereelse
transcode -i video4.avi -o video4.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
cp video4.lowquality.avi /somewhereelse
transcode -i video5.avi -o video5.lowquality.avi -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4
cp video5.lowquality.avi /somewhereelse
Here I've used the variable out so that I didn't have to write the pattern expansion twice.
Now, finally, we are ready to actually execute the commands. We could remove the 'echo' words from the string, but there might be several of them. The easiest thing to do is to pipe the commands into a subshell, by adding "|sh" onto the end of the line:
$ for i in *.avi; do out=${i/.avi/.lowquality.avi} ; echo transcode -i $i -o $out -R 0 -w 735 -y divx5 -N 0x50 -F mpeg4; echo cp $out /somewhereelse; done |sh
<output from transcode>
...
...
One of the biggest advantages over using vim to write these commands (as I used to do) is that I can now paste this code into a script for future use (if I anticipate needing it often).
Posted by Jason Hildebrand <jason@opensky.ca> Saturday Jan 15, 2005 at 0:25 PM