Encode Farm
This script takes two files as input to dispatch ffmpeg encoding jobs to multiple hosts.
The nodes file is a list of hosts to be used. Each line describes a host and is a space-delimited record with the hostname and the path (as that host sees it) to your video collection.
The queue file is a list of encoding jobs. Each line describes a job and is a tab-delimited record as follows:
- pathname to the source file, relative to the paths in the nodes file
- video track selection
- audio track selection (with an optional bitrate override, separated by a space)
- subtitle track selection (may be "f" to pull subtitles from an .en.srt file, or -1 to not include subtitles)
- optional parameters (cropping, scaling, audio downmixing, etc.)
Track selections are zero-indexed; use something like ffprobe to see what tracks are available. Adding "c" after the video or audio track will cause the selected track to be copied as-is to the output file (useful if the source tracks are already using H.265 and/or AAC at the desired bitrates).
Jobs will be dispatched to hosts not already running an ffmpeg process. If all hosts are busy and jobs remain in the queue, the script will wait for an available host. As jobs are completed, their entries in the queue are commented out with "#" and the source files are moved into .hidden directories that are to be deleted manually once the new files have been verified for correctness. Jobs may be appended to the queue at any time.
Output is H.265-encoded video with a constant rate factor of 24 and AAC-encoded audio at 192 kbps (a good rate for surround sound, but you'll probably want to override it to a lower rate for stereo or mono). If an external subtitle file is used, it is assumed to be in English and the language tags are set accordingly; otherwise, whatever language tag is set in the source file will be copied over. Output files will carry the .m4v extension.
Jobs are dispatched over SSH into Screen sessions, whether to the host running this script or to others. You must have your login process set up to not require a password or passphrase; this could be done with either a key with no passphrase or a key with a passphrase loaded into an SSH agent. Screen allows the jobs to be monitored after they are started.
The libfdk_aac codec is used for audio encoding. Your ffmpeg binary
must be compiled with it. On Gentoo Linux, build media-video/ffmpeg with
the fdk USE flag. On Arch Linux, ffmpeg-libfdk_aac is available from
AUR. Elsewhere, you'll most likely need to build ffmpeg from source
manually. There is a script available
here that can help
with this.