salfter/encode_farm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
Scott Alfter
2025-10-02 08:35:11 -07:00
commit d6ad37e225
5 changed files with 204 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
*~

49
README.md Normal file
View File

@@ -0,0 +1,49 @@
Encode Farm
===========
This script takes two files as input to dispatch ffmpeg encoding jobs to
multiple hosts.
The nodes file is a list of hosts to be used. Each line describes a host and
is a space-delimited record with the hostname and the path (as that host sees
it) to your video collection.
The queue file is a list of encoding jobs. Each line describes a job and is
a tab-delimited record as follows:
* pathname to the source file, relative to the paths in the nodes file
* video track selection
* audio track selection (with an optional bitrate override, separated by
a space)
* subtitle track selection (may be "f" to pull subtitles from an .en.srt
file, or -1 to not include subtitles)
* optional parameters (cropping, scaling, audio downmixing, etc.)
Jobs will be dispatched to hosts not already running an ffmpeg process.
If all hosts are busy and jobs remain in the queue, the script will wait
for an available host. As jobs are completed, their entries in the queue
are commented out with "#" and the source files are moved into .hidden
directories that are to be deleted manually once the new files have been
verified for correctness. Jobs may be appended to the queue at any time.
Output is H.265-encoded video with a constant rate factor of 24 and AAC-
encoded audio at 192 kbps (a good rate for surround sound, but you'll
probably want to override it to a lower rate for stereo or mono). If
an external subtitle file is used, it is assumed to be in English and the
language tags are set accordingly; otherwise, whatever language tag is set
in the source file will be copied over. Output files will carry the .m4v
extension.
Jobs are dispatched over SSH, whether to the host running this script
or to others. You must have your login process set up to not require
a password or passphrase; this could be done with either a key with no
passphrase or a key with a passphrase loaded into an SSH agent.
The libfdk_aac codec is used for audio encoding. Your ffmpeg binary
must be compiled with it. On Gentoo Linux, build media-video/ffmpeg with
the fdk USE flag. On Arch Linux, ffmpeg-libfdk_aac is available from
AUR. Elsewhere, you'll most likely need to build ffmpeg from source
manually. There is a script available at
[https://github.com/markus-perl/ffmpeg-build-script](https://github.com/markus-perl/ffmpeg-build-script)
that can help with this.

150
encode.sh Executable file
View File

@@ -0,0 +1,150 @@
#!/usr/bin/env bash
# This script takes two files as input to dispatch ffmpeg encoding jobs to
# multiple hosts.
#
# The nodes file is a list of hosts to be used. Each line describes a host and
# is a space-delimited record with the hostname and the path (as that host sees
# it) to your video collection.
#
# The queue file is a list of encoding jobs. Each line describes a job and is
# a tab-delimited record as follows:
#
# * pathname to the source file, relative to the paths in the nodes file
# * video track selection
# * audio track selection (with an optional bitrate override, separated by
# a space)
# * subtitle track selection (may be "f" to pull subtitles from an .en.srt
# file, or -1 to not include subtitles)
# * optional parameters (cropping, scaling, audio downmixing, etc.)
#
# Jobs will be dispatched to hosts not already running an ffmpeg process.
# If all hosts are busy and jobs remain in the queue, the script will wait
# for an available host. As jobs are completed, their entries in the queue
# are commented out with "#" and the source files are moved into .hidden
# directories that are to be deleted manually once the new files have been
# verified for correctness. Jobs may be appended to the queue at any time.
#
# Output is H.265-encoded video with a constant rate factor of 24 and AAC-
# encoded audio at 192 kbps (a good rate for surround sound, but you'll
# probably want to override it to a lower rate for stereo or mono). If
# an external subtitle file is used, it is assumed to be in English and the
# language tags are set accordingly; otherwise, whatever language tag is set
# in the source file will be copied over. Output files will carry the .m4v
# extension.
#
# Jobs are dispatched over SSH, whether to the host running this script
# or to others. You must have your login process set up to not require
# a password or passphrase; this could be done with either a key with no
# passphrase or a key with a passphrase loaded into an SSH agent.
#
# The libfdk_aac codec is used for audio encoding. Your ffmpeg binary
# must be compiled with it. On Gentoo Linux, build media-video/ffmpeg with
# the fdk USE flag. On Arch Linux, ffmpeg-libfdk_aac is available from
# AUR. Elsewhere, you'll most likely need to build ffmpeg from source
# manually. There is a script available at
# https://github.com/markus-perl/ffmpeg-build-script that can help with
# this.
while true
do
# get first job not completed
job="$(grep -v "^#\|^>" queue | head -1)"
if [ "$job" == "" ]
then
exit
fi
# split up arguments
eval "$(echo "$job" | sed "s/\"/\\\"/g;s/\(.*\)\t\(.*\)\t\(.*\)\t\(.*\)\t\(.*\)/src=\"\1\"; vs=\"\2\"; as=\"\3\"; ss=\"\4\"; opts=\"\5\"/")"
if [ "$(echo "$as" | grep " ")" != "" ]
then
eval "$(echo "$as" | sed "s/\(.*\) \(.*\)/as=\1; abr=\2/")"
else
abr=192
fi
dest="${src%.*}.m4v"
if [ "$(echo $vs | grep c)" == "" ]
then
vs="0:$vs"
vc="libx265 -crf 24"
else
vs="0:$(echo $vs | sed "s/c//")"
vc="copy"
fi
if [ "$(echo $as | grep c)" == "" ]
then
as="0:$as"
ac="libfdk_aac -b:a $abr"
else
as="0:$(echo $as | sed "s/c//")"
ac="copy"
fi
if [ "$ss" == "f" ]
then
src_sub="${src%.*}.en.srt"
ss="1:0"
else
src_sub=""
ss="0:$ss"
fi
# build command
cmd="ffmpeg -y -i \"$src\" "
if [ "$src_sub" != "" ]
then
cmd="$cmd -i \"$src_sub\" "
fi
cmd="$cmd -map $vs -map $as "
if [ "$ss" != "0:-1" ]
then
cmd="$cmd -map $ss "
else
cmd="$cmd -sn "
fi
cmd="$cmd $opts -pix_fmt yuv420p10le -c:v $vc -c:a $ac "
if [ "$ss" != "0:-1" ]
then
if [ "$src_sub" != "" ]
then
cmd="$cmd -c:s mov_text -metadata:s:s language=eng "
else
cmd="$cmd -c:s mov_text "
fi
fi
cmd="$cmd -metadata title= -f mp4 \"$dest\""
# dispatch job to idle host
started=0
while [ $started -eq 0 ]
do
for node in `sed "s/ .*//" nodes`
do
homedir=`grep $node nodes | sed "s/.* //"`
if [ "$(ssh $node ps auxw \| grep ffmpeg \| grep -v grep)" == "" ]
then
hidden_dir="$(dirname "$src")/.hidden"
echo "(cd $homedir && $cmd && mkdir -p \"$hidden_dir\" && mv \"$src\" \"$hidden_dir\")" | nohup ssh $node 2>&1 >/dev/null &
echo "$node $src"
started=1
sleep 5
break
fi
done
if [ $started -eq 0 ]
then
sleep 60
fi
done
# mark job as started
job_sed="$(echo "$job" | tr -d "\n" | sed "s/\//\\\\\//g;s/\"/\\\\\"/g;s/\\\'/\\\\\'/g;s/\\\$/\\\\\$/g")"
sed -i "s/$job_sed/\#$job_sed/" queue
done

2
nodes Normal file
View File

@@ -0,0 +1,2 @@
janeway /mnt/media/video
server /mnt/storage

2
queue Normal file
View File

@@ -0,0 +1,2 @@
# vid aud sub
#src 0 1 192 2 opt_params