initial commit

2015-01-30 10:45:30 -08:00
commit 49c61eb985
15 changed files with 645 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -0,0 +1,9 @@
+*~
+*.iso
+*.bz2
+tmp
+contents
+filelist
+preplist
+settings.sh
+*.sums
--- a/19
+++ b/19
@@ -0,0 +1,19 @@
+Copyright (c) 2011-2015 Scott Alfter
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
--- a/132
+++ b/132
@@ -0,0 +1,132 @@
+About
+=====
+
+This is a set of scripts I've developed over the past few years to archive
+the contents of my media server to BD-R.  You configure it to monitor
+selected directories within a containing directory.  As files are added or
+modified, they are flagged to be written (or rewritten) to the archive.  As
+files are deleted, they're flagged to not be restored (though if they've
+already archived, they'll remain archived...they just won't be restored). 
+It's already saved my butt once when the RAIDed drives in a NAS box started
+crapping out on me; it might be useful to others.
+
+Archiving Strategy
+==================
+
+As a disc image is built, the largest file that needs to be archived and
+that will fit the remaining space is added.  This process repeats until
+either no space is left or no files remain that will fit in the remaining
+space.  Many of my archive discs, as a result, are dominated by a few large
+files (usually movies or TV shows), with a few small files (songs, photos,
+or metadata files) filling in the remaining space.  It's usually when
+there's only one disc's worth of data remaining that the rest of the small
+files end up getting written in one shot.
+
+Files are written to disc as-is.  You can pop an archive disc into a
+compatible drive and read out the contents.  One of my Blu-ray players will
+even play video files straight off of my archive discs, as the files are
+already in a format it understands.
+
+Caveat
+======
+
+Files are written to a UDF 2.5 image.  Large (>4GB) files are supported. 
+What isn't supported are certain characters in filenames.  I think "$" is
+the most troublesome, but there might be one or two others.  Keep that in
+mind when you're archiving files.
+
+Requirements
+============
+
+* a MySQL server
+* cdrtools (or a compatible replacement; provides mkisofs to create disc
+  images)
+* dvd+rw-tools (provides growisofs to burn disc images)
+* dvdisaster (augments disc images with error recovery information)
+
+There might be other stuff I'm forgetting, but most of the rest of the
+scripts use fairly standard tools (tar, sed, awk, etc.)
+
+Compatibility
+=============
+
+These scripts were developed and tested and are used on Linux.  They should
+work elsewhere (Mac OS X, *BSD, Cygwin, etc.), but some OS-specific tweaks
+may be necessary.
+
+Installation
+============
+
+Create an empty database on your MySQL server, and initialize it with the
+contents of backup_index-schema.sql.  
+
+Copy settings-example.sh to settings.sh.  Customize as needed.
+
+Copy settings-example.sql to settings.sql.  Customize as needed and add its
+contents to the database.
+
+Usage
+=====
+
+The most frequently-used scripts are invoked as follows:
+
+update-index.sh
+
+  Check the configured directories for added/deleted/changed files and
+  update the database.
+
+prepare-disc.sh -br
+
+  Select files for backup and create an ISO image.  The options shown in the 
+  example create a single-layer BD-R image with 20% of the available space
+  reserved for error recovery.  Invoke with -h to see the available options.
+  You can use CD-R, DVD-R, or BD-R for archiving.  Single- and dual-layer 
+  DVD-R and BD-R are supported.
+
+  A copy of the database is included on every disc; the database is needed
+  for file recovery, as it tells us which files are on which discs.
+  
+burn-disc.sh backup_1.iso
+
+  Burn an image to disc, and then read it back in to check that it was
+  burned without error.
+
+progress.sh
+
+  Estimate the number of discs needed to complete archiving.  (It's hard-
+  coded to assume you're backing up to single-layer BD-R with 20% error
+  recovery; you'd want to adjust the constant 20020250624 (on the last line) 
+  to report accurate results for other media or other levels of redundancy.
+
+Some less-frequently-used scripts:
+
+backup-schema.sh
+
+  Back up the database schema to a local file.  This is mostly for
+  development use, if the schema changes.
+
+backup-database.sh
+
+  Back up the database schema and contents to a file, and use that to
+  replicate the database on another (presumably remote) MySQL server. 
+  Before I started writing the database to each disc, this was how I made 
+  sure I had an offsite backup of the database.
+
+rebuild.sh
+restore.sh
+wait-on-drive.sh
+
+  These are mainly of use when restoring files from the archive.  You could
+  just copy all files from all discs in sequence, but that would end up
+  restoring files you had deleted.  The purpose of these scripts is to only
+  restore the files the database knows about, which will probably be a
+  little bit faster as well.
+
+Disclaimer
+==========
+
+So far, these scripts have worked pretty well for me.  That said, they're
+worth what you paid for them. :-) The file-restore scripts have had the
+least amount of testing, as I've only needed to do one full recovery so far. 
+Your files are written to disc as-is, so to recover a single file, you could
+just look up which disc holds it and pop it in.
--- a/backup-database.sh
+++ b/backup-database.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+# NOTICE: the backup destination is hard-coded!  Replacing my email address
+# (which also happens to be the username and hostname for the offsite VPS I
+# use for backup) with the appropriate ssh credentials for your use would
+# fix this.
+#
+# This is mainly a holdover from when I wasn't writing a copy of the
+# database to every disc.  Now that I am, this script is somewhat less
+# necessary.
+
+source settings.sh
+
+mysqldump -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD -R $MYSQL_DB | bzip2 -9 >backup_index.sql.bz2
+scp backup_index.sql.bz2 scott@alfter.us:
+ssh scott@alfter.us bzcat backup_index.sql.bz2 \| mysql --password=$MYSQL_PASSWD $MYSQL_DB
+
+tar cjf backup_index_scripts.tar.bz2 *.sh
+scp backup_index_scripts.tar.bz2 scott@alfter.us:
--- a/backup-schema.sh
+++ b/backup-schema.sh
@@ -0,0 +1,5 @@
+#!/bin/bash
+source settings.sh
+
+mysqldump -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD -dR $MYSQL_DB >backup_index-schema.sql
+echo "insert into backup_index (discnum) values (0);" >>backup_index-schema.sql
--- a/backup_index-schema.sql
+++ b/backup_index-schema.sql
@@ -0,0 +1,122 @@
+-- MySQL dump 10.13  Distrib 5.5.35, for debian-linux-gnu (x86_64)
+--
+-- Host: files    Database: backup_index
+-- ------------------------------------------------------
+-- Server version	5.5.35-0ubuntu0.12.04.2
+
+/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
+/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
+/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
+/*!40101 SET NAMES utf8 */;
+/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
+/*!40103 SET TIME_ZONE='+00:00' */;
+/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
+/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
+/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
+/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
+
+--
+-- Table structure for table `backup_index`
+--
+
+DROP TABLE IF EXISTS `backup_index`;
+/*!40101 SET @saved_cs_client     = @@character_set_client */;
+/*!40101 SET character_set_client = utf8 */;
+CREATE TABLE `backup_index` (
+  `id` int(11) NOT NULL AUTO_INCREMENT,
+  `filename` varchar(300) DEFAULT NULL,
+  `filesize` bigint(20) NOT NULL,
+  `filedate` datetime NOT NULL,
+  `discnum` int(11) DEFAULT NULL,
+  `md5` char(32) CHARACTER SET latin1 DEFAULT NULL,
+  PRIMARY KEY (`id`),
+  UNIQUE KEY `filename` (`filename`)
+) ENGINE=MyISAM AUTO_INCREMENT=212282 DEFAULT CHARSET=utf8;
+/*!40101 SET character_set_client = @saved_cs_client */;
+
+--
+-- Table structure for table `excluded_dirs`
+--
+
+DROP TABLE IF EXISTS `excluded_dirs`;
+/*!40101 SET @saved_cs_client     = @@character_set_client */;
+/*!40101 SET character_set_client = utf8 */;
+CREATE TABLE `excluded_dirs` (
+  `name` varchar(1000) NOT NULL,
+  PRIMARY KEY (`name`)
+) ENGINE=MyISAM DEFAULT CHARSET=latin1;
+/*!40101 SET character_set_client = @saved_cs_client */;
+
+--
+-- Table structure for table `settings`
+--
+
+DROP TABLE IF EXISTS `settings`;
+/*!40101 SET @saved_cs_client     = @@character_set_client */;
+/*!40101 SET character_set_client = utf8 */;
+CREATE TABLE `settings` (
+  `name` varchar(20) NOT NULL,
+  `value` text NOT NULL,
+  PRIMARY KEY (`name`)
+) ENGINE=MyISAM DEFAULT CHARSET=latin1;
+/*!40101 SET character_set_client = @saved_cs_client */;
+
+--
+-- Table structure for table `tmp`
+--
+
+DROP TABLE IF EXISTS `tmp`;
+/*!40101 SET @saved_cs_client     = @@character_set_client */;
+/*!40101 SET character_set_client = utf8 */;
+CREATE TABLE `tmp` (
+  `filename` varchar(300) DEFAULT NULL,
+  `discnum` int(11) DEFAULT NULL
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+/*!40101 SET character_set_client = @saved_cs_client */;
+
+--
+-- Dumping routines for database 'backup_index'
+--
+/*!50003 DROP PROCEDURE IF EXISTS `pick_files` */;
+/*!50003 SET @saved_cs_client      = @@character_set_client */ ;
+/*!50003 SET @saved_cs_results     = @@character_set_results */ ;
+/*!50003 SET @saved_col_connection = @@collation_connection */ ;
+/*!50003 SET character_set_client  = utf8 */ ;
+/*!50003 SET character_set_results = utf8 */ ;
+/*!50003 SET collation_connection  = utf8_general_ci */ ;
+/*!50003 SET @saved_sql_mode       = @@sql_mode */ ;
+/*!50003 SET sql_mode              = '' */ ;
+DELIMITER ;;
+CREATE DEFINER=`salfter`@`%` PROCEDURE `pick_files`(size bigint)
+BEGIN
+    declare largest int;
+    declare largest_size bigint;
+    l: loop
+        set largest=null;
+        select id, filesize into largest, largest_size from backup_index where discnum is null and filesize<=size*2048 order by filesize desc limit 1;
+        if largest is null
+        then
+            leave l;
+        end if;
+        update backup_index set discnum=-1 where id=largest;
+        set size=size-truncate((largest_size-1)/2048+1, 0);
+    end loop l;
+END ;;
+DELIMITER ;
+/*!50003 SET sql_mode              = @saved_sql_mode */ ;
+/*!50003 SET character_set_client  = @saved_cs_client */ ;
+/*!50003 SET character_set_results = @saved_cs_results */ ;
+/*!50003 SET collation_connection  = @saved_col_connection */ ;
+/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
+
+/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
+/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
+/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
+/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
+/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
+/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
+/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
+
+-- Dump completed on 2014-03-06 21:59:56
+
+insert into backup_index (discnum) values (0);
--- a/burn-disc.sh
+++ b/burn-disc.sh
@@ -0,0 +1,56 @@
+#!/bin/sh
+
+device=/dev/sr0
+
+help()
+{
+cat <<EOF >&2
+Usage: $0 [options] image.iso
+
+options: -d|--dao: use DAO mode
+         -s|--speed n: set burning speed
+         -n|--no-spares: don't create BD-R/BD-RE spare block area
+         -r|--device: set device to use (default: /dev/sr0)
+EOF
+}
+
+OPTS=`getopt -o ds:nhr: --long dao,speed,no-spares,help,device -- "$@"`
+eval set -- "$OPTS"
+while true; do
+  case "$1" in
+    -d|--dao) burnopts="$burnopts -use-the-force-luke=dao"; shift;;
+    -s|--speed) burnopts="$burnopts -speed=$2"; shift 2;;
+    -n|--no-spares) burnopts="$burnopts -use-the-force-luke=spares:none"; shift;;
+    -r|--device) device=$2; shift 2;;
+    -h|--help) help; exit 1;;
+    --) shift; break;;
+    *) echo "Internal error" >&2; exit 1;;
+  esac
+done
+
+if [ "$1" == "" ]
+then
+  help
+  exit 1;
+fi
+
+growisofs $burnopts -Z $device="$1" 
+eject $device
+sleep 4
+eject -t $device
+#sleep 20
+
+err=-1
+while [ $err != 0 ]
+do
+  mount $device 2>&1 >/dev/null
+  err=$?
+  sleep 2
+done
+umount $device 2>&1 >/dev/null
+
+sleep 3
+
+dvdisaster -r -d $device -i "${1%.iso}_r.iso"
+dvdisaster -t -i "${1%.iso}_r.iso"
+rm "${1%.iso}_r.iso"
--- a/prepare-disc.sh
+++ b/prepare-disc.sh
@@ -0,0 +1,142 @@
+#!/bin/bash
+
+source settings.sh
+
+echo -n | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB >/dev/null
+if [ $? -ne 0 ]
+then
+  echo "unable to connect to database" >&2
+  exit 1
+fi
+
+help()
+{
+cat <<EOF >&2
+options: -b|--bdr: prepare for BD-R media
+         -d|--dvdr: prepare for DVD-R media
+         -c|--cdr: prepare for CD-R media
+         -l|--dual-layer: prepare for dual-layer media (DVD-R/BD-R only)
+         -r|--reserve: reserve space for dvdisaster (default: 20%)
+	 -e|--reserve-percent: change reserved percentage for dvdisaster
+         -p|--progress filename: write progress messages (default: /dev/null)
+         -h|--help: this message
+EOF
+}
+
+progress=/dev/null
+
+OPTS=`getopt -o bcdlrhp:e: --long bdr,cdr,dvdr,dual-layer,reserve,help,progress,reserve-percent -- "$@"`
+eval set -- "$OPTS"
+while true; do
+  case "$1" in
+    -b|--bdr) media=bdr; shift;;
+    -c|--cdr) media=cdr ; shift;;
+    -d|--dvdr) media=dvdr; shift;;
+    -l|--dual-layer) dual=1; shift;;
+    -r|--reserve) reserve=1; shift;;
+    -e|--reserve-percent) rsvpct="$2"; shift 2;;
+    -h|--help) help; exit 1;;
+    -p|--progress) progress="$2"; shift 2;;
+    --) shift; break;;
+    *) echo "Internal error" >&2; exit 1;;
+  esac
+done
+
+if [ "$media" == "" ]
+then
+  help; exit 1
+fi
+
+case "$media" in
+  "bdr")  if [ "$dual" == "1" ]; then cap=24438784; else cap=12219392; fi;;
+  "dvdr") if [ "$dual" == "1" ]; then cap=4171712; else cap=2295104; fi;;
+  "cdr")  cap=360000;;
+esac
+
+if [ "$rsvpct" == "" ]
+then
+  rsvpct=20
+fi
+
+if [ "$reserve" == "1" ]
+then
+  #cap=`echo $cap/5\*4 | bc`
+  cap=`echo $cap $rsvpct | awk '{printf("%.0f\n", (1-$2/100)*$1)}'`
+fi
+
+# 29 Aug 14: estimate space needed for database backup, and deduct it
+
+b1=`mysqldump -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD -R $MYSQL_DB | xz -z9  | wc -c`
+b2=`tar cJf - *.sh | wc -c`
+total=`echo \( $b1 / 2048 + 1 \) + \( $b2 / 2048 + 1 \) | bc`
+cap=`echo $cap - $total | bc`
+
+#echo === Selecting files to copy === && \
+cat <<EOF | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB | grep -Pv "^filesize\tfilename\tfiledate$" >contents
+update backup_index set discnum=null where discnum=-1;
+call pick_files($cap);
+select filesize, filename, filedate from backup_index where discnum=-1;
+EOF
+
+discnum=`echo "select max(discnum)+1 from backup_index;" | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB | tail -n 1`
+echo $discnum >>contents
+
+cat <<EOF | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB | grep -v "^filename$" >filelist
+select filename from backup_index where discnum=-1;
+EOF
+#echo "select filename from backup_index where discnum=-1;" | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB | grep -v "^filename$" >filelist
+(cat filelist; echo $discnum) >preplist
+
+
+#echo === Linking files === && \
+if [ -e tmp ]
+then
+  rm -r tmp
+fi
+mkdir tmp && \
+ln -s "`readlink -f contents`" tmp/ && \
+#sed "s/\(.*\)/d=\`dirname \"\1\"\`\nmkdir -p tmp\/\"\$d\"\nln -s $SEDROOTDIR\/\"\1\" tmp\/\"\1\"/" filelist | bash &&
+sed "s/\`/\\\\\`/g;s/\(.*\)/d=\$\(dirname \"\1\"\)\nmkdir -p tmp\/\"\$d\"\nln -s $SEDROOTDIR\/\"\1\" tmp\/\"\1\"/" filelist | bash &&
+
+# 29 Aug 14: include database and scripts in disc image
+
+cat <<EOF | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB
+update backup_index set discnum=$discnum where discnum=-1;
+EOF
+
+mysqldump -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD -R $MYSQL_DB | xz -z9 >tmp/backup_index.sql.xz
+tar cJf tmp/backup_index_scripts.tar.xz *.sh
+
+cat <<EOF | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB
+update backup_index set discnum=-1 where discnum=$discnum;
+EOF
+
+#echo === Building backup_$discnum.iso ===
+#mkisofs -iso-level 3 -f -JR -udf -V backup_$discnum -o backup_$discnum.iso tmp &>>"$progress" 
+mkisofs -allow-limited-size -iso-level 3 -f -r -udf -V backup_$discnum -o backup_$discnum.iso tmp &>>"$progress" 
+if [ $? -ne 0 ]
+then
+  echo "error in mkisofs"
+  exit $?
+fi
+rm preplist filelist contents && \
+rm -r tmp && mkdir tmp && \
+
+#echo === Gathering MD5 sums === && \
+sudo mount -o loop,ro backup_$discnum.iso tmp && \
+(cd tmp; find . -type f | sed "s/^\.\///" | grep -v ^contents\$ | sed "s/^/\"/;s/\$/\"/" | xargs md5sum) >backup_$discnum.sums && \
+sudo umount tmp && \
+rmdir tmp && \
+
+#echo === Updating index ===
+sed "s/'/''/g;s/\([0-9a-f]*\)  \(.*\)/update backup_index set discnum=$discnum, md5='\1' where filename='\2';/" backup_$discnum.sums | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB
+
+if [ "$reserve" == "1" ]
+then
+#  echo === Augmenting image with recovery data ===
+  dvdisaster -mRS02 -ci backup_$discnum.iso &>>"$progress"
+fi
+
+#echo === Done preparing backup_$discnum.iso. ===
+echo "backup_$discnum.iso"
+rm backup_$discnum.sums
--- a/progress.sh
+++ b/progress.sh
@@ -0,0 +1,7 @@
+#!/bin/bash
+#awk 'BEGIN {FS="\t"} $4 != "" {y+=$2; next} {n+=$2} END {printf("%d%% complete, %d GB remaining\n",100.0*y/(n+y), n/10740563968);}' backup-index
+source settings.sh
+total=`echo "select sum(filesize) from backup_index;" | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB | tail -n 1`
+backed_up=`echo "select sum(filesize) from backup_index where discnum is not null;" | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB | tail -n 1`
+echo `echo "scale=1; 100*$backed_up/$total" | bc`"% complete, "`echo "scale=1; ($total-$backed_up)/1073741824" | bc `" GB remaining"
+echo "(about "`echo "($total-$backed_up)/20020250624+1" | bc`" single-layer BD-R(s) with dvdisaster ECC)"
--- a/rebuild.sh
+++ b/rebuild.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+source settings.sh
+
+mount /mnt/cdrom && cp /mnt/cdrom/contents . && umount /mnt/cdrom && chmod u+w contents && \
+discnum=`tail -n 1 contents` && \
+awk 'BEGIN {FS="\t"} {print $2}' contents | grep -v ^\$ >filelist.$discnum && \
+sed "s/'/''/g;s/^/select filename,discnum from backup_index where filename='/;s/\$/';/" filelist.$discnum | mysql --default-character-set=utf8 -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB | grep NULL | sed "s/\tNULL\$//" >missing.$discnum && \
+mount /mnt/cdrom && \
+sed "s/\(.*\)/echo Summing \\\"\1\\\"...; md5sum \/mnt\/cdrom\/\\\"\1\\\" >>sums.$discnum/" missing.$discnum | bash && \
+umount /mnt/cdrom && \
+if [ -e sums.$discnum ]
+then
+  sed "s/'/''/g;s/\([0-9a-f]*\)  \/mnt\/cdrom\/\(.*\)/update backup_index set md5='\1', discnum=$discnum where filename='\2';/" sums.$discnum | mysql --default-character-set=utf8 -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB
+  rm sums.$discnum
+fi
+rm contents filelist.$discnum missing.$discnum
+
--- a/restore.sh
+++ b/restore.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+start=$1
+if [ "$start" == "" ]
+then
+  start=0;
+fi
+discs=`echo select distinct discnum from backup_index where filename like \'video/%\' and discnum is not null order by discnum\; | mysql -h mythserver -u salfter --password=taifacs backup_index | grep -v discnum`
+for i in $discs
+do
+  if [ $i -ge $start ]
+  then
+    eject /dev/sr0
+    echo Insert disc $i
+    for j in `seq 1 10`
+    do
+    	beep -f 800 -l 100
+    	beep -f 750 -l 100
+    done
+    read
+    eject -t /dev/sr0
+    sleep 40
+    sudo mount /mnt/cdrom 2>/dev/null
+    while [ $? != 0 ]
+    do
+      sleep 2
+      sudo mount /mnt/cdrom 2>/dev/null
+    done  
+    echo select filename from backup_index where discnum=$i and filename like \'video/%\'\; | mysql -h mythserver -u salfter --password=taifacs backup_index | grep -v discnum | grep -v filename | sed "s/\(.*\)/if [ \! -e \"\/mnt\/files\/\1\" ]; then cp -v \"\/mnt\/cdrom\/\1\" \"\/mnt\/files\/\1\"; fi/" | bash
+  fi
+done
+
+
--- a/settings-example.sh
+++ b/settings-example.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+export MYSQL_HOST="MYSQL_SERVER_HOSTNAME"
+export MYSQL_USER="MYSQL_USERNAME"
+export MYSQL_PASSWD="MYSQL_PASSWORD"
+export MYSQL_DB="backup_index"
+eval `echo "select name, value from settings;" | mysql -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB | grep -Pv "name\tvalue" | sed "s/\t/=\"/;s/\$/\"/"`
+PATHS=""
+for i in $DIRS
+do
+  PATHS="$PATHS $ROOTDIR/$i"
+done
+SEDROOTDIR="`echo $ROOTDIR | sed "s/\//\\\\\\\\\//g"`"
--- a/settings-example.sql
+++ b/settings-example.sql
@@ -0,0 +1,37 @@
+-- MySQL dump 10.13  Distrib 5.5.41, for debian-linux-gnu (x86_64)
+--
+-- Host: files    Database: backup_index
+-- ------------------------------------------------------
+-- Server version	5.5.41-0ubuntu0.14.04.1
+
+/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
+/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
+/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
+/*!40101 SET NAMES utf8 */;
+/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
+/*!40103 SET TIME_ZONE='+00:00' */;
+/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
+/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
+/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
+/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
+
+--
+-- Dumping data for table `settings`
+--
+
+LOCK TABLES `settings` WRITE;
+/*!40000 ALTER TABLE `settings` DISABLE KEYS */;
+INSERT INTO `settings` VALUES ('ROOTDIR','/mnt/media'),('DIRS','video itunes-media pictures books pr0n ringtones scripts podcasts');
+/*!40000 ALTER TABLE `settings` ENABLE KEYS */;
+UNLOCK TABLES;
+/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
+
+/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
+/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
+/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
+/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
+/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
+/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
+/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
+
+-- Dump completed on 2015-01-30  9:53:38
--- a/update-index.sh
+++ b/update-index.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+source settings.sh
+
+echo -n | mysql --default-character-set=utf8 -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB >/dev/null
+if [ $? -ne 0 ]
+then
+  echo "unable to connect to database"
+  exit 1
+fi
+
+echo "select name from excluded_dirs;" | mysql --default-character-set=utf8 -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB | grep -v ^name\$ | sed "s/\//\\\\\//g;s/^/s\/^/;s/\$/\\\\\/.\*\/\//" >exclude.sed
+TZ=UTC find $PATHS -type f -exec ls --full-time \{} \; | sed "s/.*$USER users //;s/\.0*//;s/ +0000//;s/ /\t/;s/ \\//\t\//;s/\(.*\)\t\(.*\)\t\(.*\)/\3\t\1\t\2/;s/$SEDROOTDIR\\///" | sed -f exclude.sed | grep -v ^\$ | sort | awk 'BEGIN {FS="\t"} {printf("%s\t%s\t%s\n",$1,$2,substr($3,1,19));}' >current-files
+echo "select filename, filesize, filedate from backup_index;" | mysql --default-character-set=utf8 -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB | grep -Pv "filename\tfilesize\tfiledate" | sort >backup-index.tmp
+sed 's/[ \t]*$//' backup-index.tmp >backup-index.tmp~ && mv backup-index.tmp~ backup-index.tmp
+sed 's/[ \t]*$//' current-files >current-files~ && mv current-files~ current-files
+diff -u backup-index.tmp current-files >changes
+grep ^- changes | grep -v ^--- >removals
+grep ^+ changes | grep -v ^+++ >additions
+rm changes
+sed "s/-\(.*\)\t\(.*\)\t\(.*\)/\1/;s/'/''/g;s/^/delete from backup_index where filename='/;s/\$/';/" removals >changes.sql
+sed "s/^+//;s/'/''/g;s/\(.*\)\t\(.*\)\t\(.*\)/insert into backup_index (filename, filesize, filedate) values ('\1\', \2, '\3');/" additions >>changes.sql
+mysql --default-character-set=utf8 -h $MYSQL_HOST -u $MYSQL_USER --password=$MYSQL_PASSWD $MYSQL_DB <changes.sql
+rm backup-index.tmp changes.sql
+echo `wc additions | awk '{print $1;}'` added, `wc removals | awk '{print $1;}'` removed
+rm current-files additions removals exclude.sed
--- a/wait-on-drive.sh
+++ b/wait-on-drive.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+err=-1
+while [ $err != 0 ]
+do
+  mount /mnt/cdrom 2>&1 >/dev/null
+  err=$?
+  sleep 2
+done
+umount /mnt/cdrom 2>&1 >/dev/null