backup/backupMgr

Remote Backup Manager

This script uses rsync to backup files from any source (e.g. Remote) to any target (usually the host, the script runs on). On completion (either successful or on error), the scripts sends a mail with the output of rsync to a specified mail address.

Dependencies

sendmail needs to be installed and configured for sending backup notifications. rsync needs to be installed on all remote sources/targets and on the host which runs the script.

Installation

Pull the repository to a location of your choice on the host.

Configuration

Configure your ssh client for all remote source or target directories specified in your configuration, such that rsync neither requires passwords, ports, etc. Use the config file under your .ssh folder. An example configuration for a host is given as follows.

Host sebastianvendt.de
        HostName sebastianvendt.de
        Port 1234
        IdentityFile /home/sebastian/sshKeys/rsyncOnSebastianvendt.de.pem

Create a configuration file, which holds the source and destination paths for the backup jobs as well as several configuration variables. An example configuration file with all required variables is given in the following.

#Backup directories and source specific exclude file
declare -A backup_folders 
backup_folders["backupuser@hostA.com:/opt/SourceFolderA"]="opt/backup/folderA"
backup_folders["/opt/SourceFolderB"]="opt/backup/folderB;/path/to/SourceFolderBSpecific/excludes.txt"

#Configuration
logPath=/home/user/logs/backupjob/
logFileName="$(date +'%Y-%m-%d_%H-%M')_backupjobs.log"
tmpLogFilePath=/tmp/tmp_backup.log
retryAttempts=2
mailFrom="backup@backupserver.com"
mailTo="user@domain.com"

#Delete logs older than rotationDays
rotationDays=14

As in the example, for each source a exclude file can be optionally provided. The path to the exclude file is appended to the destination path and separated by a semicolon. The excludes.txt can look as follows

*.log
/tmp/files/

How to run the script

The script requires as argument the path to the configuration file

/path/of/script/getRemoteBackup.sh /path/to/configuration

Automate the backup job

To automate your backup, add the script with the path to your configuration to your crontab-file. I suggest to run it in a detached screen session. You can then always reattach to the screen and check the current status for long backup jobs.

10 2 * * * sebastian screen -d -m /home/sebastian/scripts/backup/getRemoteBackup.sh /home/sebastian/scripts/backup/backupFolders.conf

The core element

The heart of the script is an rsync command:

rsync --delete --out-format="%t %f %''b" -avz $sourceFolder ${backup_folders[$sourceFolder]} | tee -a $tmpLogFilePath 2>&1

sending  side),  but  only  for  the directories that are being synchronized.  You must have asked
rsync to send the whole directory (e.g.  "dir"  or  "dir/")  without  using  a  wildcard  for  the
directory’s  contents  (e.g.  "dir/*")  since the wildcard is expanded by the shell and rsync thus
gets a request to transfer individual files, not the files’  parent  directory.   Files  that  are
excluded   from   the   transfer  are  also  excluded  from  being  deleted  unless  you  use  the
**--delete-excluded** option or mark the  rules  as  only  matching  on  the  sending  side  (see  the
include/exclude modifiers in the FILTER RULES section).
       

-a, --archive```: This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when --files-from is specified, in which case -r is not implied. Note that -a does not preserve hardlinks, because finding multiply-linked files is expensive. You must separately specify -H.

compressing  remote  shell  or  a compressing transport because it takes advantage of the implicit
information in the matching data blocks that are not explicitly sent over the connection.
See the **--skip-compress** option for the default list of file suffixes that will not be compressed.

-v, --verbose```: Be verbose :)

--out-format="%t %f %''b": Defines the output format as follows 2024/03/10 02:10:14 appdata_ocn6dip2flyl/appstore/apps.json 16.80K