backup/backupMgr

Remote Backup Manager

This script uses rsync to backup files from any source (e.g. Remote) to any target (usually the host, the script runs on). On completion (either successful or on error), the scripts sends a mail with the output of rsync to a specified mail address.

Dependencies

sendmail needs to be installed and configured for sending backup notifications. rsync needs to be installed on all remote sources/targets and on the host which runs the script.

Installation

Pull the repository to a location of your choice on the host.

Configuration

Configure your ssh client for all remote source or target directories specified in your configuration, such that rsync neither requires passwords, ports, etc. Use the config file under your .ssh folder. An example configuration for a host is given as follows.

Host sebastianvendt.de
        HostName sebastianvendt.de
        Port 1234
        IdentityFile /home/sebastian/sshKeys/rsyncOnSebastianvendt.de.pem

Create a configuration file, which holds the source and destination paths for the backup jobs as well as several configuration variables. An example configuration file with all required variables is given in the following.

#Backup directories and source specific exclude file
declare -A backup_folders 
backup_folders["backupuser@hostA.com:/opt/SourceFolderA"]="opt/backup/folderA"
backup_folders["/opt/SourceFolderB"]="opt/backup/folderB;/path/to/SourceFolderBSpecific/excludes.txt"

#Configuration
logPath=/home/user/logs/backupjob/
logFileName="$(date +'%Y-%m-%d_%H-%M')_backupjobs.log"
tmpLogFilePath=/tmp/tmp_backup.log
retryAttempts=2
mailFrom="backup@backupserver.com"
mailTo="user@domain.com"

#Delete logs older than rotationDays
rotationDays=14

As in the example, for each source a exclude file can be optionally provided. The absolute path to the exclude file is appended to the destination path and separated by a semicolon. The file needs to be placed on the host, executing the rsync command. Wildcards, as shown in the example can be used. All paths listed in the exclude file shall be given relative to the source folder of the rsync command!
The excludes.txt can look as follows

*.log
tmp/files/

How to run the script

The script requires as argument the path to the configuration file

/path/of/script/getRemoteBackup.sh /path/to/configuration

Automate the backup job

To automate your backup, add the script with the path to your configuration to your crontab-file. I suggest to run it in a detached screen session. You can then always reattach to the screen and check the current status for long backup jobs.

10 2 * * * sebastian screen -d -m /home/sebastian/scripts/backup/getRemoteBackup.sh /home/sebastian/scripts/backup/backupFolders.conf

The core element

The heart of the script is an rsync command:

rsync --delete --out-format="%t %f %''b" -avz $sourceFolder ${backup_folders[$sourceFolder]} | tee -a $tmpLogFilePath 2>&1

--delete: This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory (e.g. "dir" or "dir/") without using a wildcard for the directory’s contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files’ parent directory. Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).

-a, --archive: This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when --files-from is specified, in which case -r is not implied. Note that -a does not preserve hardlinks, because finding multiply-linked files is expensive. You must separately specify -H.

-z, --compress: With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. See the --skip-compress option for the default list of file suffixes that will not be compressed.

-v, --verbose: Be verbose :)

--out-format="%t %f %''b": Defines the output format as follows 2024/03/10 02:10:14 appdata_ocn6dip2flyl/appstore/apps.json 16.80K