Backing up your data is a popular option for securing your database. In order to create backup copies, you need additional hardware and to install a suitable backup structure. How do you secure your own network and web server against attacks and proceed to protect your databases?
If you do not backup your server regularly, you have almost no way of restoring lost files. As a result, backup copies should always play an important role in your server planning. The rsync synchronization tool/protocol is one of the many backup solutions available.
- What is rsync, and what is it used for?
- Overview of the most important rsync options
- Setting up an rsync backup on Linux servers
- 5 popular rsync backup scenarios
- Backup including character format conversion
What is rsync, and what is it used for?
The name rsync refers to a network protocol for file synchronization developed in 1996 by Andrew Tridgell (among others), who headed up the SAMBA project. In addition to the protocol, there is a tool of the same name that facilitates communication over the protocol. The GPL-licensed application can be used free of charge and transfers files from a source directory to the required target directory – locally or via a public network. The file size and creation time of all files in the source and target folders are compared during syncing, so that each sync process only copies changed files. For this reason, the rsync backup process is called an incremental backup variant, which offers the advantage that backups are fast and take up little storage space.
rsync is available for all standard UNIX operating systems like OS/2, Linux or macOS. Using the API wrapper Cygwin, it can also be used on Microsoft Windows.
Typical use scenarios for rsync include data backups and generation of mirror servers (complete server images) as well as data synchronisation in companies with various locations and weak data connections. In particular, the latter function is almost exclusively performed by modern cloud technologies today – also because weak data connections are becoming increasingly rare.
Overview of the most important rsync options
The outstanding feature of rsync backups is their efficiency. The incremental approach minimises network utilization, which is an advantage for large files in particular – regardless of whether the changes to the files are major or minimal. As a command line tool, rsync is controlled via the Terminal by default – or via the Command Prompt, if you use rsync on a Windows device. The entries have the following syntax:
rsync -(-)Options Source path Target path
As a source and target path, you specify the respective start directory and the directory where you want rsync to save the backup copy. You can define individual settings for your rsync backups using the options, which are either abbreviated to letters or written out. The table below summarises the most important parameters that can be combined with one another:
rsync backup takes all sub-directories into account
Instruction to skip files newer in the target directory than in the source directory
Distinguishes source and target files based on checksums
Symbolic links are copied as such (and not as files)
File permissions are retained
Group file permissions are retained
File time stamps (last change) are retained
File owners are retained (only if administrators)
Device data is retained
Automatic compression of the transferred files
Determines the compression level; values (“NUM”) between 0 (no compression) and 9 (maximum compression) are possible
More comprehensive details during the backup processes
Hide all details on the backup process (except error messages)
Archival mode used as standard mode and identical to the combination of options -rlptgoD
Test run in which no actual changes are made
Auxiliary menu (can only be used without indicating source and target directories or other arguments)
Restrict bandwidth (kilobytes per seconds); e.g. --bwlimit=30 (limit of 30 kbit/s)
Exclude a pattern from synchronisation; e.g. --exclude sample folder (the “Sample folder” folder is not synchronized.)
Delete all files that are in the target directory but not in the source directory
Show the duration of the rsync backups and the transfer speed
List files instead of a backup
Comprehensive report on the transferred data (number, size)
Define a maximum file size; e.g. --max-size=10MB (only files with a size up to 10 MB are transferred.)
Prevent cancellation of the backup process in the event of an error
Setting up an rsync backup on Linux servers
To use rsync on Linux operating systems, install the protocol in the package of the same name and create your backups using Terminal commands. Alternatively you can use applications like Back In Time, rsnapshot (for regular automated backups) or Unison to control the backup process via a graphical user interface. In the example below using Ubuntu, we show you the most important steps to set up backup processes using rsync:
rsync is installed by default under Ubuntu. If this is not the case, install it using the following command.
sudo apt-get install rsync
When rsync is installed, use the required Terminal commands to specify the source and target directories and the backup options. For example, you can run the standard mode (“Archive”) as follows:
rsync -a Source directory Target directory
It is advisable to use the test run (-n) and check the accuracy of the parameters and directories specified; inaccurate specifications can result in data losses in a worst case scenario. If some files are not copied as required during the process, this is often due to missing access rights. In cases like this, try running the command as an administrator by prefacing it with the “sudo” parameter.
5 popular rsync backup scenarios
Once you know the fundamental commands, rsync is an outstanding tool for copying files and making backups. You can either test and apply individual command setups or use proven combinations of the available rsync parameters. For example, the following rsync backup scenarios are particularly popular:
Standard backup with archival mode
The archival mode, which copies all files from the source directory to the target directory (including all sub-directories) and retains all authorizations, time stamps and device data, is the ideal and simplest solution in many cases as it combines various options in a single parameter. If you combine the mode with the –v parameter, you will also receive comprehensive status information during the backup process.
rsync -av Source directory Target directory
Identical copy of the source directory
Not only can rsync transfer files from A to B, it can also make identical copies of folders or entire directories. After standard archival, the files that were in the target folder before rsync, but are not in the source folder, are deleted.
rsync -av --delete Source directory Target directory
Backup excluding files of a specific format
If you want to run a backup excluding files in a specific format, you can use the --exclude command to do so. The parameter allows you to define an individual character pattern, that rsync uses as an indicator to ignore a file. The following sample code excludes text files .txt.
rsync -av --exclude'*.txt' Source directory Target directory
Backing up files with a defined minimum or maximum size
If you don’t want a specific character pattern, but the file size to influence exclusion of specific files in an rsync backup, you can use the --max-size and --min-size parameters. If you use the following command, only files of at least 10 MB and at most 100 MB are copied:
rsync -av --min-size=10MB --max-size=100MB Source directory Target directory
Backup including character format conversion
You may have to convert files into a different character format in the target directory – for example if you want to transfer data from a Mac to a Linux server. As standard, Apple devices use UTF8-MAC, which is not available on Linux systems and would cause problems with special characters and umlauts. The --iconv option lets you easily adapt the character encoding as part of the rsync backup process (from UTF8-MAC to UTF8 in the example):
rsync -av --iconv=UTF8-MAC,UTF8 Source directory Target directory