Rsync

Rsync is a free and open source software designed to transfer files between a local and a remote storage. SyncPlanet uses Rsync among other tools to trigger file discovery and to transfer files.

Scopes

In SyncPlanet, the different Rsync options are organized into 3 categories called scopes:

• Hub scope : Global rsync options, active for every transfers of the hub.
• Storage scope : Specific options for a storage unit, active for all the storage's transfers.
• Channel scope : Options for every transfers of a specific channel, in 1 direction.

Command line

Rsync is driven by entering specific options in the relevant field under the parameter tabs of the hub, storage units, and channels.

The options are passed to the rsync command in a specific order:

1. hub options
2. storage options
3. channel options

A typical rsync command line used by SyncPlanet for the hub without specific storage and channel options looks like this:

rsync --recursive --times --update --omit-dir-times <LOG_MANAGEMENT> <FILTERS_MANAGEMENT> <SOURCE> <TARGET>

      \_________________hub scope_________________/ \_____________internals_____________/

Option list

In this section you will find a breakdown of the most commonly used options for the different scopes. This list is non exhaustive as there are many different options. For options not included in this article, please see the html version of the Rsync manpage

Options are listed alphabetically and marked with icons to indicate to which elements they may be applied:

• This option can be used for the hub.
• This option can be used for storage units.
• This option can be used for channels.

--links --omit-dir-times --progress --recursive --stop-after=120 --times --update

--perms --chmod=ug=rwX,o=rX

Each option is explained below, providing the manual of Rsync, some more information and some hints.

--backup, -b

Make backups (see --suffix & --backup-dir).

Rsync manual

With this option, preexisting destination files are renamed as each file is transferred or deleted. You can control where the backup file goes and what (if any) suffix gets appended using the --backup-dir and --suffix options.

If you don't specify --backup-dir:

1. the --omit-dir-times option will be forced on
2. the use of --delete (without --delete-excluded), causes rsync to add a "protect" filter-rule for the backup suffix to the end of all your existing filters that looks like this: -f "P *~". This rule prevents previously backed-up files from being deleted.

Note that if you are supplying your own filter rules, you may need to manually insert your own exclude/protect rule somewhere higher up in the list so that it has a high enough priority to be effective (e.g., if your rules specify a trailing inclusion/exclusion of *, the auto-added rule would never be reached).

--compress, -z

Compress file data during the transfert.

Rsync manual

With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection.

Rsync supports multiple compression methods and will choose one for you unless you force the choice using the --compress-choice (--zc) option.

Run rsync --version to see the default compress list compiled into your version.

When both sides of the transfer are at least 3.2.0, rsync chooses the first algorithm in the client's list of choices that is also in the server's list of choices. If no common compress choice is found, rsync exits with an error. If the remote rsync is too old to support checksum negotiation, its list is assumed to be "zlib".

The default order can be customized by setting the environment variable RSYNC_COMPRESS_LIST to a space-separated list of acceptable compression names. If the string contains a "&" character, it is separated into the "client string & server string", otherwise the same string applies to both. If the string (or string portion) contains no non-whitespace characters, the default compress list is used. Any unknown compression names are discarded from the list, but a list with only invalid names results in a failed negotiation.

There are some older rsync versions that were configured to reject a -z option and require the use of -zz because their compression library was not compatible with the default zlib compression method. You can usually ignore this weirdness unless the rsync server complains and tells you to specify -zz.

--delete

Delete extraneous files from dest dirs.

Rsync manual

This tells rsync to delete extraneous files from the receiving side (ones that aren't on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory (e.g. "dir" or "dir/") without using a wildcard for the directory's contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files' parent directory. Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).

Prior to rsync 2.6.7, this option would have no effect unless --recursive was enabled. Beginning with 2.6.7, deletions will also occur when --dirs (-d) is enabled, but only for directories whose contents are being copied.

This option can be dangerous if used incorrectly! It is a very good idea to first try a run using the --dry-run (-n) option to see what files are going to be deleted.

If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. This is to prevent temporary filesystem failures (such as NFS errors) on the sending side from causing a massive deletion of files on the destination. You can override this with the --ignore-errors option.

The --delete option may be combined with one of the --delete-WHEN options without conflict, as well as --delete-excluded. However, if none of the --delete-WHEN options are specified, rsync will choose the --delete-during algorithm when talking to rsync 3.0.0 or newer, or the --delete-before algorithm when talking to an older rsync. See also --delete-delay and --delete-after.

--dirs, -d

Transfer directories without recursing.

Rsync manual

Tell the sending side to include any directories that are encountered. Unlike --recursive, a directory's contents are not copied unless the directory name specified is "." or ends with a trailing slash (e.g. ".", "dir/.", "dir/", etc.). Without this option or the --recursive option, rsync will skip all directories it encounters (and output a message to that effect for each one). If you specify both --dirs and --recursive, --recursive takes precedence.

The --dirs option is implied by the --files-from option or the --list-only option (including an implied --list-only usage) if --recursive wasn't specified (so that directories are seen in the listing). Specify --no-dirs (or --no-d) if you want to turn this off.

There is also a backward-compatibility helper option, --old-dirs (or --old-d) that tells rsync to use a hack of -r --exclude='/*/*' to get an older rsync to list a single directory without recursing.

--dry-run, -n

Perform a trial run with no changes made.

Rsync manual

This makes rsync perform a trial run that doesn't make any changes (and produces mostly the same output as a real run). It is most commonly used in combination with the --verbose, -v and/or --itemize-changes (-i) options to see what an rsync command is going to do before one actually runs it.

The output of --itemize-changes is supposed to be exactly the same on a dry run and a subsequent real run (barring intentional trickery and system call failures); if it isn't, that's a bug. Other output should be mostly unchanged, but may differ in some areas. Notably, a dry run does not send the actual data for file transfers, so --progress has no effect, the "bytes sent", "bytes received", "literal data", and "matched data" statistics are too small, and the "speedup" value is equivalent to a run where no file transfers were needed.

--fake-super

Store/recover privileged attrs using xattrs.

Rsync manual

When this option is enabled, rsync simulates super-user activities by saving/restoring the privileged attributes via special extended attributes that are attached to each file (as needed). This includes the file's owner and group (if it is not the default), the file's device info (device & special files are created as empty text files), and any permission bits that we won't allow to be set on the real file (e.g. the real file gets u-s,g-s,o-t for safety) or that would limit the owner's access (since the real super-user can always access/change a file, the files we create can always be accessed/changed by the creating user). This option also handles ACLs (if --acls was specified) and non-user extended attributes (if --xattrs was specified).

This is a good way to backup data without using a super-user, and to store ACLs from incompatible systems.

The --fake-super option only affects the side where the option is used. To affect the remote side of a remote-shell connection, use the --remote-option (-M) option:

rsync -av -M --fake-super" /src/ host:/dest/

For a local copy, this option affects both the source and the destination. If you wish a local copy to enable this option just for the destination files, specify -M--fake-super. If you wish a local copy to enable this option just for the source files, combine --fake-super with -M--super.

This option is overridden by both --super and --no-super.

--ignore-existing

Skip updating files that exist on receiver.

Rsync manual

This tells rsync to skip updating files that already exist on the destination (this does not ignore existing directories, or nothing would get done). See also --ignore-non-existing.

This option is a TRANSFER RULE, so don't expect any exclude side effects.

This option can be useful for those doing backups using the --link-dest option when they need to continue a backup run that got interrupted. Since a --link-dest run is copied into a new directory hierarchy (when it is used properly), using --ignore-existing will ensure that the already-handled files don't get tweaked (which avoids a change in permissions on the hard-linked files). This does mean that this option is only looking at the existing files in the destination hierarchy itself.

When --info=skip2 is used rsync will output "FILENAME exists (INFO)" messages where the INFO indicates one of "type change", "sum change" (requires -c), "file change" (based on the quick check), "attr change", or "uptodate". Using --info=skip1 (which is also implied by 2 -v options) outputs the exists message without the INFO suffix.

--links, -l

Copy symlinks as symlinks.

Rsync manual

When symlinks are encountered, recreate the symlink on the destination.

--omit-dir-times, -O

Omit directories from --times.

Rsync manual

This tells rsync to omit directories when it is preserving modification, access, and create times. If NFS is sharing the directories on the receiving side, it is a good idea to use -O. This option is inferred if you use --backup without --backup-dir.

This option also has the side-effect of avoiding early creation of missing sub-directories when incremental recursion is enabled, as discussed in the --inc-recursive section.

--progress

Show progress during transfer.

Rsync manual

This option tells rsync to print information showing the progress of the transfer. This gives a bored user something to watch. With a modern rsync this is the same as specifying --info=flist2,name,progress, but any user-supplied settings for those info flags takes precedence (e.g. --info=flist0 --progress).

While rsync is transferring a regular file, it updates a progress line that looks like this:

782448  63%  110.64kB/s    0:00:04

In this example, the receiver has reconstructed 782448 bytes or 63% of the sender's file, which is being reconstructed at a rate of 110.64 kilobytes per second, and the transfer will finish in 4 seconds if the current rate is maintained until the end.

These statistics can be misleading if rsync's delta-transfer algorithm is in use. For example, if the sender's file consists of the basis file followed by additional data, the reported rate will probably drop dramatically when the receiver gets to the literal data, and the transfer will probably take much longer to finish than the receiver estimated as it was finishing the matched part of the file.

When the file transfer finishes, rsync replaces the progress line with a summary line that looks like this:

1,238,099 100%  146.38kB/s    0:00:08  (xfr#5, to-chk=169/396)

In this example, the file was 1,238,099 bytes long in total, the average rate of transfer for the whole file was 146.38 kilobytes per second over the 8 seconds that it took to complete, it was the 5th transfer of a regular file during the current rsync session, and there are 169 more files for the receiver to check (to see if they are up-to-date or not) remaining out of the 396 total files in the file-list.

In an incremental recursion scan, rsync won't know the total number of files in the file-list until it reaches the ends of the scan, but since it starts to transfer files during the scan, it will display a line with the text "ir-chk" (for incremental recursion check) instead of "to-chk" until the point that it knows the full size of the list, at which point it will switch to using "to-chk". Thus, seeing "ir-chk" lets you know that the total count of files in the file list is still going to increase (and each time it does, the count of files left to check will increase by the number of the files added to the list).

--recursive, -r

Recurse into directories.

Rsync manual

This tells rsync to copy directories recursively. See also --dirs (-d) for an option that allows the scanning of a single directory.

See the --inc-recursive option for a discussion of the incremental recursion for creating the list of files to transfer.

--stop-after=MINS

Stop rsync after MINS minutes have elapsed.

Rsync manual

This option tells rsync to stop copying when the specified number of minutes has elapsed.

For maximal flexibility, rsync does not communicate this option to the remote rsync since it is usually enough that one side of the connection quits as specified. This allows the option's use even when only one side of the connection supports it. You can tell the remote side about the time limit using --remote-option (-M), should the need arise.

The --time-limit version of this option is deprecated.

--times, -t

Preserve modification times.

Rsync manual

This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing -t (or -a) will cause the next transfer to behave as if it used --ignore-times (-I), causing all files to be updated (though rsync's delta-transfer algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using -t).

A modern rsync that is using transfer protocol 30 or 31 conveys a modify time using up to 8-bytes. If rsync is forced to speak an older protocol (perhaps due to the remote rsync being older than 3.0.0) a modify time is conveyed using 4-bytes. Prior to 3.2.7, these shorter values could convey a date range of 13-Dec-1901 to 19-Jan-2038. Beginning with 3.2.7, these 4-byte values now convey a date range of 1-Jan-1970 to 7-Feb-2106. If you have files dated older than 1970, make sure your rsync executables are upgraded so that the full range of dates can be conveyed.

--update, -u

Skip files that are newer on the receiver.

Rsync manual

This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file's, it will be updated if the sizes are different.)

Note that this does not affect the copying of dirs, symlinks, or other special files. Also, a difference of file format between the sender and receiver is always considered to be important enough for an update, no matter what date is on the objects. In other words, if the source has a directory where the destination has a file, the transfer would occur regardless of the timestamps.

This option is a TRANSFER RULE, so don't expect any exclude side effects.

A caution for those that choose to combine --inplace with --update: an interrupted transfer will leave behind a partial file on the receiving side that has a very recent modified time, so re-running the transfer will probably not continue the interrupted file. As such, it is usually best to avoid combining this with --inplace unless you have implemented manual steps to handle any interrupted in-progress files.

Next

• Find out more about the Rsync filters for filter-specific options.
• Check the html version of the Rsync manpage from the official Rsync site.
• Read the Rsync manual page from your system:

  man rsync
  

• Visit the Rsync source code repository hosted on GitHub .