Rsync is a fast, versatile, remote (and local) file-copying tool. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. It is available on most Unix-like systems, including the FAS RC cluster and Mac OS X.
The basic syntax is:
rsync SOURCE DESTINATION where
DESTINATION are filesystem paths.
They can be local, either absolute or relative to the current working directory, or they can be remote but prefixing something like
USERNAME@HOSTNAME: to the front of them.
cp and most shell commands, a trailing
/ character on a directory name is significant — it means the contents of the directory as opposed to the directory itself.
- As a replacement for
cp— copying a single large file, but with a progress meter:rsync –progress bigfile bigfile-copy
- Make a recursive copy of local directory
foo-copy:rsync -av foo/ foo-copy/
The trailing slash on
foo-copy/is optional, but if it’s not on
foo/, the file
foo/myfilewill appear as
- Upload the directory foo on the local machine to your home directory on the cluster:
rsync -avz foo/ MYUSERNAME@login.rc.fas.harvard.edu:~/foo/
This works for individual files, too, just don’t put the trailing slashes on them.
- Download the directory foo in your home directory on the cluster to the local machine:
rsync -avz MYUSERNAME@login.rc.fas.harvard.edu:~/foo .
- Update a previously made copy of foo on the cluster after you’ve made changes to the local copy:
rsync -avz –delete foo/ MYUSERNAME@login.rc.fas.harvard.edu:~/foo/
--deleteoption has no effect when making a new copy, and therefore can be used the previous example, too (making the commands identical), but since it recursively deletes files, it’s best to use it sparingly.
- Update a previously made copy of foo on the cluster after you or someone else has already updated it from a different source:
rsync -avz –update foo/ MYUSERNAME@login.rc.fas.harvard.edu:~/foo/
--updateoptions has no effect when making a new copy, and can freely be specified in that case, also.
- Make a backup of your entire linux system to
/mnt/MYBACKUPDRIVE:rsync -a –exclude /proc/ –exclude /sys/ –exclude /tmp/ –exclude /var/tmp/ –exclude /mnt/ –exclude /media/ /mnt/MYBACKUPDRIVE
--excludeoptions, if appropriate.
See rdiff-backup for a better way of making backups.
DESTINATION are on different machines with fast CPUs, especially if they’re on different networks (e.g. your home computer and the FASRC cluster), it’s recommended to add the
-z option to compress the data that’s transferred.
This will cause more CPU to be used on both ends, but it is usually faster.
File Attributes, Permissions, Ownership, etc.
rsync does not copy recursively, preserve timestamps, preserve non-default permissions, etc.
There are individual options for all of these things, but the option
-a, which is short for archive mode, sums up many of these (
-rlptgoD) and is best for producing the most exact copy.
-A (preserve ACLs),
-X (preserve extended attributes), and
-H (preserve hardlinks) may also be desired on rare occasions.)
Note that if you are copying files not owned by you, preserving file ownership only works if you are root at the destination. If you are copying between systems on different authentication infrastructures, and the user/group does not exist at the destination, the numeric id is used. If that numeric id corresponds to a different user/group, the files will appear to be owned by that other user/group. If the user/group does exist on the destination, and the numeric id is different, the numeric id changes accordingly. The option
--numeric ids changes this behavior, but introduces some issues of its own, so is not recommended by default.
Updating a Copy
Rsync’s delta-transfer algorithm allows you to efficiently update copies you’ve previously made by only sending the differences needed to update the
DESTINATION instead of re-copying it from scratch.
However, there are some addition options you will probably want to use depending on the type of copy you’re trying to maintain.
If you want to maintain a mirror, i.e. the
DESTINATION is to be an exact copy of the
SOURCE, then you will want to add the
This deletes stuff in the
DESTINATION that is no longer in the
Be careful with this option!
If you incorrectly specify the
DESTINATION you may accidentally delete many files.
See also the
--delete-excluded option if you’re adding
--exclude options that were not used when making the original copy.
If you’re updating a master copy, i.e. the
DESTINATION may have files that are newer than the versions in
SOURCE, you will want to add the
This will leave those files alone, not revert them to the older copy in
Progress, Verbosity, Statistics
Verbose mode — list each file transferred.
vs makes it more verbose.
Show a progress meter for each file transfer (not a progress meter for the whole operation).
If you have many small files, this can significantly slow down the transfer.
Print a short paragraph of statistics at the end of the session, like average transfer rate, total numbers of files transferred, etc.
Other Useful Options
Perform a dry-run of the session instead of actually modifying the DESTINATION.
Most useful when adding multiple
-voptions, especially for verifying
--deleteis doing what you want.
Skip some parts of the