bup: It backs things up

bup is a program that backs things up. It's short for "backup." Can you believe that nobody else has named an open source program "bup" after all this time? Me neither.

Despite its unassuming name, bup is pretty cool. To give you an idea of just how cool it is, I wrote you this poem:

                         Bup is teh awesome
                      What rhymes with awesome?
                        I guess maybe possum
                       But that's irrelevant.

Hmm. Did that help? Maybe prose is more useful after all.

Reasons bup is awesome

bup has a few advantages over other backup software:

• It uses a rolling checksum algorithm (similar to rsync) to split large files into chunks. The most useful result of this is you can backup huge virtual machine (VM) disk images, databases, and XML files incrementally, even though they're typically all in one huge file, and not use tons of disk space for multiple versions.

• It uses the packfile format from git (the open source version control system), so you can access the stored data even if you don't like bup's user interface.

• Unlike git, it writes packfiles directly (instead of having a separate garbage collection / repacking stage) so it's fast even with gratuitously huge amounts of data. bup's improved index formats also allow you to track far more filenames than git (millions) and keep track of far more objects (hundreds or thousands of gigabytes).

• Data is "automagically" shared between incremental backups without having to know which backup is based on which other one - even if the backups are made from two different computers that don't even know about each other. You just tell bup to back stuff up, and it saves only the minimum amount of data needed.

• You can back up directly to a remote bup server, without needing tons of temporary disk space on the computer being backed up. And if your backup is interrupted halfway through, the next run will pick up where you left off. And it's easy to set up a bup server: just install bup on any machine where you have ssh access.

• Bup can use "par2" redundancy to recover corrupted backups even if your disk has undetected bad sectors.

• Even when a backup is incremental, you don't have to worry about restoring the full backup, then each of the incrementals in turn; an incremental backup acts as if it's a full backup, it just takes less disk space.

• You can mount your bup repository as a FUSE filesystem and access the content that way, and even export it over Samba.

• It's written in python (with some C parts to make it faster) so it's easy for you to extend and maintain.

Reasons you might want to avoid bup

• It's not remotely as well tested as something like tar, so it's more likely to eat your data. It's also missing some probably-critical features, though fewer than it used to be.

• It requires python 3.7 or newer, a C compiler, and an installed git version >= 1.5.6. It also requires par2 if you want fsck to be able to generate the information needed to recover from some types of corruption.

• It currently only works on Linux, FreeBSD, NetBSD, OS X >= 10.4, Solaris, or Windows (with Cygwin, and WSL). Patches to support other platforms are welcome.

• Any items in "Things that are stupid" below.

Notable changes introduced by a release

• Changes in 0.33.10 as compared to 0.33.9
• Changes in 0.33.9 as compared to 0.33.8
• Changes in 0.33.8 as compared to 0.33.7
• Changes in 0.33.7 as compared to 0.33.6
• Changes in 0.33.6 as compared to 0.33.5
• Changes in 0.33.5 as compared to 0.33.4
• Changes in 0.33.4 as compared to 0.33.3
• Changes in 0.33.3 as compared to 0.33.2
• Changes in 0.33.2 as compared to 0.33.1
• Changes in 0.33.1 as compared to 0.33
• Changes in 0.33 as compared to 0.32
• Changes in 0.32.2 as compared to 0.32.1
• Changes in 0.32.1 as compared to 0.32
• Changes in 0.32 as compared to 0.31
• Changes in 0.31 as compared to 0.30.1
• Changes in 0.30.1 as compared to 0.30
• Changes in 0.30 as compared to 0.29.3
• Changes in 0.29.3 as compared to 0.29.2
• Changes in 0.29.2 as compared to 0.29.1
• Changes in 0.29.1 as compared to 0.29
• Changes in 0.29 as compared to 0.28.1
• Changes in 0.28.1 as compared to 0.28
• Changes in 0.28 as compared to 0.27.1
• Changes in 0.27.1 as compared to 0.27

Test status

main

Getting started

From source

• Check out the bup source code using git:

  sh git clone https://github.com/bup/bup

• This will leave you on the main branch, which is perfect if you would like to help with development, but if you'd just like to use bup, please check out the latest stable release like this:

  sh git checkout 0.33.10

You can see the latest stable release here: https://github.com/bup/bup/tags

• Install the required python libraries (including the development libraries).

For bup fuse you will need to install python-fuse rather than fusepy. For example, in Debian, install python3-fuse rather than python3-fusepy.

On very recent Debian/Ubuntu versions, this may be sufficient (run as root):

```sh
apt-get build-dep bup
```

Otherwise try this:

```sh
apt-get install python3-dev python3-fuse
apt-get install python3-pyxattr python3-pytest
apt-get install python3-distutils
apt-get install pkg-config linux-libc-dev libacl1-dev
apt-get install gcc make acl attr rsync
apt-get isntall python3-pytest-xdist # optional (parallel tests)
apt-get install par2 # optional (error correction)
apt-get install libreadline-dev # optional (bup ftp)
apt-get install python3-tornado # optional (bup web)

```

On Cygwin, install python, make, rsync, and gcc4.

If you would like to use the optional bup web server on systems without a tornado package, you may want to try this:

```sh
pip install tornado
```

• Build:

  sh make

At the moment the build treats compiler warnings as errors. If the build fails as a result, try this:

sh CFLAGS=-Wno-error ./configure make

• Run the tests:

  sh make long-check

  or if you're in a bit more of a hurry:

  sh make check

  Additional (and some slightly differing) tests will be run if you're either root, or running under fakeroot, and for the root case, in order to include all of the tests, you'll want to make sure the fuse and loop modules are loaded, which you can do for Linux via (as root):

  sh modprobe fuse modprobe loop

  If you have the Python xdist module installed, then you can probably run the tests faster by adding the make -j option (see ./HACKING for additional information):

  sh make -j check

  The tests should pass (with some skipped tests that weren't applicable in your environment). If they don't pass for you, stop here and send an email to bup-list@googlegroups.com. Though if there are symbolic links along the current working directory path, the tests may fail. Running something like this before "make test" should sidestep the problem:

  sh cd "$(pwd -P)"

• You can install bup via "make install", and override the default destination with DESTDIR and PREFIX.

Files are normally installed to "$DESTDIR/$PREFIX" where DESTDIR is empty by default, and PREFIX is set to /usr/local. So if you wanted to install bup to /opt/bup, you might do something like this:

```sh
make install DESTDIR=/opt/bup PREFIX=''
```

• The Python version that bup will use is determined by the python-config program chosen by ./configure, which will search for a reasonable version unless BUP_PYTHON_CONFIG is set in the environment. You can see which Python executable was chosen by looking at the configure output, or examining config/config.var/bup-python-config, and you can change the selection by re-running ./configure.

• If you want to specify your own CPPFLAGS, CFLAGS, or LDFLAGS, you can set them for individual make invocations, e.g. make CFLAGS=-O0 check, or persistently via ./configure with CFLAGS=-O0 ./configure. At the moment, make clean clears the configuration, but we may change that at some point, perhaps by adding and requiring a make distclean to clear the configuration.

From binary packages

Binary packages of bup are known to be built for the following OSes:

• Debian
• Ubuntu
• pkgsrc.se (NetBSD, Dragonfly, and others)
• NetBSD
• Arch Linux
• macOS (Homebrew)

Using bup

• Get help for any bup command:

  sh bup help bup help init bup help index bup help save bup help restore ...

• Initialize the default bup repository (~/.bup -- you can choose another by either specifying bup -d DIR ... or setting the BUP_DIR environment variable for a command):

  sh bup init

• Make a local backup (-v or -vv will increase the verbosity):

  sh bup index /etc bup save -n local-etc /etc

• Restore a local backup to ./dest:

  sh bup restore -C ./dest local-etc/latest/etc ls -l dest/etc

• Look at how much disk space your backup took:

  sh du -s ~/.bup

• Make another backup (which should be mostly identical to the last one; notice that you don't have to specify that this backup is incremental, it just saves space automatically):

  sh bup index /etc bup save -n local-etc /etc

• Look how little extra space your second backup used (on top of the first):

  sh du -s ~/.bup

• Get a list of your previous backups:

  sh bup ls local-etc

• Restore your first backup again:

  sh bup restore -C ./dest-2 local-etc/2013-11-23-11195/etc

• Make a backup to a remote server which must already have the 'bup' command somewhere in its PATH (see /etc/profile, etc/environment, ~/.profile, or ~/.bashrc), and be accessible via ssh. Make sure to replace SERVERNAME with the actual hostname of your server:

  sh bup init -r SERVERNAME:path/to/remote-bup-dir bup index /etc bup save -r SERVERNAME:path/to/remote-bup-dir -n local-etc /etc

• Make a remote backup to ~/.bup on SERVER:

  sh bup index /etc bup save -r SERVER: -n local-etc /etc

• See what saves are available in ~/.bup on SERVER:

  sh bup ls -r SERVER:

• Restore the remote backup to ./dest:

  sh bup restore -r SERVER: -C ./dest local-etc/latest/etc ls -l dest/etc

• Defend your backups from death rays (OK fine, more likely from the occasional bad disk block). This writes parity information (currently via par2) for all of the existing data so that bup may be able to recover from some amount of repository corruption:

  sh bup fsck -g

• Use split/join instead of index/save/restore. Try making a local backup using tar:

  sh tar -cvf - /etc | bup split -n local-etc -vv

• Try restoring the tarball:

  sh bup join local-etc | tar -tf -

• Look at how much disk space your backup took:

  sh du -s ~/.bup

• Make another tar backup:

  sh tar -cvf - /etc | bup split -n local-etc -vv

• Look at how little extra space your second backup used on top of the first:

  sh du -s ~/.bup

• Restore the first tar backup again (the ~1 is git notation for "one older than the most recent"):

  sh bup join local-etc~1 | tar -tf -

• Get a list of your previous split-based backups:

  sh GIT_DIR=~/.bup git log local-etc

• Save a tar archive to a remote server (without tar -z to facilitate deduplication):

  sh tar -cvf - /etc | bup split -r SERVERNAME: -n local-etc -vv

• Restore the archive:

  sh bup join -r SERVERNAME: local-etc | tar -tf -

That's all there is to it!

No