My name is Philipp C. Heckel and I write about nerdy things.
This site moved here recently from blog.philippheckel.com!

zfsu: ZFS utils for offsite backup, retention and maintaining a slow mirror


Administration, Linux

zfsu: ZFS utils for offsite backup, retention and maintaining a slow mirror


My laptop runs ZFS as its root file system (see this blog post) — meaning that I can snapshot my root file system and I can send it to another machine as a backup very easily. Unfortunately, while ZFS provides the raw functionality, there is no great tool to manage offsite backups and retention. To ease this pain, I wrote/forked and packaged a few helper scripts which I called zfsu, a collection of ZFS utilities.

It consists of the following tools: zfsu tx (aka zfstx) maintains a mirror of a ZFS pool over the network. zfsu ret (aka zfsret) is a simple script to apply local retention (destroy snapshots) of a file system and its snapshots. zfsu res (aka zfsres) is a script to resilver a slow mirror, e.g. a HDD disk if mirrored with a SSD.


Content


0. Code available on GitHub

All the code for this post is available in a zfsu Project on GitHub. Feel free to poke around there and/or steal bits and pieces from it.

1. Installation

I host a package of the scripts in my Debian repository, so if you are Debian/Ubuntu based, feel free to install it like this:

If you’re using a different system, simply download and place these scripts in your PATH.

2. zfsu tx (aka zfstx)

zfstx lets you pull ZFS snapshots from a remote host into the local zpool, meaning that you need to install this script on the backup machine itself, not on the machine that’s being backed up. While this may seem strange at first, it’s great if you’re backing up many different machines, because you can manage the schedule and logic in one place.

The tool was originally developed by Jaako as part of his zfs_transfer tool. I merely adjusted it a little and packaged it for my needs. Many thanks to him for this great tool.

2.1. Usage

2.2. Examples

3. zfsu ret (aka zfsret)

zfsret is a simple script to apply local retention (destroy snapshots) of a file system and its snapshots. Its meant to complement the zfstx utility to only maintain a certain number of snapshots on the backup host. It can also be used to locally prune snapshots on demand, or regularly.

The retention logic is pretty simple as you can only specify how many of the last snapshots to keep. There is no sophisticated logic (yet) to keep X per week or Y per month.

3.1. Usage

3.2. Examples

4. zfsu res (aka zfsres)

zfsres is a tool specifically written for a ZFS mirror/RAID1 between a slow/HDD and a fast/SSD disk. zfsres will regularly resilver the slow HDD and thereby not make the array slow for normal write/read operations, but still allow for a RAID1 configuration of two disks of vastly different speeds.

This tool covers a pretty exotic use case. I wrote it because my laptop has a SSD and a HDD, and I wanted to keep a second copy on the HDD. Note that this tool only makes sense in this scenario.

4.1. Usage

4.2. Examples

5. Example: My setup

I use these scripts to back up 5 different machines, some of which have ZFS file systems, and others that do not. My backup hub’s ZFS pool looks like this (shortened):

Backups of these machines are tiggered in a cronjob by a custom wrapper script called gobackup, which uses the zfsu utilities. It pretty much looks like this (representative excerpt):

Leave a comment

I'd very much like to hear what you think of this post. Feel free to leave a comment. I usually respond within a day or two, sometimes even faster. I will not share or publish your e-mail address anywhere.