I tend to hop from Linux distro to Linux distro. One of the things that makes doing so much easier is keeping my home folder on a separate disk. That way I can re-install distributions to my heart’s content without fear of losing my files and settings.

I’m also a big fan of ZFS (ZFS on Ubuntu Server). That means jumping through a few extra hoops to setup ZFS on a separate drive as well as re-importing the zpool every time I swap distributions, but I find it’s well worth it. Here’s a handy guide on how to do just that! I’ll be showing the steps for Fedora and Kubuntu, but they should generally apply to other distros as well.

Disclaimer: I’ve not a ZFS expert, but these steps have worked very well for me on multiple systems. YMMV.

One quick note: ZFS works best with plenty of RAM (it will use everything available to keep data cached). If you are on a RAM-limited system, you can do something similar with encrypted XFS or EXT4.

Pre-Step: Setup Encrypted Home Drive

I’ll be configuring ZFS to use an encryption key stored on the root drive. This is only secure if the root drive is also encrypted. Make sure when you install Linux you tell the installer to use drive encryption.

It will look like this in Fedora:

And like this in Ubuntu:

You’ll be asked to set a password that’s used to encrypt your root drive. You cannot change this password and you’ll be asked to enter this password every time you boot your computer, so make sure you do not forget it!

Don’t worry about configuring your 2nd drive with your home folder during the installation. I find it’s much easier to have the distribution do it’s typical install, then go back and mount your new /home. Just make sure that you create yourself as an administrator or have a root password set.

Once you’ve installed your new distro, reboot into it, but don’t log in. Your computer will get grumpy if you’re logged into a desktop environment while swapping out your home directory.

Press Control-Alt-F3 to get to a terminal window then log in as yourself if you made your account an administrator, or ‘root’ if you did not.

ZFS Installation

Fedora

Make sure Fedora up-to-date

sudo dnf -y update

If there are any updates, reboot (sudo reboot), press Control-Alt-F3, and log back in.

Install ZFS for Fedora by following the official steps below, do not use the zfs-fuse package included with Fedora: https://openzfs.github.io/openzfs-docs/Getting%20Started/Fedora/index.html

Kubuntu/Ubuntu

sudo apt install zfsutils-linux

Creating a New Home Drive

If you already have created a home drive and are re-attaching after re-installing Linux, skip to Importing an Existing Home Drive.

To make things easier, I’ll be running all of the commands as root by first running:

sudo -s

Create an encryption key that will be used to encrypt and decrypt your home drive. Make sure this is only stored on an encrypted root drive and that you have backed up this key somewhere safe. If you lose this key you will lose all access to your drive. You’ve been warned 😉

dd if=/dev/urandom of=/etc/home.key bs=32 count=1 && chmod 600 /etc/home.key

Next you’ll need to find out the name of your drive. Since easy names (e.g. sda, sdb) can change, we want to set it up by something that will not change. I’ll be using the device’s physical location.

Let’s make sure we know which drive has Linux installed on it, and which is going to be used for our home drive, by runing:

lsblk

This will list all of your drives (also called block devices), any partitions on them, and where those partitions are mounted. My output (on a virtual machine) looks like the following. On real hardware, your devices will probably be called sda and sdb (if they’re SATA), or nvme0n1 and nvme1n1 (if they’re nvme):

NAME                                          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sr0                                            11:0    1    2G  0 rom    
zram0                                         251:0    0  5.8G  0 disk  [SWAP]
vda                                           252:0    0   64G  0 disk   
├─vda1                                        252:1    0    1G  0 part  /boot
└─vda2                                        252:2    0   63G  0 part   
 └─luks-a954d91b-fda3-4c22-90a6-2b35554129b1 253:0    0   63G  0 crypt /home
                                                                       /
vdb                                           252:16   0  128G  0 disk  

I can see here that my disk with Linux installed on it is called vda, since it has multiple partitions (vda1 and vda2) that are all mounted (as /boot and /). The disk with nothing installed on it is vdb. Therefore, I’ll need to check the physical location of vdb. Please comment below if you’re having trouble figuring out which drive is which and I’ll try to give you a hand!

To list all disks by their location, run:

ls -lh /dev/disk/by-path/

The result will look something like this:

total 0
lrwxrwxrwx. 1 root root  9 Jan 28 09:42 pci-0000:00:1f.2-ata-1 -> ../../sr0
lrwxrwxrwx. 1 root root  9 Jan 28 09:42 pci-0000:00:1f.2-ata-1.0 -> ../../sr0
lrwxrwxrwx. 1 root root  9 Jan 28 09:42 pci-0000:07:00.0 -> ../../vda
lrwxrwxrwx. 1 root root 10 Jan 28 09:42 pci-0000:07:00.0-part1 -> ../../vda1
lrwxrwxrwx. 1 root root 10 Jan 28 09:42 pci-0000:07:00.0-part2 -> ../../vda2
lrwxrwxrwx. 1 root root  9 Jan 28 09:42 pci-0000:08:00.0 -> ../../vdb

This tells me that the path I’ll be using is /dev/disk/by-path/pci-0000:08:00.0, since that’s the one that’s being called vdb (see the end of the last line).

We’re finally ready to create our ZFS filesystem! First we create a zpool that encompasses all of the drives we’ll be using (we’ll just be using one, but ZFS can be mirrored or RAIDed in more advanced setups).

The command we’ll run is:

zpool create homepool -O xattr=sa -O acltype=posixacl -O atime=off -O compression=lz4 -O encryption=aes-256-gcm -O keyformat=raw -O keylocation=file:///etc/home.key -o ashift=12 /dev/disk/by-path/[your disk here]

Here’s what some of those options mean:
ashift=12 : This specifies the drive’s block size. From what I’ve cobbled together, use the number 12 for most use cases unless it’s a Samsung NVME or you know your drive uses 8K clusters. In that case, use 13.
homepool: this is the name we’ve given to the zpool. You can use something else if you’d prefer.
compression=lz4: This compresses all data, increases the performance of ZFS, and essentially costs no additional CPU resources. More information here: https://www.servethehome.com/the-case-for-using-zfs-compression/
encyption=aes-256-gcm: Use AES 256 GCM encryption which is both highly secure and hardware accelerated

Now, let’s check out that brand new zpool!

zpool status

You should see something like this:

  pool: homepool
 state: ONLINE
config:

        NAME                STATE     READ WRITE CKSUM
        homepool            ONLINE       0     0     0
          pci-0000:08:00.0  ONLINE       0     0     0

errors: No known data errors

A zpool is a container for filesystems. Now that we’ve got one, we can create a filesystem where our home drive will live. In all of the steps below, replace [user] with your username.

zfs create homepool/[user]

To see information on this new filesystem, you can run:

zfs list

Now, let’s replace our old home drive (that was created when Linux was installed) with the filesystem on our second drive:

cd /home
mv /home/[user] /home/[user].bak
mkdir /home/[user]/
zfs set mountpoint=/home/[user] homepool/[user]
zfs set mountpoint=none homepool
chmod --reference=/home/[user].bak /home/[user]
mv /home/[user].bak/* /home/[user]/
mv /home/[user].bak/.* /home/[user]/
rmdir /home/[user].bak
chown -R [user]:[user] /home/[user]
#For Fedora and other distros with selinux, run the next line too:
restorecon -vR /home

Linux doesn’t yet load keys for encrypted zfs mounts automatically. You’ll need to create a simple service to automatically load zfs encryption keys on boot.
Like most good things, this is from the Arch Linux wiki: https://wiki.archlinux.org/title/ZFS#Unlock_at_boot_time:_systemd
You MUST do this before you reboot or you will not be able to log in graphically. If you forget, press Control-Alt-F3 and log into the console.

nano /etc/systemd/system/zfs-load-key.service

Type in the following (if you’re uncomfortable typing by hand, you should be able to switch to the graphical login (Fedora: Control-Alt-F2, *buntu Control-Alt-F1) and copy paste).

[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/zfs load-key -a
StandardInput=tty-force

[Install]
WantedBy=zfs-mount.service

Next, tell Linux to start the new service every time it boots:

systemctl enable zfs-load-key

Finally, reboot and log in normally to make sure everything works as anticipated:

reboot

Now you should have a fully functioning install with encrypted ZFS home directory! Remember to backup /etc/home.key somewhere secure that *isn’t* in your home directory, since you’ll need to copy this key back any time you re-install Linux. I’d recommend an encrypted USB key.

If you have multiple users, you can follow those same steps to create a zfs filesystem for each of them in zpool you created.

Steam

If you use Steam and want to keep your game installations separate so they don’t get backed up with zfs snapshots, you can create a separate filesystem for it.

mkdir -p /home/[user]/.local/share/Steam

sudo zfs create homepool/[user]/steam -o mountpoint=/home/[user]/.local/share/Steam

Importing an Existing Home Drive

Only follow these steps if you’ve re-installed Linux. They aren’t necessary if you just created a new zpool above.

After you’ve re-installed Linux, make sure you complete ZFS Installation above. Once those are done, you can proceed from here.

If you are not the root user yet, run:

sudo -s

Next, you’ll need to copy your backed up key to /etc/home.key

If it’s stored on an encrypted flash drive, it may be easiest to log in graphically, restore the file, then log out and return the console with Control-Alt-F3.

Once it’s restored, make sure it still has the correct permissions

chown root:root /etc/home.key && chmod 600 /etc/home.key

Rename your existing home directory:

cd /home
mv /home/[user] /home/[user].bak
mkdir /home/[user]
chmod --reference=/home/[user].bak /home/[user]
chown [user]:[user] /home/[user]

List all zpools the system can find for import:

zpool import

You should see your homepool listed

   pool: homepool
     id: 16378698673868876678
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        homepool            ONLINE
          pci-0000:08:00.0  ONLINE

You can now import it by name and mount the zfs filesystems:

zpool import homepool

Before the filesystems can be mounted, we’ll need to create and enable the ZFS key loading service.

nano /etc/systemd/system/zfs-load-key.service

Type in the following:

[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/zfs load-key -a
StandardInput=tty-force

[Install]
WantedBy=zfs-mount.service

Next, tell Linux to start the new service every time it boots, and start it now:

systemctl enable --now zfs-load-key

Finally, we can mount all of the zfs filesystems:

zfs mount -a

You can confirm they are mounted by typing:

mount

The last line of the output should be something similar to:

homepool/adam on /home/adam type zfs (rw,noatime,seclabel,xattr,posixacl)

If you’re on Fedora, be sure to run the following to make selinux happy:

restorecon -vR /home

Reboot and log in. Since all of your personal settings are saved to your home drive, everything should be exactly how you left it!

Memory Usage

If you find ZFS is using too much memory (apps keep crashing), you can adjust how much RAM ZFS uses for its cache (how much of your drive it keeps in memory for quick access).

To test different settings, set the maximum arch size in bytes and then clear the cache. This setting is temporary, so if you run intro trouble, just reboot.

sudo echo "8589934592" > /sys/module/zfs/parameters/zfs_arc_max
sudo echo 3 > /proc/sys/vm/drop_caches

Once you’ve found a size that works for you, you can set the size permanently.

sudo echo "options zfs zfs_arc_max=8589934592" >> /etc/modprobe.d/zfs.conf