Ubuntu Home Server Setup Part II
Welcome to Part II of my Ubuntu Home Server build! In Part I, I did a very basic Ubuntu Server install. In this part, I’ll be creating a ZFS pool and volumes to store all my data on.
Other parts of this guide can be found at:
Home Server With Ubuntu
Setup
I’ll be setting up a server with 8 physical drives.
Disk 0: SSD for OS
Disk 1: SSD for ZFS Intent Log (improves write performance)
(read fantastic information about it here: http://nex7.blogspot.com/2013/04/zfs-intent-log.html)
Disk 2: SSD for L2ARC caching (improves read performance)
Disk 3 – 7: HDDs for ZFS Pool (where all my data with be stored)
Quick disclosure: I’m *far* from a ZFS expert. From what I’ve gleaned, this should suffice for home / small business use. If you’re planning something enterprise-grade, find an expert!
Install Ubuntu
Perform a regular Ubuntu server installation, or use an existing server.
SSH Into the server, rather than using the console. You’ll want to be able to copy and paste when you setup the zpool.
Install ZFS
sudo apt install zfsutils-linux
Create the ZPOOL
I’ll be using RAIDZ (which is like RAID-5) to get redundancy on my disks without losing too much usable space.
ZFS offers many other options, like RAID0, 1, 6, etc. Use whichever is appropriate for your workload.
It is very strongly recommended to not use disk names like sdb, sdc, etc. Those might change across reboots.
Many of the articles I’ve read suggest using UUIDs . However, my experience on Ubuntu Server is that these are not assigned to blank disks. Therefore, I will be using disk paths instead.
These are verbose and a bit of a pain to type, but they make sure you know exactly what disk you are referring to should you need to swap drives in the future. They will also not change on reboots.
To see your installed disks run:
ls -lh /dev/disk/by-path
My output looks like
adam@normandy:~$ ls -lh /dev/disk/by-path
total 0
lrwxrwxrwx 1 root root 9 Jul 8 09:06 pci-0000:00:1f.2-ata-5 -> ../../sr0
lrwxrwxrwx 1 root root 9 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 9 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:2:0 -> ../../sdc
lrwxrwxrwx 1 root root 9 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:3:0 -> ../../sdd
lrwxrwxrwx 1 root root 9 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:4:0 -> ../../sde
lrwxrwxrwx 1 root root 9 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:5:0 -> ../../sdf
lrwxrwxrwx 1 root root 9 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:6:0 -> ../../sdg
lrwxrwxrwx 1 root root 9 Jul 8 09:06 pci-0000:02:00.0-scsi-0:0:7:0 -> ../../sdh
I chose to install Linux on my 1st drive (sda). I’ll be using sdb for the ZIL, sdc for L2ARC, and sdd, sde, sdf, sdg, and sdh to for the data pool.
First, I’ll setup the data pool. This is where SSH is handy, since you can copy/paste your paths from above.
In my example below, I’m naming my pool “data.” You can use a different name if you’d like. If your setup is like mine, you’ll create one pool with many volumes in it.
I’m using drives with 4k physical sectors, so I’m adding the option: -o ashift=12
This should increase performance, but at the cost of total storage space. You an remove this option if you don’t think it’s a good fit for you.
sudo zpool create data -o ashift=12 raidz /dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:3:0 /dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:4:0 /dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:5:0 /dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:6:0 /dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:7:0
To confirm this worked, run:
zpool list
You should have something like:
adam@normandy:~$ zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
data 18.1T 238K 18.1T - 0% 0% 1.00x ONLINE -
Next I’ll tell ZFS to use sdb as the ZFS Intent Log
sudo zpool add data log /dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:1:0
Then I’ll tell ZFS to use sdc as the L2ARCH cache
sudo zpool add data cache /dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:2:0
If I run zpool status, I should see my data, ZIL, and cache drives
adam@normandy:/data/download/secure$ zpool status
pool: data
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
pci-0000:02:00.0-scsi-0:0:3:0 ONLINE 0 0 0
pci-0000:02:00.0-scsi-0:0:4:0 ONLINE 0 0 0
pci-0000:02:00.0-scsi-0:0:5:0 ONLINE 0 0 0
pci-0000:02:00.0-scsi-0:0:6:0 ONLINE 0 0 0
pci-0000:02:00.0-scsi-0:0:7:0 ONLINE 0 0 0
logs
pci-0000:02:00.0-scsi-0:0:1:0 ONLINE 0 0 0
cache
pci-0000:02:00.0-scsi-0:0:2:0 ONLINE 0 0 0
errors: No known data errors
Create the Filesystem
Now that the zpool exists, we can create filesystems on top of it.
A pool can have multiple filesystems. I’ll create one for media, and one for virtual machines (because that’s what I need).
sudo zfs create data/media
sudo zfs create data/vm
To confirm it was created correctly run:
zfs list
And it should look something like this:
adam@normandy:~$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
data 210K 14.0T 36.7K /data
data/media 35.1K 14.0T 35.1K /data/media
data/vm 35.1K 14.0T 35.1K /data/vm
I would also suggest the following tweaks. Combined, they increased my zfs throughput 50-100%! They were recommended by https://unicolet.blogspot.com/2013/03/a-not-so-short-guide-to-zfs-on-linux.html and https://www.servethehome.com/the-case-for-using-zfs-compression/ as I searched for solutions to my less-than-stellar zfs performance.
zfs set xattr=sa data/media
zfs set atime=off data/media
zfs set compression=lz4
All of your zfs filesystems are automatically mounted.
adam@normandy:~$ mount
...
data on /data type zfs (rw,xattr,noacl)
data/media on /data/media type zfs (rw,xattr,noacl)
data/vm on /data/vm type zfs (rw,xattr,noacl
You can use them just as you would any mounted filesystem. That’s it!