Record and Reverie

It’s been a while since I posted here. I’ve been busy with a new job, new house, and a bunch of other things. One of these things was setting up my new file server. This is something that’s been in the works for a long time, as can be seen from the various posts on ZFS. I spent a long time researching this, and finally came up with my solution:

I did consider FreeNAS for a really long time. It is essentially a FreeBSD install with most of the administrative work done for you through a web-based GUI. It hit most of my checkboxes in that it supported ZFS, AFP, Bonjour, and a few others. While this is nice, I found it also to be limiting when one wants to stray off the beaten path. I didn’t want to lose ZFS, but I wanted something where I could tinker. I decided to go with FreeBSD.

One of the first things I tried with FreeBSD, as the topic indicates, is to put the root filesystem on ZFS. I followed an excellent tutorial to start, but decided to make a few modifications:

• I wanted to put the root filesystem some place other than the root of the pool. Since my large media collection is going to be stored on the same pool, I wanted the ability to selectively snapshot the boot filesystem, and possibly revert it, without affecting the media partition. So, I put the root filesystem on on a sub filesystem of the pool.
• The trend in disk storage is looking like it will go to 4K sector sizes in the future. ZFS cannot migrate a pool from 512 byte sector sizes to 4K sector sizes without a reformat/restore. I wanted a 4K sector size pool from the start.
• I wanted to use ZFS swap instead of swap partitions.
• I was going to spent several iterations getting things right and didn’t want to type on that blasted 12 year old keyboard the same commands over and over.

So, I accomplished this whole task through a series of scripts, that I would modify. If the script did something wrong, I’d destroy the pool and start over. The end result is a simple setup that I decided to share with the world:

1. Download and burn/image one of the FreeBSD iso images or USB images. I used 8.2, the most recent at the time of this writing, and did the USB image since I had flash drives to spare but no DVD ROM drives.
2. Boot the machine and enter the recovery mode (Fixit).
3. Start the networking:

   mkdir /var/db
   dhclient em0
   ln -s /dist/usr/bin /usr/bin
   

4. Transfer over the script directory:

   scp -r username@10.0.0.10:Desktop/FreeBSD/\* .

   In this case, 10.0.0.10 is my laptop with sshd running.

5. Start sshd:

   ./FreeBSDSSHSetup.sh

6. ssh into the machine from my laptop and execute:

   ./FreeBSDInstall.sh

7. Reboot the server and enjoy.

The FreeBSD directory is available for those who would like to copy my efforts, though they will likely need some modification to suit your needs. The gptzfsboot, pmbr, and zfsloader files came from this site which are modified versions that work with 4K drives. The versions available in 8.2 do not work with 4K sector sizes, but future versions will. My install script

• Partitions ad8, 10, 12, 14, 16, 18 to contain the boot loader and zpool information (6 SATA drives),
• Loads kernel modules and sets up a gnop drive with 4K sectors,
• Creates a raidz2 pool using the gnop drive as one of the drives forcing 4K sectors, exports the pool,
• Destroys the gnop drive, re-import the pool, which had been forced to 4K sectors,
• Sets up root filesystem on zroot/root (zroot is the root of the pool, root is the root filesystem, as discussed above), and
• Performs a standard zroot install, except no swap partition.

I actually added the swap later, which is simply:

zfs create -V 2G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
zfs set checksum=off zroot/swap

And that’s it. I have my 8TB raid running, with 2 drive redundancy, booting off the raid. Setting up AFP and the like requires more tinkering than with FreeNAS, but the instructions are easy to find and would be redundant to list here.

Tags: Servers, ZFS