Full Disk Encryption (with ZFS root) for FreeBSD 9.x

Boot from freebsd memdisk (usb stick), use „shell“. For the german of us, enter

kbdcontrol -l german.iso

Clean up the disk, maybe use -F

gpart destroy ada0

Create new MBR disk

gpart create -s mbr ada0

Create first partition (/boot), 504 is 4K aligned

gpart add -b 504 -s 20971520 -t freebsd ada0

Create next partition (root)

gpart add -b 20972448 -t freebsd ada0

Set active partition

gpart set -a active -i 1 ada0

Clean up the disk, maybe use -F

gpart destroy ada0

Create new MBR disk

gpart create -s mbr ada0

Create first partition, 504 is 4K aligned

gpart add -b 504 -a 4k -t freebsd ada0

Create BSD type

gpart create -s bsd ada0s1

Create first partition slice (/boot)

gpart add -s 10G -a 4k -t freebsd-zfs ada0s1

Create root partition slice

gpart add -a 4k -t freebsd-zfs ada0s1

Set first partition as active

gpart set -a active -i 1 ada0

Add bootcode to hd

gpart bootcode -b /boot/boot0 ada0

Add boot1 stage

dd if=/boot/zfsboot of=/dev/ada0s1 count=1

Add boot2 stage

dd if=/boot/zfsboot of=/dev/ada0s1a skip=1 seek=1024

Use this if your system is not gpt compatible.

Clean up the disk, maybe use -F

gpart destroy ada0

Create new GPT disk

gpart create -s mbr ada0

Create bootcode, compatibility bootcode for mbr→gpt on mbr

gpart bootcode -b /boot/pmgr ada0

Math: 40*512b = 20kb + 88*512b = 44kb = 64kb/4kb = 16 ⇒ perfect alignment for 4kb drives AND 512b

gpart add -b 40 -s 88 -t freebsd-boot ada0

GPT ZFS boot code on p1

gpart add -p /boot/gptzfsboot -i 1 ada0

/boot partition, uncrypted, still correct alligned, 10485760kb/4096kb=2560

gpart add -s 10G -t freebsd-zfs ada0

And the installation partition, filling all the disk

gpart add -t freebsd-zfs ada0

Now, jump to „Part2“.

Only use Part1.1 OR Part1.3!

The original text is captured from https://www.dan.me.uk/blog/2012/05/06/full-disk-encryption-with-zfs-root-for-freebsd-9-x/comment-page-1/, all credits to him! I made a copy here if the page is down or the link is lost.

Be careful, i think the alignment of 4K does not match here!

—

To follow on from my post about full disk encryption (well almost), this is how to do the same but with a ZFS filesystem. Like the other post, your /boot folder (which contains your kernel and modules) will not be encrypted, but the rest of your filesystem will be.

One disadvantage of this method is that you have to enter a passphrase for EACH disk in your ZFS system each boot. Encryption inside ZFS will appear at some point – but until then this will suffice !

Boot from any FreeBSD 9 install medium (except bootonly), and choose Live CD at the install menu.

For the purposes of this article, I will assume that you’re using 4 disks (da0, da1, da2, da3), a 10GB /boot (this will be mirrored on each of the 4 disks), and the remaining space as a raidz1 (roughly similar to RAID5) ZFS filesystem. The contents will be encrypted with 256-bit AES-XTS encryption with a 4 kb random data partial key and a secondary passphrase (required to type on each boot).

If your CPU supports the AESNI flag, the crypto(4) framework will utilise this too.

First we need to remove any existing GPT or MBR partition tables on each of the disks (ignore any ‘invalid argument’ messages):

  gpart destroy -F da0
  gpart destroy -F da1
  gpart destroy -F da2
  gpart destroy -F da3

Now we need to create a new GPT partition table on each disk:

  gpart create -s gpt da0
  gpart create -s gpt da1
  gpart create -s gpt da2
  gpart create -s gpt da3

We will now create a 64kb boot partition (this contains the boot loader only, so is safe and required to be unencrypted):

  gpart add -s 128 -t freebsd-boot da0
  gpart add -s 128 -t freebsd-boot da1
  gpart add -s 128 -t freebsd-boot da2
  gpart add -s 128 -t freebsd-boot da3

Next, we will create the /boot partition – you can adjust the sizes here if you need, but i’d suggest not shrinking it too much or you’ll get into problems when doing OS upgrades later… Note: this is mirrored not striped across the disks for maximum resilience – so will use 10GB on each disk for 10GB total usable space.

  gpart add -s 10G -t freebsd-zfs da0
  gpart add -s 10G -t freebsd-zfs da1
  gpart add -s 10G -t freebsd-zfs da2
  gpart add -s 10G -t freebsd-zfs da3

Finally, we will assign the remaining space on each disk to the root ZFS partition. This will be encrypted before we build ZFS on top of it.

  gpart add -t freebsd-zfs da0
  gpart add -t freebsd-zfs da1
  gpart add -t freebsd-zfs da2
  gpart add -t freebsd-zfs da3

Now that we’ve created daXp1 (bootloader), daXp2 (/boot partition), daXp3 (root partition) – we need to write the boot loader code to each disk:

  gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
  gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
  gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2
  gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da3

Jump to Part2.

Ok, next we will build a ramdisk to mount on /boot/zfs – this helps us mount things temporarily…

  mdconfig -a -t malloc -s 128m -u 2
  newfs -O2 /dev/md2
  mount /dev/md2 /boot/zfs

Now we will load the modules required for ZFS and encryption:

  kldload opensolaris
  kldload zfs
  kldload geom_eli

Next, we build a ZFS mirror for the /boot partition and mount it temporarily (to house the encryption key) – ignore any mention of unable to mount:

  zpool create bootdir mirror /dev/da0p2 /dev/da1p2 /dev/da2p2 /dev/da3p2
  zpool set bootfs=bootdir bootdir
  mkdir /boot/zfs/bootdir
  zfs set mountpoint=/boot/zfs/bootdir bootdir
  zfs mount bootdir

Now we generate a random 4kb encryption key that will form (along with passphrase) the encryption key for the disk:

  dd if=/dev/random of=/boot/zfs/bootdir/encryption.key bs=4096 count=1

We have everything we need to start encrypting the disks now… Enter your passphrase twice for each init phase and once again for each attach phase below:

  geli init -b -B /boot/zfs/bootdir/da0p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da0p3
  geli init -b -B /boot/zfs/bootdir/da1p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da1p3
  geli init -b -B /boot/zfs/bootdir/da2p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da2p3
  geli init -b -B /boot/zfs/bootdir/da3p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da3p3
  geli attach -k /boot/zfs/bootdir/encryption.key /dev/da0p3
  geli attach -k /boot/zfs/bootdir/encryption.key /dev/da1p3
  geli attach -k /boot/zfs/bootdir/encryption.key /dev/da2p3
  geli attach -k /boot/zfs/bootdir/encryption.key /dev/da3p3

Now that we have encrypted and mounted the partitions, we can build a ZFS root filesystem on top of it like so:

  zpool create zroot raidz1 /dev/da0p3.eli /dev/da1p3.eli /dev/da2p3.eli /dev/da3p3.eli
  zfs set mountpoint=/boot/zfs/zroot zroot
  zfs mount zroot
  zfs unmount bootdir
  mkdir /boot/zfs/zroot/bootdir
  zfs set mountpoint=/boot/zfs/zroot/bootdir bootdir
  zfs mount bootdir

Note we unmounted the old boot mirror and re-mounted it within the root filesystem. This will be used later to copy the kernel and modules into.

Ok, now we create all our ZFS mounts with various options as follows:

  zfs set checksum=fletcher4 zroot
  zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
  chmod 1777 /boot/zfs/zroot/tmp
  zfs create zroot/usr
  zfs create zroot/usr/home
  cd /boot/zfs/zroot; ln -s /usr/home home
  zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
  zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
  zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
  zfs create zroot/var
  zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
  zfs create -o exec=off -o setuid=off zroot/var/db
  zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
  zfs create -o exec=off -o setuid=off zroot/var/empty
  zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
  zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
  zfs create -o exec=off -o setuid=off zroot/var/run
  zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
  chmod 1777 /boot/zfs/zroot/var/tmp

Now we’re ready to install FreeBSD onto the new ZFS partitions. We’re going to install the base, all sources and a generic kernel – this takes some time so please be patient…

  cd /boot/zfs/zroot
  unxz -c /usr/freebsd-dist/base.txz | tar xpf -
  unxz -c /usr/freebsd-dist/kernel.txz | tar xpf -
  unxz -c /usr/freebsd-dist/src.txz | tar xpf -

Now we can set /var/empty to readonly:

  zfs set readonly=on zroot/var/empty

And now we’re ready to chroot into the installed system to setup the configuration:

  chroot /boot/zfs/zroot

Now that the base system and kernel are installed, we can move our /boot folder to it’s final place on the ZFS unencrypted mirror and do a little housekeeping:

  cd /
  mv boot bootdir/
  ln -fs bootdir/boot
  mv bootdir/encryption.key bootdir/boot/
  mv bootdir/*.eli bootdir/boot/

We need to setup an initial /etc/rc.conf which will mount all ZFS filesystems on boot:

  echo ‘zfs_enable=”YES”‘ > /etc/rc.conf
  touch /etc/fstab

And an initial /boot/loader.conf that will load ZFS, encryption and settings for encrypted disks on boot:

  echo ‘vfs.zfs.prefetch_disable=”1″‘ > /boot/loader.conf
  echo ‘vfs.root.mountfrom=”zfs:zroot”‘ >> /boot/loader.conf
  echo ‘zfs_load=”YES”‘ >> /boot/loader.conf
  echo ‘aesni_load=”YES”‘ >> /boot/loader.conf
  echo ‘geom_eli_load=”YES”‘ >> /boot/loader.conf
  echo ‘geli_da0p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
  echo ‘geli_da0p3_keyfile0_type=”da0p3:geli_keyfile0″‘ >> /boot/loader.conf
  echo ‘geli_da0p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf
  echo ‘geli_da1p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
  echo ‘geli_da1p3_keyfile0_type=”da1p3:geli_keyfile0″‘ >> /boot/loader.conf
  echo ‘geli_da1p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf
  echo ‘geli_da2p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
  echo ‘geli_da2p3_keyfile0_type=”da2p3:geli_keyfile0″‘ >> /boot/loader.conf
  echo ‘geli_da2p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf
  echo ‘geli_da3p3_keyfile0_load=”YES”‘ >> /boot/loader.conf
  echo ‘geli_da3p3_keyfile0_type=”da3p3:geli_keyfile0″‘ >> /boot/loader.conf
  echo ‘geli_da3p3_keyfile0_name=”/boot/encryption.key”‘ >> /boot/loader.conf

The above settings tell the OS which encryption keyfile to use for each disk partition.

Now you can set your root password:

  passwd root

And configure your timezone:

  tzsetup

And setup a dummy /etc/mail/aliases file to prevent sendmail warnings:

  cd /etc/mail
  make aliases

Now you can configure any additional settings you require (such as adding new users, configuring networking or setting sshd to run on boot) – when you’re done, we need to exit the chroot:

  exit

Now, we need to make sure the bootloader can read our ZFS pool cache (or it wont mount our ZFS disks on boot):

  cd /boot/zfs
  cp /boot/zfs/zpool.cache /boot/zfs/zroot/boot/zfs/zpool.cache

Finally, we need to unmount all the ZFS filesystems and configure their final mountpoints…

  zfs unmount -a
  zfs set mountpoint=legacy zroot
  zfs set mountpoint=/tmp zroot/tmp
  zfs set mountpoint=/usr zroot/usr
  zfs set mountpoint=/var zroot/var
  zfs set mountpoint=/bootdir bootdir

Now we can ‘reboot’ and remove the media while the computer reboots. Do this as soon as you can.

The computer should reboot into a ZFS-based filesystem, booted from a software RAID array on fully protected disks with all but /boot partition encrypted. Note: it will ask you to enter a passphrase for each disk parition used above (4 times) – you should take care to enter the correct passwords as it will treat any passwords missed as a failed disk (you get 3 attempts at each password)

Once it’s booted, you can login and run sysinstall to configure other options like networking and startup programs (like SSH!)

The only point to note is that when you do an OS upgrade, during the “mergemaster” stage, it will complain that /boot is a symlink not a directory. Simply tell it to ignore/do nothing and it will install the files as normal.

Enjoy!

You have a problem with your zpool.cache. It should be found at /boot/zfs/zpool.cache. If you want to regenerate it you should boot the freebsd live system and choose shell (i am using the memdisk from usb, works fine). Boot from your stick (e.g. ufs:/dev/da0) and do the following (or something equal to that for your system):

It may be needed to set the usb stick read-write, so just remount it:

mount -o rw /

Then:

zpool export zroot
zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache zroot
cp /tmp/zpool.cache /mnt/boot/zfs/

In short: create a zpool.cache and copy it to your bootdir. If your bootdir is on a zpool,

zpool import bootdir
cp /tmp/zpool.cache /bootdir/boot/zfs/

CHECK THE FILE. I had copied it to /bootdir/zfs which is wrong in the setup we did here ;)…

Create a new (secure) backup drive. Attach some usb disk. If this is not a zfs backup drive yet, first destroy the old content. Be sure what you are doing! Your data will be lost!

gpart destroy -F da0

Then, init the geli crypt

geli init -e AES-XTS -l 256 -s 4096 /dev/da0

Attach the geli device

geli attach /dev/da0

Create a new zpool…

zpool create zbackup /dev/da0.eli

List your old snapshots

zfs list -t snapshot

Destroy your old snapshot

zfs destroy -r zroot@oldsnapshot

Create a new recursive snapshot

zfs snapshot -r zroot@newsnapshot

Now, transfer your snapshot to the new disk.

zfs send -R zroot@newsnapshot | zfs receive -Fdvu zbackup

Export (detach) your backup pool.

zpool export zbackup

Detach geli.

geli detach /dev/da0.eli

Next time, we try to do incremental zfs snapshot transfer since last backup ;)

Import backup zpool (don't forget -R for alternate root or your root will be over'mounted)

zpool import -R /zbackup zbackup

Create new snapshot of your root

zfs snapshot -r zroot@20130326-2026

Send the diff to the backup! :)

zfs send -R -i zroot@20130208-1752 zroot@20130326-2026 | zfs receive -Fuv zbackup
(-Fduv?)

I love zfs.

Source: http://people.freebsd.org/~rse/iscsi/iscsi.txt

kldload /boot/kernel/iscsi_initiator.ko
iscontrol -v -d targetaddress=10.0.0.1 initiatorname=`hostname`

Edit /etc/iscsi.conf and add some devices which were discovered

idisk1 {
  authmethod      = CHAP
  chapIName       = user1
  chapSecret      = secret123456
  initiatorname   = server.example.com
  TargetName      = iqn.example.com:idisk1
  TargetAddress   = 10.0.0.1:3260,1
}

iscontrol -c /etc/iscsi.conf -n idisk1
iscontrol -c /etc/iscsi.conf -n idisk2
iscontrol -c /etc/iscsi.conf -n idisk3

New devices like /dev/da0, /dev/da1 should appear.

Killing the processes with term or kill doesn't seem to work. It should work with

kill <pid> -HUP or
killall -HUP iscontrol

But please, first detach a possible geli and do zfs export.

Source of this idea: http://unix.derkeiler.com/Mailing-Lists/FreeBSD/questions/2007-12/msg01469.html

Be warned and think about what you are doing here!

gpool destroy -F da0

zpool create -o version=28 zbck /dev/da0
zfs create -V 1024g zbck/zsvol

with keyfile:

geli init -K /root/p_a.key -s 4096 -l 256 /dev/zvol/zbck/zsvol
geli attach -k /root/p_a.key /dev/zvol/zbck/zsvol

with passphrase:

geli init -s 4096 -l 256 /dev/zvol/zbck/zsvol
geli attach /dev/zvol/zbck/zsvol

zpool create -o version=28 zsbck /dev/zvol/zbck/zsvol.eli

Done!