The new box is quite fast, running on a core i7 CPU, with 12Gb of RAM and 4 1.5Tb disks in raid10,f2 - so i thought it would be nice to spend some of that power on encrypting the volume groups my virtual machines will be running on.  Before moving it to the datacenter, I’m playing around with it, and ran some benchmarks on dm-crypt, to get a feeling about the performance loss. Oh, and I’m running Ubuntu Jaunty.

First I ran a benchmark on a 500Gb encrypted volume. The disks have been configured in linux software raid10,f2, using a 256Kb chunk size. I used cryptsetup on the md device, to create an encrypted dm-crypt device, attached it, and marked it as a physical volume for LVM. I’ve left out the output from the commands.

root@cipher:~# cryptsetup luksFormat /dev/md3 -c aes -s 256 -h sha256
root@cipher:~# cryptsetup luksOpen /dev/md3 crypted
root@cipher:~# pvcreate /dev/mapper/crypted

After this, I created a volume group using the freshly created pv, and activated a 500Gb logical volume, created an xfs filesystem and mounted it:

root@cipher:~# vgcreate crypted_vg /dev/mapper/crypted
root@cipher:~# lvcreate -n test_lv -L500G crypted_vg
root@cipher:~# mkfs.xfs /dev/crypted_vg/test_lv
root@cipher:~# mount /dev/crypted_vg/test_lv /mnt/

On the mounted filesystem, I ran bonnie++ to benchmark performance of dm-crypt. During the benchmark cpu load of 6 of the 8 cores were mostly at 0. The two remaining cores were taking up 80%+ load, by the bonnie++ and the kcryptd processes:

root@cipher:~# bonnie++ -d . -s 24000 -u rvdm:rvdm
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
cipher       24000M 73629  97 62418   6 40186   5 55728  79 98903   9 477.3   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1516   5 +++++ +++   990   2  1193   3 +++++ +++   866   2
cipher,24000M,73629,97,62418,6,40186,5,55728,79,98903,9,477.3,1,16,1516,5,+++++,+++,990,2,1193,3,+++++,+++,866,2

What’s missing in the output above are the commands to create a directory for bonnie to work in, and to make it writable for the right user.

After the benchmark, I wiped the LV, VG and PV, killed the dm-crypt mapping with cryptsetup, and was left with a new usable md device. To keep the tests as similar as possible, I’ve created a new vg, containing the entire md device. After creating a new 500Gb LV, mounting and xfs-ing it, I ran the second benchmark:

root@cipher:~# bonnie++ -d . -s 24000 -u rvdm:rvdm
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
cipher       24000M 82330  98 157554  17 55584   8 67361  82 226305  22 595.0   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1154   2 +++++ +++   925   2  1241   3 +++++ +++  1186 3
cipher,24000M,82330,98,157554,17,55584,8,67361,82,226305,22,595.0,1,16,1154,2,+++++,+++,925,2,1241,3,+++++,+++,1186,3

It’s clear from the output that the overhead from using dm-crypt is quite big; up to 100% in some cases. Also, CPU usage increases significantly. I will be using dm-crypt for crypting some of the virtual machines, but I’ll be sure to only use it when it’s worth the performance penalty..

Note: I realise the bonnie++ output gets chopped off. To be honest, I’m too lazy to fix that