Recently I wanted to place my server in a (shared) co-location facility, unable to use a private network for the IPMI module in my supermicro server. The original plan was to use a local iptables firewall on the IPMI module itself, as it runs an ARM linux implementation. Unfortunately the iptables install included with the firmware is severely broken, and can’t be used:

# iptables -I INPUT -p tcp -s x.x.x.x -j ACCEPT
iptables v1.2.11: Couldn't load target `standard':/lib/iptables/libipt_standard.so: cannot open shared object file: No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
#

Crosscompiling iptables is an option, but would also mean extensive modification of the flash image, as there is no (easy) way to transfer files to the BMC, and flash storage space is limited.

The customized linux install also doesn’t allow for safe shutdown of unneeded services, and this would still expose the module to the internet. This post at serverfault shows that I’m not the only one looking for a solution for this.

In the end, a friend (thanks mtp!) came up with a simple solution to at least limit access to the public IPMI IP – simply removing the default gateway, and setting routes to some static IP’s that are allowed to connect. In combination with a small enough netmask, this is secure enough for my purposes.

Bookmark It

The new box is quite fast, running on a core i7 CPU, with 12Gb of RAM and 4 1.5Tb disks in raid10,f2 – so i thought it would be nice to spend some of that power on encrypting the volume groups my virtual machines will be running on.  Before moving it to the datacenter, I’m playing around with it, and ran some benchmarks on dm-crypt, to get a feeling about the performance loss. Oh, and I’m running Ubuntu Jaunty.

First I ran a benchmark on a 500Gb encrypted volume. The disks have been configured in linux software raid10,f2, using a 256Kb chunk size. I used cryptsetup on the md device, to create an encrypted dm-crypt device, attached it, and marked it as a physical volume for LVM. I’ve left out the output from the commands.

root@cipher:~# cryptsetup luksFormat /dev/md3 -c aes -s 256 -h sha256
root@cipher:~# cryptsetup luksOpen /dev/md3 crypted
root@cipher:~# pvcreate /dev/mapper/crypted

After this, I created a volume group using the freshly created pv, and activated a 500Gb logical volume, created an xfs filesystem and mounted it:

root@cipher:~# vgcreate crypted_vg /dev/mapper/crypted
root@cipher:~# lvcreate -n test_lv -L500G crypted_vg
root@cipher:~# mkfs.xfs /dev/crypted_vg/test_lv
root@cipher:~# mount /dev/crypted_vg/test_lv /mnt/

On the mounted filesystem, I ran bonnie++ to benchmark performance of dm-crypt. During the benchmark cpu load of 6 of the 8 cores were mostly at 0. The two remaining cores were taking up 80%+ load, by the bonnie++ and the kcryptd processes:

root@cipher:~# bonnie++ -d . -s 24000 -u rvdm:rvdm
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
cipher       24000M 73629  97 62418   6 40186   5 55728  79 98903   9 477.3   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1516   5 +++++ +++   990   2  1193   3 +++++ +++   866   2
cipher,24000M,73629,97,62418,6,40186,5,55728,79,98903,9,477.3,1,16,1516,5,+++++,+++,990,2,1193,3,+++++,+++,866,2

What’s missing in the output above are the commands to create a directory for bonnie to work in, and to make it writable for the right user.

After the benchmark, I wiped the LV, VG and PV, killed the dm-crypt mapping with cryptsetup, and was left with a new usable md device. To keep the tests as similar as possible, I’ve created a new vg, containing the entire md device. After creating a new 500Gb LV, mounting and xfs-ing it, I ran the second benchmark:

root@cipher:~# bonnie++ -d . -s 24000 -u rvdm:rvdm
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
cipher       24000M 82330  98 157554  17 55584   8 67361  82 226305  22 595.0   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1154   2 +++++ +++   925   2  1241   3 +++++ +++  1186 3
cipher,24000M,82330,98,157554,17,55584,8,67361,82,226305,22,595.0,1,16,1154,2,+++++,+++,925,2,1241,3,+++++,+++,1186,3

It’s clear from the output that the overhead from using dm-crypt is quite big; up to 100% in some cases. Also, CPU usage increases significantly. I will be using dm-crypt for crypting some of the virtual machines, but I’ll be sure to only use it when it’s worth the performance penalty..

Note: I realise the bonnie++ output gets chopped off. To be honest, I’m too lazy to fix that

Bookmark It