System encryption with passphrase protected usb key

Introduction

The past month a lot of security issues have popped up in my work and personal life most prominent of which was the DAO hack. All these have lead me to think a lot about security and prompted me to learn and utilize new methods I had never used before. All in all I can say that the events of the past few months have made me quite a bit paranoid.

As a result of that I started reading about full system encryption of the root file system and wanted to apply it in a newly installed ArchLinux system. The Archwiki has a very extensive guide on system encryption which can function as a reference to anyone wanting to do the same. One scenario not covered by that guide is how you can encrypt the system with a password protected keyfile located on a USB stick. This is the scenario we are going to cover in this guide.

Encrypting an entire system

We will be using the Device Mapper crypt module in order to encrypt block devices using the Linux Kernel’s crypto API. We assume a very simple setup with 2 partitions. The first partition will be the boot parition and the other is going to be the root partition which we will encrypt.

luks-logo.png

We will use the Logical Volume Manager (LVM) in order to have a flexible root parition logical volume on top of a LUKS encrypted partition. We will essentially be using the LVM on Luks methodology of the Archwiki but with a big change that will allow us to have a passphrase protected keyfile in a USB stick. I will assume you are attempting to install an ArchLinux machine following the wiki and explain the different/additional steps that need to be taken in order to achieve the encryption.

Preparing the disk

For extra safety you can securely wipe the entire disk using Luks as can be seen here. After that is done and depending on whether you have an UEFI motherboard or not create an UEFI or an MBR boot partition. Also create a root partition. To do so you can use parted. Once you have created the partitions you will need to format them. In the examples below we will assume an UEFI partition and that your drive is /dev/sda. Adjust the commands depending on your drive name.

Create the 2 partitions.

(parted) mkpart ESP fat32 1MiB 513MiB
(parted) set 1 boot on
(parted) mkpart primary ext4 513MiB 100%

Format them accordingly. Here we are creating an UEFI boot partition and an ext4 root partition.

mkfs.fat -F32 /dev/sda1
mkfs.ext4 /dev/sda2

Now we can use cryptsetup in order to create the encrypted container on top of the root partition. You can choose a lot of different options for the encryption like the hash algorithm used for key derivation, or the number of iterations to be used for passphrase processing.

cryptsetup luksFormat /dev/sda2

After that you will have to open the container.

cryptsetup open --type luks /dev/sda2 lvm

The decrypted container is now available at /dev/mapper/lvm.

Preparing the logical volumes

Now we are going to create a physical volume on top of the opened LUKS container.

pvcreate /dev/mapper/lvm

Subsequently create a volume group and create the root logical volume for it. You should change VolName with the name you would like your volume group to have.

vgcreate VolName /dev/mapper/lvm
lvcreate -l 100%FREE VolName -n root

Finally you should format the logical volume and mount it.

mkfs.ext4 /dev/mapper/VolName-root
mount /dev/mapper/VolName-root /mnt

Preparing and configuring the boot partition

For our example we have an UEFI boot partition on /dev/sda1. You can always adjust this guide to any other type of boot partition your system may have. Mount the boot partition and continue with the installation procedure up to the point where you deal with initramfs.

mkdir -p /mnt/boot
mount /dev/sda1 /mnt/boot

The bootloader loads the kernel and the initramfs scripts from the boot partition. The new iteration of initramfs is called mkinitcpio and is essentially a very small early userspace environment which loads various kernel modules and sets up all necessary things before handing control over to init.

We can use the already existing encrypt and lvm2 hooks of mkinitcpio. To enable them edit /etc/mkinitcpio.conf and add them in the HOOKS line. They should be added before the filesystems hook. so in essence it should look like this:

HOOKS="... encrypt lvm2 ... filesystems ..."

Run the following in order to create the updated initcpio scripts.

mkinitcpio -p linux

Now you should figure out the UUID of your physical device. You can do so by running:

blkid /dev/sda2

/dev/sda2: UUID="8197c881-160c-465c-a15c-96b59as26157" TYPE="crypto_LUKS" PARTUUID="fe8d1a97-d10b-43c9-a748-972b0af8a09b"

Replace /dev/sda2 with the partition of your root filesystem. Once that is done then you can edit your bootloader to add the following kernel arguments, which will be picked up by the encrypt initcpio module and decrypt your device at boot. If for example you are using systemd-boot then you should edit /boot/loader/entries/entry.conf like so:

title Arch Linux
linux /vmlinuz-linux
initrd /intel-ucode.img
initrd /initramfs-linux.img

options cryptdevice=UUID=8197c881-160c-465c-a15c-96b59as26157:VolName root=/dev/mapper/VolName-root quiet rw

Remember to change VolName to the name of the volume group you created.

Finally make sure to properly populate /etc/fstab so that after decryption the logical root partition is properly mounted at boot:

# /etc/fstab: static file system information
#
# <file system> <dir>   <type>  <options>       <dump>  <pass>
# /dev/mapper/VolName-root
UUID=8197c881-160c-465c-a15c-96b59as26157       /               ext4            rw,relatime,data=ordered        0 1

# /dev/sda1
UUID=0C02-13D4          /boot           vfat            rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro    0 2
/dev/mapper/VolName-root /mnt ext4 defaults,errors=remount-ro 0 2

Once that is done then you can simply reboot and you will be prompted for the passphrase to unlock your encrypted root partition at boot. Congratulations you have now encrypted your root partition!

Using an encrypted USB stick

Having an encrypted partition that decrypts simply by typing a password is nice but if a malicious actor learns of your password then he can without any trouble decrypt your root partition and gain access to your data. A method by which you can make this more difficult is to use a 2-factor authentication by having a USB stick with an encrypted passphrase which will act as a key to decrypt the root partition. As such a malicious actor would need physical access to both your USB stick containing the keyfile and to the password that unlocks it.

prompt.png

Create the keyfile

The simplest way to achieve the 2fa effect is to again use LUKS in order to create an encrypted keyfile inside the USB stick. Let’s assume now that the usb stick is located at /dev/sdc1 and that it uuid is 1193c881-267f-134f-123a-12b34as56357.

We now have to mount the usb stick, create a random keyfile in it and encrypt it with Luks. You can change OurKey with whatever name you would like the decrypted Luks volume of the key to have. Also you should replace /dev/sdb2 with the root partition you want to encrypt with that key.

mkdir -p /mnt/stick
mount /dev/sdc1/ /mnt/stick
dd if=/dev/zero of=/mnt/stick/key.luks count=2057 
cryptsetup --align-payload=1 luksFormat /mnt/stick/key.luks 
cryptsetup luksOpen /mnt/stick/key.luks OurKey
dd if=/dev/urandom of=/dev/mapper/OurKey
cryptsetup luksAddKey /dev/sdb2 /dev/mapper/OurKey

Create an initcpio hook

We will create an initcpio hook so that the bootloader can prompt us for the passphrase and decrypt the encrypted partition during the boot process. The beauty of initcpio is that it’s all simply shell scripts and as such they are quite easy to understand.

First of all you should decide on a name for your hook. I called mine lefcrypt but you can use whichever name you want. To create a hook you need to create 2 files under 2 different directories.

Create /usr/lib/initcpio/install/lefcrypt:

#!/bin/bash

build() {
    # Copied from the encrypt hook install script
    local mod

    add_module loop
    add_module dm-crypt
    if [[ $CRYPTO_MODULES ]]; then
        for mod in $CRYPTO_MODULES; do
            add_module "$mod"
        done
    else
        add_all_modules '/crypto/'
    fi

    add_binary "cryptsetup"
    add_binary "dmsetup"
    add_file "/usr/lib/udev/rules.d/10-dm.rules"
    add_file "/usr/lib/udev/rules.d/13-dm-disk.rules"
    add_file "/usr/lib/udev/rules.d/95-dm-notify.rules"
    add_file "/usr/lib/initcpio/udev/11-dm-initramfs.rules" "/usr/lib/udev/rules.d/11-dm-initramfs.rules"


    add_runscript
}

help() {
    cat <<HELPEOF
This is our custom hook for decrypting a keyfile from a USB stick.
HELPEOF
}

The above essentially prepares the script, states the required modules for the script to run and also provides a help docstring which will appear if you typed mkinitcpio -H lefcrypt.

Also create /usr/lib/initcpio/hooks/lefcrypt:

#!/usr/bin/bash

run_hook() {
    modprobe -a -q dm-crypt >/dev/null 2>&1
    modprobe loop
    [ "${quiet}" = "y" ] && CSQUIET=">/dev/null"

cat << "EOF"

                                          ___.-----.___
                                       .-'. . . . . . .`-.
                                     .'  ` . . . . . .  ' `.
                                   .' ` ` . . . . . . '  '  `.
 .----------------------..--.     / `` ` ` ` _.---._ ' ' ' '  \
|  ,                 `--||--.\   / ` ` ` `.-'_.---._`-.' ' ' ' \
|  `                 ,--||--'/  [\ ` ` `.'.-' ..| ..`-.`.' ' '' \
`-----------------------`'--'  _[/ ` ``/.' \ .. |..  / `.\' ' '  \
   |       |                  / / ` ` // `` \  .| . /' ' \\' ' _  \
   |        \__.---------.___/|\ - ` // `. ` \.---./'' .' \\ - _ - |
   |    _.--' ` `` ` ` ` `  /-| \ -  ` = -`. / ___ \ .'- = ||- = - |
   || .'   `` ``  ``` `` `.' -|||::= `---.__/_/   \ \  _.-'||= _ = |
   \ /  ` ``  ` ` ` `.---' [] | |::  ||[_]    ___  \|-'- = ||- _ - |
   |(O]================-------| |    ||[_]   (O__) ||------||- - - |
   / \ ' ''    ''    `---. [] | |::  ||[_]____     /|-._ = ||- _ - |
   || `._ ' ''  ' ''    ' `. -|||::= ,---'  \ \___/ / - `-.||= _ = |
   |     `--.__ ' ' '''' '__\-| / -  , = -.' \     / `. =  ||- = - |
   |        /  `---------'   \|/ _ ' \\ .''' /`---'\`  `. // - _ - |
   |       |                  \_\ ' ' \\ '  /. .|.. \``  // ` `   /
 .--------------------..--.     [\'' ' \`.'/ .. | .. \ .'/ `  ` `/
|  ,               `--||--.\    [/' ' ' `.`-._ .|. _.-'.' `  ` `/
|  `               ,--||--'/     \ ' ' '  `-._`---'_.-' ` ` ` `/
 `--------------------`'--'       \' ' ' '   .`---'.   ` `  ` /
                                   `.'' '' '. . . . `` `  ` .'
                                     `.'' '. . . . . . ` `.'
                                       `-.___ . . . ___.-'
                                             `-----'

Provide the captain's command authorization code for the USB stick:
EOF

    #obtain the key
    mkdir -p /mnt/usbstick
    resolved=$(resolve_device  /dev/disk/by-uuid/1193c881-267f-134f-123a-12b34as56357)
    mount -t ext4 "$resolved" /mnt/usbstick
    cryptsetup -T 5 luksOpen /mnt/usbstick/key.luks OurKey

    #unlock the root partition
    cryptsetup --key-file /dev/mapper/OurKey luksOpen /dev/disk/by-uuid/8197c881-160c-465c-a15c-96b59as26157 lvm

    #clean up the key
    cryptsetup luksClose OurKey
}

The above is a really simple script which uses the UUID of the usb stick in order to find the key and prompt the user to decrypt it. I could not resist putting a Star Trek reference at the prompt. Apologies :). You should change the UUIDs with your drive’s actual UUIDs and also OurKey with the name you provided for your LUKS encrypted key partition.

In order to use this hook you have to include it in the /etc/mkinitcpio.conf and put it instead of encrypt like so:

HOOKS="... lefcrypt lvm2 ... filesystems ..."

Finally you should create the new initramfs image by issuing mkinitcpio:

mkinitcpio -p linux

==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 4.6.3-1-ARCH
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [lefcrypt]
  -> Running build hook: [lvm2]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 4.6.3-1-ARCH
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
  -> Running build hook: [lefcrypt]
  -> Running build hook: [lvm2]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful

Now once you reboot you can simply input the USB stick and be prompted for the passphrase. Congratulations you now have 2fa in the encryption of your root partition!

Removing the simple passphrase decryption

If you followed this guide step by step then you will still have the option to decrypt the system using the simple passphrase key you created in the first section. If you have confirmed that the USB stick decryption works perfectly then you can safely remove the simple passphrase key.

First check how many keys are used by the encrypted root partition:

cryptsetup luksDump /dev/sda2 | grep BLED

Key Slot 0: ENABLED
Key Slot 1: ENABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

You should only see 2. Key slot 0 should be the very first simple passphrase key and Key slot 1 the one we just created on the USB stick. You can generally query a lot of information about the keys as can be seen in the related wiki.

If you have used different passphrases for the USB encrypted stick and for the normal passphrase key then it’s quite easy to remove the key without even specifying the slot.

cryptsetup luksRemoveKey /dev/sda2
Enter LUKS passphrase to be deleted:

If you have used the same password then you have to also specify the slot when removing the key.

cryptsetup luksRemoveKey /dev/sda2 0
Enter any remaining LUKS passphrase:

After this action is complete then the only way to decrypt your root filesystem and gain access to your machine would be by using the key located inside the USB stick.

Conclusion

We have presented a way to use a password protected encrypted key located in a usb stick to decrypt the root filesystem of your computer. This provides us with a lot of security and the ability to perform a 2-factor authentication when booting the system in order to protect our data if the computer ever falls into the hands of a malicious actor.

There are disadvantages to this approach. If a malicious actor ever gains access to both your key and your keyfile it is Game Over. At the same time the boot partition needs to be unencrypted to perform the bootloading process. This can introduce vulnerabilities which an attacker could take advantage of. There are some methods that can be followed in order to secure the unencrypted boot partition, such as having it located on an external drive etc.

The presented method is not perfect, but it provides superior security in comparison to a totally unencrypted system and provides a nice basis from which the curious reader can explore many other methods of disk encryption. I hope you enjoyed this post and please don’t hesitate to leave some comments explaining how you use encryption to protect your data and what kind of improvements you believe can be made in the method presented here.

Developing for Snappy Ubuntu from any distro using LXC

Introduction

Lately I have been working a lot with Snappy Ubuntu Core. It’s a very nice, minimal, transactionally updated Ubuntu for IoT devices. Being an ArchLinux user I got annoyed by the fact that I need to have an Ubuntu host for developing. I started developing from an Ubuntu VM running inside my ArchLinux host.

As many of you probably already know working with Virtual Machines is slow, cumbersome and a pain since among others you have to recreate your host’s development environment.

A much more attractive alternative is an LXC container which is what this post will address. Our host system will be an ArchLinux and we will setup an Ubuntu 15.10 LXC to develop for Snappy Ubuntu core in. LXC creation for other Linux distros should be similar so the post can definitely be followed up to a point. Note that this guide is also largely based on the relating post on LXC in the ArchLinux wiki.

lxc.png

Setting up the host

Getting required packages

We will need lxc, arch-install-scripts and bridge-utils.

user@host:$ sudo pacman -S lxc arch-install-scripts bridge-utils

To ensure that your lxc is appropriately configured run:

user@host:$ lxc-checkconfig

--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: missing
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
Bridges: enabled
Advanced netfilter: enabled
CONFIG_NF_NAT_IPV4: enabled
CONFIG_NF_NAT_IPV6: enabled
CONFIG_IP_NF_TARGET_MASQUERADE: enabled
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

If you see output like the above then your lxc setup should be okay. Notice that User namespace is missing. That should only be the case if you run on ArchLinux since here it’s not currently supported at the moment of this post’s writing. That will mean we have to always run all lxc commands with root privileges.

Create the Container

In order to create our container we base it off the download template which gives us the option to choose from different OS/ARCH combinations. Below you can see the appropriate command and the selections you need to make in order to obtain Ubuntu Willy (15.10) for the amd64 architecture.

user@host:$ sudo lxc-create -t download -n snappydev

Setting up the GPG keyring
Downloading the image index

---
DIST    RELEASE ARCH    VARIANT BUILD
---
centos  6       amd64   default 20160129_02:16
centos  6       i386    default 20160129_02:16
centos  7       amd64   default 20160129_02:16
debian  jessie  amd64   default 20160128_22:42
debian  jessie  armel   default 20160111_22:42
debian  jessie  armhf   default 20160111_22:42
debian  jessie  i386    default 20160128_22:42
debian  sid     amd64   default 20160128_22:42
debian  sid     armel   default 20160111_22:42
debian  sid     armhf   default 20160111_22:42
debian  sid     i386    default 20160128_22:42
debian  squeeze amd64   default 20160128_22:42
debian  squeeze armel   default 20150826_22:42
debian  squeeze i386    default 20160128_22:42
debian  wheezy  amd64   default 20160128_22:42
debian  wheezy  armel   default 20160111_22:42
debian  wheezy  armhf   default 20160111_22:42
debian  wheezy  i386    default 20160128_22:42
fedora  21      amd64   default 20160129_01:27
fedora  21      armhf   default 20160112_01:27
fedora  21      i386    default 20160129_01:27
fedora  22      amd64   default 20160129_01:27
fedora  22      armhf   default 20160112_01:27
fedora  22      i386    default 20160129_01:27
gentoo  current amd64   default 20160129_14:12
gentoo  current armhf   default 20160111_14:12
gentoo  current i386    default 20160129_14:12
opensuse        12.3    amd64   default 20160129_00:53
opensuse        12.3    i386    default 20160129_00:53
oracle  6.5     amd64   default 20160129_11:40
oracle  6.5     i386    default 20160129_11:40
plamo   5.x     amd64   default 20160129_21:36
plamo   5.x     i386    default 20160129_21:36
ubuntu  precise amd64   default 20160129_03:49
ubuntu  precise armel   default 20160112_03:49
ubuntu  precise armhf   default 20160112_03:49
ubuntu  precise i386    default 20160129_03:49
ubuntu  trusty  amd64   default 20160129_03:49
ubuntu  trusty  arm64   default 20150604_03:49
ubuntu  trusty  armhf   default 20160112_03:49
ubuntu  trusty  i386    default 20160129_03:49
ubuntu  trusty  ppc64el default 20160129_03:49
ubuntu  vivid   amd64   default 20160129_03:49
ubuntu  vivid   arm64   default 20150604_03:49
ubuntu  vivid   armhf   default 20160112_03:49
ubuntu  vivid   i386    default 20160129_03:49
ubuntu  vivid   ppc64el default 20160129_03:49
ubuntu  wily    amd64   default 20160129_03:49
ubuntu  wily    arm64   default 20150604_03:49
ubuntu  wily    armhf   default 20160112_03:49
ubuntu  wily    i386    default 20160129_03:49
ubuntu  wily    ppc64el default 20160129_03:49
ubuntu  xenial  amd64   default 20160129_03:49
ubuntu  xenial  armhf   default 20160112_03:49
ubuntu  xenial  i386    default 20160129_03:49
---

Distribution: ubuntu
Release: wily
Architecture: amd64

Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created an Ubuntu container (release=wily, arch=amd64, variant=default)

To enable sshd, run: apt-get install openssh-server

For security reason, container images ship without user accounts
and without a root password.

Use lxc-attach or chroot directly into the rootfs to set a root password
or create user accounts.

As you can see from above type: ubuntu, wily, amd64 and wait until the proper image and root file system has been downloaded. Once done, your container will have been created, but is not yet running.

Setting up networking

Before we proceed any further we will need to address the networking issue. Our container won’t have any networking capability if we leave it just like that so let’s get to work.

We will need to setup a network bridge between the host and the container. There are two main ways to accomplish this and they are quite different. One is with netctl which I would recommend only if you are using a wired connection and the other is with systemd-networkd if you are using a wifi connection. The second way can also be used for a wired connection, if you are already using systemd-networkd.

Wired Connection

For a wired connection you can use netctl so make sure you have installed the appropriate package.

user@host:$ sudo pacman -S netctl
Setup network bridge with netctl

To create a bridge with netctl create the file /etc/netctl/lxcbridge and paste the following into it:

Description="LXC bridge"
Interface=br0
Connection=bridge
BindsToInterfaces=('eno1')
IP=static
Address=10.0.2.1/24
FwdDelay=0

In the BindsToInterfaces use the interface you would like to bind to. At Address you can just leave the same as what I have or use any valid address as long as you remember it, since it’s going to be needed to configure the LXC itself.

Now let’s switch to and start the bridge:

user@host:$ sudo netctl switch-to lxcbridge
user@host:$ sudo netctl start lxcbridge

If you do ifconfig right now you will be able to see that the bridge interface is up and running.

Wifi Connection

If you are using wifi then theoretically you can still try to use netctl as before to make the bridge work. There is even a guide you can use. Personally I had trouble making it work and by researching the topic it seems that it’s easier to achieve a network bridge for wifi if you are using systemd-networkd. If your systemd version is greater or equal to 210 then it should come with networkd built-in. Thankfully at the time of this post’s writing the systemd version of Archlinux is 228.

Also make sure you have the packages wpa_supplicant, dnsmasq

user@host:$ sudo pacman -S wpa_supplicant dnsmasq
Setup wireless network with networkd-systemd

I will assume you have never used networkd-systemd before. If you already have been using it succesfully then you can skip to the next section.

First of all disable any other service that was managing your wireless connection, like say Network Manager.

user@host:$ sudo systemctl disable NetworkManager

Then we can go ahead and start/enable the relevant systemd services:

user@host:$ sudo systemctl enable systemd-networkd.service
user@host:$ sudo systemctl start systemd-networkd.service
user@host:$ sudo systemctl enable systemd-resolved.service
user@host:$ sudo systemctl start systemd-resolved.service

For compatibility with the old /etc/resolve.conf, delete the old file and symlink the systemd equivalent in its place.

user@host:$ sudo rm -rf /etc/resolv.conf
user@host:$ ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

Now add the network configuration under /etc/systemd/network/. Create a file with any name ending in the .network suffix. I like naming the file from the interface it’s made for. So for a wireless interface named wlp4s0 I create the file /etc/systemd/network/wlp4s0.network.

[Match]
Name=wlp4s0

[Network]
DHCP=yes

[DHCP]
RouteMetric=20

The important parts are the Name which should match the interface name, and the fact that we want DHCP. If you want a static IP this is where you would set it up. The RouteMetric is to give priority to a potential wired connection over the wireless one.

Now we have to enable the wpa_supplicant service for our wireless interface:

user@host:$ sudo systemctl enable wpa_supplicant@wlp4s0.service

Replace wlp4s0 with your interface name. This should create a configuration file: /etc/wpa_supplicant/wpa_supplicant-wlp4s0.conf. Make sure it contains the following in the beginning:

ctrl_interface=/run/wpa_supplicant
update_config=1
fast_reauth=1
ap_scan=1

After that the network SSID-passphrase combinations should follow. If you don’t know how to use wpa_supplicant to connect to a wireless connection then this article will certainly be useful to you.

Setup a network bridge with networkd-systemd

Now is time to create the actual bridge using networkd-systemd. Create the file /etc/systemd/network/lxcbr0.netdev and paste the following in there:

[NetDev]
Name=lxcbr0
Kind=bridge

The name can be anything but it will be what identifies this bridge. Also create /etc/systemd/network/lxcbr0.network and put the following in it making sure the name matches the one above.

[Match]
Name=lxcbr0

[Network]
IPForward=yes
IPMasquerade=yes

Subsequently enable and start the lxc-net service which will help us in setting up the bridge.

user@host:$ sudo systemctl enable lxc-net
user@host:$ sudo systemctl start lxc-net

Finally make sure to check that the contents of /etc/default/lxc closely match the following:

# LXC_AUTO - whether or not to start containers at boot
LXC_AUTO="true"

# BOOTGROUPS - What groups should start on bootup?
#    Comma separated list of groups.
#    Leading comma, trailing comma or embedded double
#    comma indicates when the NULL group should be run.
# Example (default): boot the onboot group first then the NULL group
BOOTGROUPS="onboot,"

# SHUTDOWNDELAY - Wait time for a container to shut down.
#    Container shutdown can result in lengthy system
#    shutdown times.  Even 5 seconds per container can be
#    too long.
SHUTDOWNDELAY=5

# OPTIONS can be used for anything else.
#    If you want to boot everything then
#    options can be "-a" or "-a -A".
OPTIONS=

# STOPOPTS are stop options.  The can be used for anything else to stop.
#    If you want to kill containers fast, use -k
STOPOPTS="-a -A -s"

USE_LXC_BRIDGE="true"  # overridden in lxc-net

[ ! -f /etc/default/lxc-net ] || . /etc/default/lxc-net

Especially assert that USE_LXC_BRIDGE= is set to true.

Miscellaneous network requirements

Since we are now using a network bridge we have to make sure that the kernel parameter that allows packet forwarding between interfaces is enabled.

user@host:$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0
user@host:$ sudo sysctl net.-w ipv4.ip_forward=1
net.ipv4.ip_forward = 1

As shown above, query the kernel value with sysctl and if it is 0 then enable. Finally use the following command to enable the host to perform NAT using iptables:

user@host:$ sudo iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE

Make sure to replace eno1 with the interface that your host is actually using.

Configuring the Container

The configuration file of the container is located at /var/lib/lxc/snappydev/config where snappydev is the name you gave to your LXC container. It should be almost empty.

General and networking attributes of the config file

Below is a snapshot of the config file, where networking has also been taken care of:

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)

# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.arch = x86_64

# Container specific configuration
lxc.rootfs = /var/lib/lxc/snappydev/rootfs
lxc.utsname = snappydev

# Network configuration
lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
# lxc.network.ipv4 = 192.168.100.10/24
# lxc.network.ipv4.gateway = 192.168.100.1
lxc.network.name = eth0

Up until lxc.utsname it is auto-generated by the template, so no need to worry about it. After that comes the network configuration. You can copy it as is if you have configured the lxcbridge with netctl the same way I did.

If you have setup the bridge with a static IP then you can add the ip and the gateway as shown above with lxc.network.ipv4 and lxc.network.ipv4.gateway set to the same IP addresses as your had in your bridge configuration setup (/etc/netctl/lxcbridge for netctl and /etc/systemd/network/lxcbr0.network for systemd-networkd). I have them commented out since the guide assumes you use lxc-net or netcl with DHCP in which case you don’t need to specify IPs. Also make sure that lxc.network.link is the same as the Name field in the bridge configuration file.

ArchLinux and systemd Host specific configuration

If you are on an ArchLinux host like I am then you definitely need to add a few more entries at the end of the LXC configuration file:

# By default, lxc symlinks /dev/kmsg to /dev/console, this leads to journald running
# at 100% CPU usage all the time. To prevent the symlink we use this:
lxc.kmsg = 0
# Maintain devpts consistency
lxc.pts = 1024
## required for systemd
lxc.autodev = 1
lxc.hook.autodev=/var/lib/lxc/snappydev/autodev

Setting lxc.kmsg to 0 is required as the comment also explains in order to avoid a symlink that will lead to 100% CPU utilization.

Setting lxc.pts to 1024 ensures that devpts is mounted as a new instance and the container doesn’t just get the host’s instance which would have pretty negative results.

Setting lxc.autodev to 1 enables the container’s rootfs LXC to mount a fresh tmpfs under /dev (limited to 100k) and fill in a minimal set of initial devices. This is required only if the container has systemd based initialization, which Ubuntu 15.10 does.

Through the use of lxc.hook.autodev additional devices in the containers /dev directory are created. So in order to accomplish this create the file /var/lib/lxc/snappydev/autodev and paste the following inside:

#!/bin/bash
cd ${LXC_ROOTFS_MOUNT}/dev
mkdir net
mknod net/tun c 10 200
chmod 0666 net/tun

Also make sure that it is executable

user@host:$ chmod +x /var/lib/lxc/snappydev/autodev

This hook runs in the container’s namespace after mounting has been done and after any mount hooks have run, but before the pivot_root and creates some extra nodes that are needed under /dev so that systemd can properly work in the container.

Using the container

Now that we are done with the preparation of the host LXC setup we can finally enjoy the fruit of our labour and run the container.

How to start it

We can start the container with:

user@host:$ sudo lxc-start -n snappydev

If all went well you can see that the container is running by checking it with lxc-ls:

user@host:$ sudo lxc-ls -f
NAME       STATE    IPV4           IPV6  GROUPS  AUTOSTART  
----------------------------------------------------------
snappydev  RUNNING  192.168.1.164  -     -       NO

If it has not started properly then you should debug it by attempting to start the container in the foreground and with Debugging log priority:

user@host:$ sudo lxc-start -n snappydev -F --logpriority=DEBUG

This should provide enough information for you to find out what may be wrong with your container’s setup. One error that can be quite common is if your container initialization is stuck at:

[ ... ] Starting LSB: Raise network interfaces....

If you see that then please double check your networking configuration because that means you have made a mistake with it.

How to enter it

There are two ways to enter the container. lxc-console which gives you a login prompt, and lxc-attach which drops you directly at the root prompt.

Normally the default user should be ubuntu with password ubuntu but at the time of writing this post this seemed not to work for me. So we will have to go in the hard way. Go in as root and set the ubuntu user’s password.

user@host:$ sudo lxc-attach -n snappydev
group: cannot find name for group ID 19
root@snappydev:/# ls
bash:ls: command not found

ls not found? What trickery is this? Let’s see where does the $PATH point to

root@snappydev:/# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/bin/

Seems we have inherited the ArchLinux layout where binaries are under /usr/local/. That’s easy to fix. Simply add the following in the container’s /root/.bashrc to be able to access ubuntu binaries as root inside the container.

# inside the container we inherit, the PATH as set in archlinux and that's not gonna work for us in Ubuntu
export PATH="/bin/:/usr/bin/:/sbin/$PATH"

In any case the important part here is to set the ubuntu user’s password.

root@snappydev:/# passwd ubuntu
Enter new UNIX password
Retype new UNIX password
passwd: password updated successfully

Now from our host we can get a login as a normal user by typing:

user@host:$ sudo lxc-console -n snappydev

Connected to tty 1
Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself

Ubuntu 15.10 snappydev pts/0

snappydev login: ubuntu
Password: 
Last login: Sat Jan 30 10:40:01 UTC 2016 on pts/0
ubuntu@snappydev:~$

One important note. When you log out of the container and want to quite the login prompt and go back to the host you may find that <Ctrl-C> or <Ctrl-D> does not work. The right key combination to use here would be <Ctrl-Q-a>

Conclusion – or a Snappy beginning

Now that we finally managed to enter the container we should get some common dev packages and add the snappy ppa so that we can download the snappy tools.

ubuntu@snappydev:~$  sudo apt-get install software-properties-common
ubuntu@snappydev:~$  sudo apt-add-repository ppa:snappy-dev/tools
ubuntu@snappydev:~$  sudo apt-get update
ubuntu@snappydev:~$  sudo apt-get install snappy-tools

From here and on you can treat the container like a normal Ubuntu 15.10 host and follow the Snappy Guide to get started or if you are feeling a bit more adventurous and want to experiment with Snappy Ethereum you can read the tutorial I wrote on it.

As usual if you have any feedback/comments/suggestions on the post feel free to leave a message down here. Till next time!