Linux with Encryption and Checksumming Is Hard
Summary / tl;dr
To create an install of Ubuntu with an encrypted LVM and btrfs
partitions (still unencrypted /boot
sadly), as executed on a few machines at this point.
Please be safe if you're modifying this for your own purposes. Much of this is copied from Ask Ubuntu: How can I install Ubuntu encrypted with LUKS with dual-boot? with modifications for btrfs.
gparted
- create partition/dev/sda1
asfat32
named "efi",/dev/sda2
asbtrfs
named "pastboot",/dev/sda3
as unformatted named "candybar"- command line from here on out, except for ubuntu installer
- make some disks to install into
lsblk
to check on dev treesudo cryptsetup --label candybar_crypt luksFormat /dev/sda3
sudo cryptsetup luksOpen /dev/sda3 candybar_crypt
ll /dev/mapper
to check the mount- If you have not overwritten the free space on this disk before, set aside a few hours for the machine to run this -
sudo dd if=/dev/zero of=/dev/mapper/candybar_crypt bs=16M status=progress
- 2.6 hours for me on a 1TB laptop disk. Background on this is beyond the scope of this article, but this is a critical step for ensuring the integrity of the encryption. sudo blkid /dev/sda3
You will need the disk UUID later - do NOT use the partid, use the disk IDsudo pvcreate /dev/mapper/candybar_crypt
sudo vgcreate candybar-vg /dev/mapper/candybar_crypt
sudo lvcreate --name swap --size 32G candybar-vg
sudo lvcreate --name futureboot --size 1G candybar-vg
# for the hopeful
sudo lvcreate --name root1 --size 40G candybar-vg
sudo lvcreate --name root2 --size 40G candybar-vg
# two OS installs possible
sudo lvcreate --name databox --extents 100%FREE candybar-vg
lsblk
# double check yer work
sudo mkswap --label swap /dev/mapper/candybar--vg-swap
sudo mkfs.btrfs --label futureboot /dev/mapper/candybar--vg-futureboot
sudo mkfs.btrfs --label root1 /dev/mapper/candybar--vg-root1
sudo mkfs.btrfs --label root2 /dev/mapper/candybar--vg-root2
sudo mkfs.btrfs --label databox /dev/mapper/candybar--vg-databox
- run through that installer, you should only need to specify
/
,/boot
, EFI, swap, and a mount point for/disks/databox
. Don't select format - you already formatted above! - post-run fixup
-
sudo mount /dev/mapper/candybar--vg-root1 /mnt
-
here comes the btrfs differences from the other article...
-
sudo mount /dev/sda2 /mnt/@/boot
-
sudo mount /dev/sda1 /mnt/@/boot/efi
-
sudo mount --bind /mnt/@home /mnt/@/home
-
sudo mount --bind /dev /mnt/@/dev
-
ahh! before you chroot and tell grub you love what it does, correct the subvolume mistake!
-
sudo btrfs subvolume list /mnt
-
sudo btrfs subvolume get-default /mnt
-
you will notice that the record ID for @ is probably not the default! Set it to be. Mine was ID
257
-
sudo btrfs subvolume set-default 257 /mnt
-
sudo btrfs subvolume get-default /mnt
# check your qwork!
-
sudo chroot /mnt/@
-
> mount -t proc proc /proc
-
> mount -t sysfs sys /sys
-
> mount -t devpts devpts /dev/pts
-
edit
/etc/crypttab
to contain something likecandybar_crypt UUID=c726eaf2-09b2-44d7-882e-7a717f8776a9 none luks,retry=3,lvm=candybar-vg
-
update-initramfs -k all -c
- check for errors on initramfs for invalid UUID, double check output fromlsblk
above
-
- you should be good to reboot! You should see a prompt for the new volume you specified in the crypttab - ie,
candybar_crypt
- you still need to deal with the many failings of ubuntu - for instance, my hardware immediately requires an install of synapics or the trackpad stops receiving movement events during mousedown. (oh libinput, i will deal with you soon)
I hope this works for you!
Original intro
Often with computers, I'll "go down a rabbit hole" or "yak shave". I tend to feel focused on the path forward, and yearn to keep going. Sometimes when I encounter resistance I tend to feel angry and push harder, instead of stopping to look for alternate ways. So, I can wind up doing a lot of research in this work, and I thought I would help share that here.
Background
Right now I'm trying to set up some servers for personal cloud things, for publishing, collaborating, and a few other things. There are myriad ways to do this, but I tend to approach it like a software project, with a repo containing scripts that handle the management of the service in a consistent way.
My personal machines are in a wild state of disrepair, so doing this work is proving difficult. My personal laptop is running Ubuntu 14.04 which, while still supported, is 4 years old now and recently lost suspend functionality after a kernel security update. I am having trouble using my preferred modern editors, because my compiler is too old to build them. I would reformat it, but the state of my backup restore is currently untested. I would test that, but I am discovering my backup media are corrupt (which I will write about elsewhere), and I can only trust the one copy running on a server with ZFS, which is itself in a similar state of ancient disrepair. I need to write about this, and deleted my old Wordpress blog like 6 years ago, and never was satisfied enough with any other blog software to get anything working again (which I will write about elsewhere), so I'm hacking static HTML manually with some Markdown.
I haven't had time for about 3 years to get to some of these maintenance tasks. I couldn't even really start on the computer work until I filed 3 years of paperwork sitting on top of my desk. Things have been busy, and I am glad to be taking some time for a minute.
The reason I am setting up personal clouds and open source privacy-preserving computers is because I am taking time right now to research some of the problems created by technology. There is a lot of press about the ills of the internet and tech companies, and I not only don't want to be a part of the problem, I want to be part of the solution. That required finding out the true shape of the problem. (there will be more I will write about elsewhere - part of this work is setting up Tech Problems Work!)
Current Situation
I want a linux install onto encrypted media (which i have now) with storage that is verified (which i do not have now) and otherwise working hardware (suspend / resume, backlight, secure boot, etc). I have been trying Ubuntu 18.04. The default encrypted install uses unencrypted EFI System Partition and Boot partition, then a LUKS partition that acts as an LVM physical volume (pv
) with a virtual group (vg
) containing logical volumes (lv
) for swap and root as ext4. I can install this to a USB disk I have around, which can boot two computers, which is handy and works with my "computers shouldn't have storage built in" ideology (that I will write about elsewhere).
Ideally, I whould be able to do something like:
- EFI (should be secured from hardware)
- LUKS
- LVM pv
- LVM boot btrfs (should be able to be reached by grub)
- LVM root-1 btrfs
- LVM root-2 btrfs (I keep a second OS install partition around!)
- LVM swap
- LVM data btrfs
- LVM pv
and I ran into an article which points to much of the work needed to be done, but doesn't have everything. This crashes the Ubuntu ubiquity
OS installer. So I found a much more lengthy article which has a lengthy script that I downloaded and took apart, but which still crashes the installer, and leaves my disks in a state that does not appear recoverable as the instructions might suggest. That is, the root drive does not have the expected LSB / POSIX contents, but instead has a '@'
dir in it which does appear to, but it's not clear what work is left needing to be done to bring this disk to its desired finish.
So, I'm about to start from the default ubuntu install, and transform one step at a time closer to my desired outcome, installing and reboot testing all the while. Not a very productive use of my time, but I don't really see a way to make this work and meet my goal otherwise.
All of this just to get to a state where I can work regularly again. It makes me question my ideologies and rules in my head right now, but the very purpose of the work I am doing right now is to live by these ideologies and see if they can be used to solve greater problems created by technology and the tech industry right now. It's frustrating and challening on many levels.
Please write me, and let me know what you think I should do!
Updates
2018-10-25 #1
The default Ubuntu 18.04 encrypted install won't reinstall over itself and boot correctly. I wasn't expecting to fail on step 1!
To elaborate, in the Ubuntu 18.04 installer, if you select "Erase disk and install Ubuntu" and tick the box for "Encrypt the new Ubuntu installation for security" it automatically selects "Use LVM with the new Ubuntu installation" as it uses LVM on top of the encrypted partition to make both swap and root. The result looks like this:
- sda - OS install target disk
- sda1 - EFI /boot/efi (fat32)
- sda2 - /boot (ext4)
- sda3 - LUKS sda3_crypt (sidenote: ubiquity devs: is it a good idea to name w hw position?)
- /dev/mapper/sda3_crypt LVM pv, vg ubuntu-vg
- LVM root (ext4)
- LVM swap_1
- /dev/mapper/sda3_crypt LVM pv, vg ubuntu-vg
So far, so good. The OS installed onto the target will boot and run fine. However, if you
- restart the installer
- mount the volumes again as shown above
- select the partitions as install targets matching their roles above
- continue with the OS install
you will find the result doesn't boot. It tries, then drops you into the initramfs
shell.
Well, if the installer can't install back onto it's own partition creation, then I don't have much hope here!
I'm assuming the installer didn't know to add the LUKS or LVM setup into the boot sequence after the reinstall, but that seems a little silly. As far as I know there's not an "advanced installer" (which is a good thing! focus on one installer!) so this is the same OS install expected to be used in enterprise environments, for workstations, developers, all sorts. Maybe I shouldn't be still using Ubuntu, but it seems like both the one closest to Debian and with the most effort behind it.
Also, if it's not clear, I'm totally capable of doing all of this and figuring out anything in this chain of problems. It's just that
- this isn't really what I want to be spending my time doing
- if I'm looking for something to recommend others use for security, privacy, autonomy, and respect uhhhhh this is not really gonna work out for others!
Mobile OS vendors realized encryption by default fits the new modern world we live in. Maybe Ubuntu needs more time before they leap to that, but i think the process could be a more refined than this! I'm still happy Linux decided (Solaris first, but still) that checksumming file systems are important, and support btrfs
though! Hopefully I'll get to that part soon.
2018-10-25 #2
Ha! Of course this fixes it (second part with the chroot patchup): Ask Ubuntu: How can I install Ubuntu encrypted with LUKS with dual-boot?
Also, really handy when people write down the incantation to chroot, I always forget the right mounts and steps.
2018-10-25 #3
Trying with the same as above, but switching the two ext4
-> btrfs
. And here's the busybox prompt from initramfs
again!
I didn't realize before, but Ubuntu by default with btrfs uses subvolumes (like in ZFS where you can nest filesystem mounts and have separate properties like quotas, replication, etc). There's the system one, '@'
, and the user data one '@home'
. I can see where this would be useful, but it's a little of an extra complication without asking, and requires working around when using tools.
(For reference, the Ubuntu Wiki btrfs
page.)
But the funny part - why would this fail to boot? I didn't change the grub config after the installer finished, all I did was add /etc/crypttab
and run update-initramfs
which... maybe didn't see the right pathing because of how I chrooted? Unsure! More debugging and research, this time in the btrfs camps.
2018-10-26 #1
I kept looking around the target using the installer disk shell and I couldn't find anything wrong. /etc/crypttab
seemed in order, also /etc/fstab
. I kinda expected obvious problems there. So, nowhere else to go, I tried booting to the target again, to be able to see if there's an error message. There isn't any, just an (initramfs)
prompt. So, what's the problem?
Paying a little more attention, I noticed after boot I saw the password entry screen in plymouth
for unlocking the LVM filesystem. Right after that is when we drop into the shell. So, /boot
is being read correctly, because we're getting to the point where we can see that we need to load root. And the LUKS partition is unlocked correctly.
But then what? Let's run mount
in here and see... oh dear. The subvolume is not being selected correctly upon mount. Where is this being set? crypttab
? Somewhere in LVM metadata?
Wait... everyone keeps talking about not resetting the default subvol. Did that happen to mine?
YES.
Ok! remount in this mapper, then reset the default from the stock (5) to the root '@'
(257)...
And it boots, and I can login. How did that get reset? Who knows! But it did, and I fixed it. Next, onto encrypted /boot
for increased tamper resistence. We're almost at the finish line!
2018-10-26 #2
To recap, after install I already know I need to:
- fix
/etc/crypttab
to contain the LUKS device - fix the btrfs default subvol to point to root instead of the bare base.
Back to our lovely predicament. The difference between last run and the current is moving the /boot
btrfs partition into the LUKS LVM. Install, and then...
The 'grub-efi-amd64-signed' package failed to install into / target/. Without the GRUB boot loader, the installed system will not boot.
and then if I click continue...
Installer crashed
We're sorry; the installer crashed. After you close this window, we'll allow you to file a bug report using the integrated bug reporting tool. This will gather information about your system and your installation process. The details will be sent to our bug tracker and a developer will attend to the problem as soon as possible.
This sounds reminiscent of the link I posted earlier - Ubuntu: ManualFullSystemEncryption - Community Help Wiki. The crashing installer is described, then there are a boatload of scripts that run. Let's see what they do!
encryptinstallation
seems to think it needs to:
mountPartitions
: First I'm going to mount as I would anyway,chroot
in, and write the/etc/crypttab
as above.receiveScripts
: I've already downloaded theencryptinstallationchroot
andrefreshgrub
scripts to check as I go.addEncryptionKeys
: I don't think I want to do this! Why would I add key files like this? Where would I keep them?setUpEncryptionAccess
: This just seems to be grabbing UUIDs, which i have already been doingallowInstallation
: Why do I need universe and multiverse?enableHibernation
: gah,/etc/polkit-1/localauthority/50-local.d/com.ubuntu.enable-hibernate.pkla
- of course it's gonna be polkit magicsetDefaultGrub
:GRUB_ENABLE_CRYPTODISK=y
in/etc/default/grub
fixEfi
: not sure why/boot/efi/startup.nsh
would need to be edited?setUpDecryption
: this whole weird counterpoint to load the keys fromaddEncryptionKeys
which I still think I don't want to do at all.processChroot
: oh hey! What I'm already doing, then runs theencryptinstallationchroot
script, which we can dig into after this.removeTemporaryFiles
: remove downloaded scripts
refreshgrub
isn't called from this script, but encryptinstallationchroot
is.
encryptinstallationchroot
appears to think it needs to:
-
installApplications
: update and installincron libnotify-bin yad
? Why? -
setAutoGrubUpdate
: aha,Start refreshgrub automatically when /boot has been changed.
and that's all, but that makes me a little grumpy! This smells like too much for the user side to take on, and I think this looks pretty gnarly to my grizzled eyes. Why would grub not be doing these sorts of things itself? Are there bugs about this? Where is the chain of issues and mailing messages about this?
Having /boot
be unencrypted feels... dumb. This feels pretty useless, if all of the built kernel images are sitting out in the open. Why is this so challenging for existing tools? Clearly I know some things about the boot process, but I don't know enough to say why the author of refreshgrub
needed to write that.
This is going to stick inn my craw for a while, but I need to move on and have a working system. Checksumming and the availability of a scrub process already puts me ahead of where I am now. I'm not sure how the scrubs are automated, but so long as it's a possibility, it can be figured out post-install.
Feels like we're done, for now.