Linux with Encryption and Checksumming Is Hard

Start 2018-10-24 › Pub 2018-10-24 › Edit 2019-03-31

Summary / tl;dr

To create an install of Ubuntu with an encrypted LVM and btrfs partitions (still unencrypted /boot sadly), as executed on a few machines at this point.

Please be safe if you're modifying this for your own purposes. Much of this is copied from Ask Ubuntu: How can I install Ubuntu encrypted with LUKS with dual-boot? with modifications for btrfs.

I hope this works for you!

Original intro

Often with computers, I'll "go down a rabbit hole" or "yak shave". I tend to feel focused on the path forward, and yearn to keep going. Sometimes when I encounter resistance I tend to feel angry and push harder, instead of stopping to look for alternate ways. So, I can wind up doing a lot of research in this work, and I thought I would help share that here.

Background

Right now I'm trying to set up some servers for personal cloud things, for publishing, collaborating, and a few other things. There are myriad ways to do this, but I tend to approach it like a software project, with a repo containing scripts that handle the management of the service in a consistent way.

My personal machines are in a wild state of disrepair, so doing this work is proving difficult. My personal laptop is running Ubuntu 14.04 which, while still supported, is 4 years old now and recently lost suspend functionality after a kernel security update. I am having trouble using my preferred modern editors, because my compiler is too old to build them. I would reformat it, but the state of my backup restore is currently untested. I would test that, but I am discovering my backup media are corrupt (which I will write about elsewhere), and I can only trust the one copy running on a server with ZFS, which is itself in a similar state of ancient disrepair. I need to write about this, and deleted my old Wordpress blog like 6 years ago, and never was satisfied enough with any other blog software to get anything working again (which I will write about elsewhere), so I'm hacking static HTML manually with some Markdown.

I haven't had time for about 3 years to get to some of these maintenance tasks. I couldn't even really start on the computer work until I filed 3 years of paperwork sitting on top of my desk. Things have been busy, and I am glad to be taking some time for a minute.

The reason I am setting up personal clouds and open source privacy-preserving computers is because I am taking time right now to research some of the problems created by technology. There is a lot of press about the ills of the internet and tech companies, and I not only don't want to be a part of the problem, I want to be part of the solution. That required finding out the true shape of the problem. (there will be more I will write about elsewhere - part of this work is setting up Tech Problems Work!)

Current Situation

I want a linux install onto encrypted media (which i have now) with storage that is verified (which i do not have now) and otherwise working hardware (suspend / resume, backlight, secure boot, etc). I have been trying Ubuntu 18.04. The default encrypted install uses unencrypted EFI System Partition and Boot partition, then a LUKS partition that acts as an LVM physical volume (pv) with a virtual group (vg) containing logical volumes (lv) for swap and root as ext4. I can install this to a USB disk I have around, which can boot two computers, which is handy and works with my "computers shouldn't have storage built in" ideology (that I will write about elsewhere).

Ideally, I whould be able to do something like:

and I ran into an article which points to much of the work needed to be done, but doesn't have everything. This crashes the Ubuntu ubiquity OS installer. So I found a much more lengthy article which has a lengthy script that I downloaded and took apart, but which still crashes the installer, and leaves my disks in a state that does not appear recoverable as the instructions might suggest. That is, the root drive does not have the expected LSB / POSIX contents, but instead has a '@' dir in it which does appear to, but it's not clear what work is left needing to be done to bring this disk to its desired finish.

So, I'm about to start from the default ubuntu install, and transform one step at a time closer to my desired outcome, installing and reboot testing all the while. Not a very productive use of my time, but I don't really see a way to make this work and meet my goal otherwise.

All of this just to get to a state where I can work regularly again. It makes me question my ideologies and rules in my head right now, but the very purpose of the work I am doing right now is to live by these ideologies and see if they can be used to solve greater problems created by technology and the tech industry right now. It's frustrating and challening on many levels.

Please write me, and let me know what you think I should do!

Updates

2018-10-25 #1

The default Ubuntu 18.04 encrypted install won't reinstall over itself and boot correctly. I wasn't expecting to fail on step 1!

To elaborate, in the Ubuntu 18.04 installer, if you select "Erase disk and install Ubuntu" and tick the box for "Encrypt the new Ubuntu installation for security" it automatically selects "Use LVM with the new Ubuntu installation" as it uses LVM on top of the encrypted partition to make both swap and root. The result looks like this:

So far, so good. The OS installed onto the target will boot and run fine. However, if you

you will find the result doesn't boot. It tries, then drops you into the initramfs shell.

Well, if the installer can't install back onto it's own partition creation, then I don't have much hope here!

I'm assuming the installer didn't know to add the LUKS or LVM setup into the boot sequence after the reinstall, but that seems a little silly. As far as I know there's not an "advanced installer" (which is a good thing! focus on one installer!) so this is the same OS install expected to be used in enterprise environments, for workstations, developers, all sorts. Maybe I shouldn't be still using Ubuntu, but it seems like both the one closest to Debian and with the most effort behind it.

Also, if it's not clear, I'm totally capable of doing all of this and figuring out anything in this chain of problems. It's just that

Mobile OS vendors realized encryption by default fits the new modern world we live in. Maybe Ubuntu needs more time before they leap to that, but i think the process could be a more refined than this! I'm still happy Linux decided (Solaris first, but still) that checksumming file systems are important, and support btrfs though! Hopefully I'll get to that part soon.

2018-10-25 #2

Ha! Of course this fixes it (second part with the chroot patchup): Ask Ubuntu: How can I install Ubuntu encrypted with LUKS with dual-boot?

Also, really handy when people write down the incantation to chroot, I always forget the right mounts and steps.

2018-10-25 #3

Trying with the same as above, but switching the two ext4 -> btrfs. And here's the busybox prompt from initramfs again!

I didn't realize before, but Ubuntu by default with btrfs uses subvolumes (like in ZFS where you can nest filesystem mounts and have separate properties like quotas, replication, etc). There's the system one, '@', and the user data one '@home'. I can see where this would be useful, but it's a little of an extra complication without asking, and requires working around when using tools.

(For reference, the Ubuntu Wiki btrfs page.)

But the funny part - why would this fail to boot? I didn't change the grub config after the installer finished, all I did was add /etc/crypttab and run update-initramfs which... maybe didn't see the right pathing because of how I chrooted? Unsure! More debugging and research, this time in the btrfs camps.

2018-10-26 #1

I kept looking around the target using the installer disk shell and I couldn't find anything wrong. /etc/crypttab seemed in order, also /etc/fstab. I kinda expected obvious problems there. So, nowhere else to go, I tried booting to the target again, to be able to see if there's an error message. There isn't any, just an (initramfs) prompt. So, what's the problem?

Paying a little more attention, I noticed after boot I saw the password entry screen in plymouth for unlocking the LVM filesystem. Right after that is when we drop into the shell. So, /boot is being read correctly, because we're getting to the point where we can see that we need to load root. And the LUKS partition is unlocked correctly.

But then what? Let's run mount in here and see... oh dear. The subvolume is not being selected correctly upon mount. Where is this being set? crypttab? Somewhere in LVM metadata?

Wait... everyone keeps talking about not resetting the default subvol. Did that happen to mine?

YES.

Ok! remount in this mapper, then reset the default from the stock (5) to the root '@' (257)...

And it boots, and I can login. How did that get reset? Who knows! But it did, and I fixed it. Next, onto encrypted /boot for increased tamper resistence. We're almost at the finish line!

2018-10-26 #2

To recap, after install I already know I need to:

Back to our lovely predicament. The difference between last run and the current is moving the /boot btrfs partition into the LUKS LVM. Install, and then...

The 'grub-efi-amd64-signed' package failed to install into / target/. Without the GRUB boot loader, the installed system will not boot.

and then if I click continue...

Installer crashed

We're sorry; the installer crashed. After you close this window, we'll allow you to file a bug report using the integrated bug reporting tool. This will gather information about your system and your installation process. The details will be sent to our bug tracker and a developer will attend to the problem as soon as possible.

This sounds reminiscent of the link I posted earlier - Ubuntu: ManualFullSystemEncryption - Community Help Wiki. The crashing installer is described, then there are a boatload of scripts that run. Let's see what they do!

encryptinstallation seems to think it needs to:

refreshgrub isn't called from this script, but encryptinstallationchroot is.

encryptinstallationchroot appears to think it needs to:

and that's all, but that makes me a little grumpy! This smells like too much for the user side to take on, and I think this looks pretty gnarly to my grizzled eyes. Why would grub not be doing these sorts of things itself? Are there bugs about this? Where is the chain of issues and mailing messages about this?

Having /boot be unencrypted feels... dumb. This feels pretty useless, if all of the built kernel images are sitting out in the open. Why is this so challenging for existing tools? Clearly I know some things about the boot process, but I don't know enough to say why the author of refreshgrub needed to write that.

This is going to stick inn my craw for a while, but I need to move on and have a working system. Checksumming and the availability of a scrub process already puts me ahead of where I am now. I'm not sure how the scrubs are automated, but so long as it's a possibility, it can be figured out post-install.

Feels like we're done, for now.