Linux with Encryption and Checksumming Is Hard
Often with computers, I'll "go down a rabbit hole" AKA "yak shave". Part of the problem is that I get so focused on the way forward that I'm taking I want to keep going. Another part of the problem is that when I encounter any resistance I tend to get angry and push harder, instead of stopping to look for alternate ways. Please keep an eye out for alternatives in here, and let me know what you think I should do.
Right now I'm trying to set up some servers for personal cloud things, for publishing, collaborating, and a few other things. There are myriad ways to do this, but I tend to approach it like a software project, with a repo containing scripts that handle the management of the service in a consistent way.
My personal machines are in a wild state of disrepair, so doing this work is proving difficult. My personal laptop is running Ubuntu 14.04 which, while still supported, is 4 years old now and recently lost suspend functionality after a kernel security update. I am having trouble using my preferred modern editors, because my compiler is too old to build them. I would reformat it, but the state of my backup restore is currently untested. I would test that, but I am discovering my backup media are corrupt (which I will write about elsewhere), and I can only trust the one copy running on a server with ZFS, which is itself in a similar state of ancient disrepair. I need to write about this, and deleted my old Wordpress blog like 6 years ago, and never was satisfied enough with any other blog software to get anything working again (which I will write about elsewhere), so I'm hacking static HTML manually with some Markdown.
I haven't had time for about 3 years to get to some of these maintenance tasks. I couldn't even really start on the computer work until I filed 3 years of paperwork sitting on top of my desk. Things have been busy, and I am glad to be taking some time for a minute.
The reason I am setting up personal clouds and open source privacy-preserving computers is because I am taking time right now to research some of the problems created by technology. There is a lot of press about the ills of the internet and tech companies, and I not only don't want to be a part of the problem, I want to be part of the solution. That required finding out the true shape of the problem. (there will be more I will write about elsewhere - part of this work is setting up Tech Problems Work!)
I want a linux install onto encrypted media (which i have now) with storage that is verified (which i do not have now) and otherwise working hardware (suspend / resume, backlight, secure boot, etc). I have been trying Ubuntu 18.04. The default encrypted install uses unencrypted EFI System Partition and Boot partition, then a LUKS partition that acts as an LVM physical volume (
pv) with a virtual group (
vg) containing logical volumes (
lv) for swap and root as ext4. I can install this to a USB disk I have around, which can boot two computers, which is handy and works with my "computers shouldn't have storage built in" ideology (that I will write about elsewhere).
Ideally, I whould be able to do something like:
- EFI (should be secured from hardware)
- LVM pv
- LVM boot btrfs (should be able to be reached by grub)
- LVM root-1 btrfs
- LVM root-2 btrfs (I keep a second OS install partition around!)
- LVM swap
- LVM data btrfs
- LVM pv
and I ran into an article which points to much of the work needed to be done, but doesn't have everything. This crashes the Ubuntu
ubiquity OS installer. So I found a much more lengthy article which has a lengthy script that I downloaded and took apart, but which still crashes the installer, and leaves my disks in a state that does not appear recoverable as the instructions might suggest. That is, the root drive does not have the expected LSB / POSIX contents, but instead has a
'@' dir in it which does appear to, but it's not clear what work is left needing to be done to bring this disk to its desired finish.
So, I'm about to start from the default ubuntu install, and transform one step at a time closer to my desired outcome, installing and reboot testing all the while. Not a very productive use of my time, but I don't really see a way to make this work and meet my goal otherwise.
All of this just to get to a state where I can work regularly again. It makes me question my ideologies and rules in my head right now, but the very purpose of the work I am doing right now is to live by these ideologies and see if they can be used to solve greater problems created by technology and the tech industry right now. It's frustrating and challening on many levels.
Please write me, and let me know what you think I should do!
The default Ubuntu 18.04 encrypted install won't reinstall over itself and boot correctly. I wasn't expecting to fail on step 1!
To elaborate, in the Ubuntu 18.04 installer, if you select "Erase disk and install Ubuntu" and tick the box for "Encrypt the new Ubuntu installation for security" it automatically selects "Use LVM with the new Ubuntu installation" as it uses LVM on top of the encrypted partition to make both swap and root. The result looks like this:
- sda - OS install target disk
- sda1 - EFI /boot/efi (fat32)
- sda2 - /boot (ext4)
- sda3 - LUKS sda3_crypt (sidenote: ubiquity devs: is it a good idea to name w hw position?)
- /dev/mapper/sda3_crypt LVM pv, vg ubuntu-vg
- LVM root (ext4)
- LVM swap_1
- /dev/mapper/sda3_crypt LVM pv, vg ubuntu-vg
So far, so good. The OS installed onto the target will boot and run fine. However, if you
- restart the installer
- mount the volumes again as shown above
- select the partitions as install targets matching their roles above
- continue with the OS install
you will find the result doesn't boot. It tries, then drops you into the
Well, if the installer can't install back onto it's own partition creation, then I don't have much hope here!
I'm assuming the installer didn't know to add the LUKS or LVM setup into the boot sequence after the reinstall, but that seems a little silly. As far as I know there's not an "advanced installer" (which is a good thing! focus on one installer!) so this is the same OS install expected to be used in enterprise environments, for workstations, developers, all sorts. Maybe I shouldn't be still using Ubuntu, but it seems like both the one closest to Debian and with the most effort behind it.
Also, if it's not clear, I'm totally capable of doing all of this and figuring out anything in this chain of problems. It's just that
- this isn't really what I want to be spending my time doing
- if I'm looking for something to recommend others use for security, privacy, autonomy, and respect uhhhhh this is not really gonna work out for others!
Mobile OS vendors realized encryption by default fits the new modern world we live in. Maybe Ubuntu needs more time before they leap to that, but i think the process could be a more refined than this! I'm still happy Linux decided (Solaris first, but still) that checksumming file systems are important, and support
btrfs though! Hopefully I'll get to that part soon.
Ha! Of course this fixes it (second part with the chroot patchup): Ask Ubuntu: How can I install Ubuntu encrypted with LUKS with dual-boot?
Also, really handy when people write down the incantation to chroot, I always forget the right mounts and steps.
Trying with the same as above, but switching the two
btrfs. And here's the busybox prompt from
I didn't realize before, but Ubuntu by default with btrfs uses subvolumes (like in ZFS where you can nest filesystem mounts and have separate properties like quotas, replication, etc). There's the system one,
'@', and the user data one
'@home'. I can see where this would be useful, but it's a little of an extra complication without asking, and requires working around when using tools.
(For reference, the Ubuntu Wiki
But the funny part - why would this fail to boot? I didn't change the grub config after the installer finished, all I did was add
/etc/crypttab and run
update-initramfs which... maybe didn't see the right pathing because of how I chrooted? Unsure! More debugging and research, this time in the btrfs camps.
I kept looking around the target using the installer disk shell and I couldn't find anything wrong.
/etc/crypttab seemed in order, also
/etc/fstab. I kinda expected obvious problems there. So, nowhere else to go, I tried booting to the target again, to be able to see if there's an error message. There isn't any, just an
(initramfs) prompt. So, what's the problem?
Paying a little more attention, I noticed after boot I saw the password entry screen in
plymouth for unlocking the LVM filesystem. Right after that is when we drop into the shell. So,
/boot is being read correctly, because we're getting to the point where we can see that we need to load root. And the LUKS partition is unlocked correctly.
But then what? Let's run
mount in here and see... oh dear. The subvolume is not being selected correctly upon mount. Where is this being set?
crypttab? Somewhere in LVM metadata?
Wait... everyone keeps talking about not resetting the default subvol. Did that happen to mine?
Ok! remount in this mapper, then reset the default from the stock (5) to the root
And it boots, and I can login. How did that get reset? Who knows! But it did, and I fixed it. Next, onto encrypted
/boot for increased tamper resistence. We're almost at the finish line!
To recap, after install I already know I need to:
/etc/crypttabto contain the LUKS device
- fix the btrfs default subvol to point to root instead of the bare base.
Back to our lovely predicament. The difference between last run and the current is moving the
/boot btrfs partition into the LUKS LVM. Install, and then...
The 'grub-efi-amd64-signed' package failed to install into / target/. Without the GRUB boot loader, the installed system will not boot.
and then if I click continue...
We're sorry; the installer crashed. After you close this window, we'll allow you to file a bug report using the integrated bug reporting tool. This will gather information about your system and your installation process. The details will be sent to our bug tracker and a developer will attend to the problem as soon as possible.
This sounds reminiscent of the link I posted earlier - Ubuntu: ManualFullSystemEncryption - Community Help Wiki. The crashing installer is described, then there are a boatload of scripts that run. Let's see what they do!
encryptinstallation seems to think it needs to:
mountPartitions: First I'm going to mount as I would anyway,
chrootin, and write the
receiveScripts: I've already downloaded the
refreshgrubscripts to check as I go.
addEncryptionKeys: I don't think I want to do this! Why would I add key files like this? Where would I keep them?
setUpEncryptionAccess: This just seems to be grabbing UUIDs, which i have already been doing
allowInstallation: Why do I need universe and multiverse?
/etc/polkit-1/localauthority/50-local.d/com.ubuntu.enable-hibernate.pkla- of course it's gonna be polkit magic
fixEfi: not sure why
/boot/efi/startup.nshwould need to be edited?
setUpDecryption: this whole weird counterpoint to load the keys from
addEncryptionKeyswhich I still think I don't want to do at all.
processChroot: oh hey! What I'm already doing, then runs the
encryptinstallationchrootscript, which we can dig into after this.
removeTemporaryFiles: remove downloaded scripts
refreshgrub isn't called from this script, but
encryptinstallationchroot appears to think it needs to:
installApplications: update and install
incron libnotify-bin yad? Why?
Start refreshgrub automatically when /boot has been changed.
and that's all, but that makes me a little grumpy! This smells like too much for the user side to take on, and I think this looks pretty gnarly to my grizzled eyes. Why would grub not be doing these sorts of things itself? Are there bugs about this? Where is the chain of issues and mailing messages about this?
/boot be unencrypted feels... dumb. This feels pretty useless, if all of the built kernel images are sitting out in the open. Why is this so challenging for existing tools? Clearly I know some things about the boot process, but I don't know enough to say why the author of
refreshgrub needed to write that.
This is going to stick inn my craw for a while, but I need to move on and have a working system. Checksumming and the availability of a scrub process already puts me ahead of where I am now. I'm not sure how the scrubs are automated, but so long as it's a possibility, it can be figured out post-install.
Feels like we're done, for now.
2018-11-02 Summary / tl;dr
The summary, as executed on a few machines at this point. Please be safe if you're modifying this for your own purposes. Much of this is just copied from Ask Ubuntu: How can I install Ubuntu encrypted with LUKS with dual-boot?
gparted- create partition
/dev/sda3as unformatted named "candybar"
- command line from here on out, except for ubuntu installer
- make some disks to install into
lsblkto check on dev tree
sudo cryptsetup --label candybar_crypt luksFormat /dev/sda3
ll /dev/mapperto check
sudo pvcreate /dev/mapper/candybar_crypt
sudo vgcreate candybar-vg /dev/mapper/candybar_crypt
sudo lvcreate --name swap --size 32G candybar-vg
sudo lvcreate --name futureboot --size 1G candybar-vg
# for the hopeful
sudo lvcreate --name root1 --size 40G candybar-vg
sudo lvcreate --name root2 --size 40G candybar-vg
# two OS installs possible
sudo lvcreate --name databox --extents 100%FREE candybar-vg
# double check yer work
sudo mkswap --label swap /dev/mapper/candybar--vg-swap
sudo mkfs.btrfs --label futureboot /dev/mapper/candybar--vg-futureboot
sudo mkfs.btrfs --label root1 /dev/mapper/candybar--vg-root1
sudo mkfs.btrfs --label root2 /dev/mapper/candybar--vg-root2
sudo mkfs.btrfs --label databox /dev/mapper/candybar--vg-databox
- run through that installer, you should only need to specify swap, EFI,
/boot, and a mount point for
/disks/databox. Don't select format - you already formatted above!
- post-run fixup
sudo mount /dev/mapper/candybar--vg-root1 /mnt
here comes the btrfs differences from the other article...
sudo mount /dev/sda2 /mnt/@/boot
sudo mount /dev/sda1 /mnt/@/boot/efi
sudo mount --bind /mnt/@home /mnt/@/home
sudo mount --bind /dev /mnt/@/dev
ahh! before you chroot and tell grub you love what it does, correct the subvolume mistake!
sudo btrfs subvolume list /mnt
sudo btrfs subvolume get-default /mnt
you will notice that the record ID for @ is probably not the default! Set it to be. Mine was ID
sudo btrfs subvolume set-default 257 /mnt
sudo btrfs subvolume get-default /mnt
# check your qwork!
sudo chroot /mnt/@
> mount -t proc proc /proc
> mount -t sysfs sys /sys
> mount -t devpts devpts /dev/pts
/etc/crypttabto contain something like
candybar_crypt UUID=c726eaf2-09b2-44d7-882e-7a717f8776a9 none luks,retry=3,lvm=candybar-vg
update-initramfs -k all -c
- you should be good to reboot! You should see a prompt for the new volume you specified in the crypttab - ie,
- you still need to deal with the many failings of ubuntu - for instance, my hardware immediately requires an install of synapics or the trackpad stops receiving movement events during mousedown. (oh libinput, i will deal with you soon)
I hope this works for you!