Last time, we figured out why not being able to retrieve EFI variables while installing Linux results in an empty UEFI boot menu and an inaccessible BIOS setup menu. Now, we want to discover what is going wrong and what we can do about it.

Errata: Hardcoded boot menu keys

In the last post, I mentioned that Fujitsu supposedly hardcoded the F2 and F12 boot keys to invoke specific Boot slots. This is a stupid idea because overwriting the Boot slots that the keys refer to will render those keypresses useless.

However, UEFI specified a variable-based way to override boot key mappings (KeyXXXX), and Fujitsu used that to provide the default key mappings for F2 and F12. Of course, if Boot0000 gets overwritten erroneously, that doesn’t mean that the respective Key entry gets updated, which resulted in the issues we have seen.

This does not make the issue any less stupid, but at least it’s specification-supported stupidity.

Excursion: UEFI variable iteration

UEFI provides a limited set of interfaces for accessing variables, with operations that essentially amount to “iterate variable names” (GetNextVariableName), “get variable contents” (GetVariable), and “set variable contents” (SetVariable).

Variable iteration works by providing a buffer for the variable GUID (a rough classification/namespacing feature), a buffer for the variable name, and the matching size of the buffer for the variable name.

For the first iteration, the interface user fills in a string of size 0 into the variable name buffer to indicate that the iteration should start from the beginning of the list. The GUID and name are then copied to the respective buffers. On subsequent iterations, the contents must be kept as-is to allow the UEFI implementation to retrieve the following variable successfully.

On a successful iteration, the call to GetNextVariableName returns EFI_SUCCESS. If the following variable is not found, the function returns EFI_NOT_FOUND. If the name buffer is too small to hold the full variable name, the function returns EFI_BUFFER_TOO_SMALL, the passed buffer size is updated with the required buffer size, and the user is meant to resize the buffer to fit the name and try again. If any of the involved pointers are a null-pointer, the specification mandates a return value of EFI_INVALID_PARAMETER.

How not to do error handling

Tracking back to where the issues started, I inspected the kernel log for any EFI-related lines and soon found what I was looking for:

[    5.215411] efivars: get_next_variable: status=8000000000000002

Searching for this line in the Linux kernel source code, an error handling switch in the efivarfs implementation can be found:

do {
    variable_name_size = 1024;

    status = efivar_get_next_variable(&variable_name_size,
                      variable_name,
                      &vendor_guid);
    switch (status) {
    case EFI_SUCCESS:
        // [...]
        break;
    // [omitted cases for EFI_UNSUPPORTED and EFI_NOT_FOUND]
    default:
        printk(KERN_WARNING "efivars: get_next_variable: status=%lx\n",
               status);
        status = EFI_NOT_FOUND;
        break;
    }
} while (status != EFI_NOT_FOUND);

This code is part of a function responsible for retrieving the list of EFI variables, adding entries to a list if successful, and stopping iteration if not.

However, iteration also stops when an unexpected error is encountered and (relatively) silently pretends we hit the normal exit case. No attempts are made to mitigate an error that, in theory, shouldn’t exist in this case. This evidently isn’t great since it allowed the bug to persist for the last ten years.

Down the UEFI rabbit hole

Whenever there is an unexpected error code, there usually is a place where that error code is returned. The first step is to check whether this is somewhere within the kernel, so I got started with printk-debugging:

[    5.229441] Starting efivar_init!
[    5.229446] pre efivarfs efivar_get_next_variable(0xffffc90000043c68, 0xffff88810bc8f400, 0xffffc90000043c70)
[    5.229448] enter drv/fw efivar_get_next_variable(0xffffc90000043c68, 0xffff88810bc8f400, 0xffffc90000043c70)
[    5.229449] enter runtime-wrapper virt_eft_get_next_variable(0xffffc90000043c68, 0xffff88810bc8f400, 0xffffc90000043c70)
[    5.229460] pre runtime-wrapper efi_call_virt(get_next_variable, 0xffffc90000043c68, 0xffff88810bc8f400, 0xffffc90000043c70)
[    5.229467] post runtime-wrapper efi_call_virt(get_next_variable, ...) -> 0x8000000000000002
[    5.229473] leave runtime-wrapper virt_eft_get_next_variable() -> 0x8000000000000002
[    5.229475] leave drv/fw efivar_get_next_variable() -> 0x8000000000000002
[    5.229476] post efivarfs efivar_get_next_variable() -> 0x8000000000000002
[    5.229478] efivars: get_next_variable: status=8000000000000002

Unfortunately, all that this confirmed to me is that the error code is coming from the UEFI implementation itself. The purpose of efi_call_virt is to switch to a memory management and interrupt context where accessing UEFI services is allowed, which makes easy debugging beyond this point next to impossible.

After a quick (unfruitful) detour into unpacking and reverse engineering the UEFI image, I was out of options to try and posted the first part of my investigation online. In the Hacker News discussion, Matthew Garrett suggested running both Linux and Windows under a debug build of EDK II (the de-facto UEFI reference implementation) to track down the difference in how the respective system calls the involved UEFI functions.

Instrumenting UEFI

It’s not immediately obvious how to build OVMF images based on EDK II since they appear to have supported at least three different build systems in the past, but I managed to cobble together a set of commands that successfully builds a bootable image from source:

make -C BaseTools
. edksetup.sh
build -p OvmfPkg/OvmfPkgX64.dsc -a X64 -b DEBUG -n 6 -t GCC5

The resulting image will not be affected by the same bug the actual hardware has. Still, it allowed me to continue printf-debugging until I found the specific detail that Fujitsu’s UEFI implementation cannot handle since this build of UEFI will output DEBUG log lines via the debug console. To make them accessible under QEMU, I had to add a few additional options to my VM invocation:

qemu-system-x86_64 -drive if=pflash,file=./Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd,format=raw -net none -s -debugcon file:debug.log -global isa-debugcon.iobase=0x402

In the meantime, I also ensured that FreeBSD can successfully retrieve the list of UEFI variables on real hardware since an open-source operating system is generally more pleasant to debug than a closed-source operating system.

A few log lines and two VM startups later, I successfully captured UEFI logs of both Linux and FreeBSD, and I was ready to check their calls to GetNextVariableName for differences. Since I’m not well-versed in printf specifiers, I only logged all three involved pointers, as well as the value of the VariableNameSize parameter.

[Linux]
AH532: VariableServiceGetNextVariableName(0xFFFFC9000076FDA8 <0x400>, 0xFFFF88800261A400, 0xFFFFC9000076FDB0)
AH532: VariableServiceGetNextVariableName(0xFFFFC9000076FDA8 <0x400>, 0xFFFF88800261A400, 0xFFFFC9000076FDB0)
AH532: VariableServiceGetNextVariableName(0xFFFFC9000076FDA8 <0x400>, 0xFFFF88800261A400, 0xFFFFC9000076FDB0)

[FreeBSD]
AH532: VariableServiceGetNextVariableName(0xFFFFFE004917AD58 <0x200>, 0xFFFFF80001187800, 0xFFFFFE004917AD60)
AH532: VariableServiceGetNextVariableName(0xFFFFFE004917AD58 <0x200>, 0xFFFFF80001187800, 0xFFFFFE004917AD60)
AH532: VariableServiceGetNextVariableName(0xFFFFFE004917AD58 <0x200>, 0xFFFFF80001187800, 0xFFFFFE004917AD60)

Assuming that something as simple as memory mapping wouldn’t be messed up, and since all the pointer alignments are otherwise unchanged, that really only leaves the value of VariableNameSize as the cause for issues, but surely, this can’t be what UEFI is actually getting stuck on, right?

Well, at least I was thinking wrong because one tiny patch to the kernel later, retrieving the UEFI variable name list on real hardware worked like a charm:

diff --git a/fs/efivarfs/vars.c b/fs/efivarfs/vars.c
index 9e4f47808bd5a..297e2b2120b6b 100644
--- a/fs/efivarfs/vars.c
+++ b/fs/efivarfs/vars.c
@@ -394,7 +394,7 @@ int efivar_init(int (*func)(efi_char16_t *, efi_guid_t, unsigned long, void *),
         */

        do {
-               variable_name_size = 1024;
+               variable_name_size = 512;

                status = efivar_get_next_variable(&variable_name_size,
                                                  variable_name,

# ls -al /sys/firmware/efi/efivars/
total 0
drwxr-xr-x 2 root root    0 23. Apr 02:54 .
drwxr-xr-x 4 root root    0 23. Apr 02:54 ..
-rw-r--r-- 1 root root   51 23. Apr 02:54 AbsoluteVar-69d88529-db90-4e0f-839b-bfc1b89ea989
-rw-r--r-- 1 root root   12 23. Apr 02:54 AcpiGlobalVariable-af9ffd67-ec10-488a-9dfc-6cbf5ee22c2e
-rw-r--r-- 1 root root   64 23. Apr 02:54 AdvancedPage-397faf4e-893e-468f-992d-9acf30c52142
[...]

Great, another issue that looks like the universe really hates this particular family of laptops. I haven’t managed to track down the exact place in the binary where things start to go wrong (nor have I spent much time trying to do so), but this seems substantial enough to submit a workaround patch to the Linux kernel anyway.

A Mad Patch-Party

The first revision of the patch tried to stay mindful of the fact that simply lowering the limit down to 512 may break some existing setups, but (in hindsight) ended up too complicated for what it was trying to achieve since it tried to minimize the number of memory allocations for setups that need no more than 1024 bytes.

The second revision went for a slightly more straightforward approach, where the loop would start off with the old limit of 1024 and continuously halve the advertised buffer size until the first successful response was received. Excluding the fourth revision of the patch (which never made it to the mailing list), this would have been my favorite approach of all available options since hardware with a similar quirk (but at a lower buffer length) would have been covered as well.

I originally only sent the third revision to satisfy a request for the simplest patch that would solve the problem (which is to decrease the buffer size to 512 bytes). Despite my concerns regarding regressions (and intermediary confusion on whether Linux would even support file names longer than 256 characters, which take up 512 bytes when encoded in UTF-16), this patch version was merged into the mainline kernel. I’m not sure how to feel that a justification that I explicitly rejected beforehand ended up being edited into the patch before it was applied (and, worst of all, including a typo), but at least the issue is fixed now.

This patch has since landed in all currently supported stable branches of the Linux kernel. It has even made it in time to be included in the beta installation images of Ubuntu 24.04 (and presumably all Ubuntu installation images going forward).

Postmortem

Looking back at the chain of events, seeing how many things needed to be in place for the bug to be as destructive as it was is interesting:

  • Fujitsu/Phoenix messed up their implementation of GetNextVariableName.
  • Linux doesn’t care that the function is not implemented entirely to specification, ignoring the error.
  • Linux does not fall back to GetVariable when accessing a variable directly (and doesn’t manually use it to populate specification-required variables such as BootOrder).
  • efibootmgr does not care that something is obviously wrong (because it can’t find mandated variables) and tries to create a new boot menu configuration anyway.
  • Fujitsu/Phoenix did not put their default boot entries into any of the slots that aren’t the first ones that would be used.
  • Fujitsu/Phoenix did not use the variable hooking mechanisms they have to protect Boot0000 and Boot0001.

Acknowledgements

Many thanks to…

  • … Hacker News and Lobsters for discussing and linking many related issues and potential leads for me to follow up on
  • … Matthew Garrett for hinting at a debug build of EDK II
  • … Ard Biesheuvel for helping with massaging my patches into a state where they are actually kernel-ready (despite me being very stubborn at times)
  • … Ubuntu Forums moderators for offering to update twelve-year-old forum threads (so that other users may unbrick their laptops)
  • … various fellow university students for proofreading things before I post them on the internet