Virtualization levels on AArch64
Being a fairly recent architecture, AArch64 has a relatively clean virtualization design. At its core, AArch64
virtualization happens along two axes: Secure/Non-Secure, and Exception Level. In the AArch64 model, the system runs two
"worlds" concurrently: First, there is the "secure world", which boots the system and which is responsible for the first
stages of secure boot authentication. This is what TrustZone refers to. Next to this secure world lies the normal world,
in which the operating system kernel and all user applications run. Each of the two worlds has its own exception levels
(EL) 0-2, with EL0 being the least privileged intended for applications, EL1 being the operating system kernel, and EL2
being the hypervisor. Code in higher exception levels can set interrupts and memory mappings for lower exception levels,
and code from lower exception levels can call into higher exception levels using special instructions.
Above EL2, there is one more exception level, EL3. Unlike EL0-2, EL3 is shared between the secure and normal worlds and
manages the two worlds. The only way for execution to pass between the secure and normal worlds is to go through EL3.
EL3 is where the system's boot starts, and where the earliest bootloader stages are executed.
For KVM virtualization to work, the linux kernel has to be booted running in EL2. Depending on the exact CPU version, it
will then either continue running entirely in EL2, or the kernel will install a small piece of code managing EL2 in EL2,
then drop the rest of itself to EL1. The QRB2210 SoC on the Uno Q board has ARM Cortex A53 cores which implement Arm
v8.0-A, which due to its age lacks the VHE extension necessary to run the entire kernel at EL2, so on this hardware the
kernel will install a kvm stub in EL2 and then drop to EL1. This comes at some performance penalty.
QHEE, or Qualcomm NIH'ing the hypervisor
Over the past decades, the operating system landscape has decisively settled along the lines of linux/android, windows,
and whatever apple is doing. No manufacturer of smartphone SoCs would want to waste a bunch of money on building a
custom OS and forego the network effects of running linux/android on their hardware.
While the OS landscape has coalesced around this small bunch of SoC vendor-agnostic options, alas the SoC vendors did
not sleep. With the advancement of client security in modern smartphones towards secure boot and secure enclaves
becoming a ubiquituous, mandatory feature for things like DRM and mobile payment, SoC vendors saw an opportunity to lock
their SoC platforms into their own "secure" OS ecosystem. While the SoC is still running linux/android as its main OS,
on almost all current SoCs, underneath that linux layer runs a large, complex stack of vendor-specific, closed source
"trusted" or "secure" OS and applications.
While the technical reason for the existence of these software stacks is primarily DRM and mobile payments, since
they're intricately linked with the SoC's secure boot process, usually they are impossible to remove or substitute even
if you don't want to use DRM or mobile payments at all.
On Qualcomm SoCs, the shape of that software stack is roughly like this: In EL3 you have the "secure monitor", which
handles security state coordination between the secure and normal worlds along with core power management functionality.
In the secure world, you have qualcomm's proprietary trusted execution environment (TEE) stack, which then hosts things
like DRM or mobile payment as applications. On the other side of the divide, in the normal world is where it gets weird.
Not content with placing a large amount of privileged software in the secure world, Qualcomm decided that for secure
boot enabled firmware loading, they want their own custom hypervisor running in the normal world's EL2 instead of KVM.
On older chips, this hypervisor is called QHEE and is closed source, on newer chips it's called Gunyah and is nominally
open source (though good luck getting a PR accepted, finding documentation, or getting it to run on non-Qualcomm
hardware).
From an embedded developer's perspective, this choice seems like the typical case of company ego combined with the
typical captialist approach to capture value at the expense of one's customers. With company ego, I mean that instead of
relying on proven, open source technology that continuously improves to everyone's benefit, Qualcomm NIH'ed its own
hypervisor as if its engineers were any better at this than the greater kernel community. Funnily enough, every such
company thinks they're better than everyone else at this, since Samsung and several others have also reinvented this
particular wheel.
From a commercial perspective, this move is interesting since it is de-facto enforced through the platform's secure boot
guarantees, and through the high cost of bringing up another stack outside of the vendor-supported BSP. Clearly,
Qualcomm (and others) saw this as an opportunity to lock customers into their own, proprietary platform. Despite this
rather sub-optimal state of affairs, outside of Qualcomm platforms, there is some shimmer of hope. Around 2020, Google
introduced a new "protected" KVM mode to the linux kernel that replicates some of the security features Qualcomm's
hypervisors target in the standard linux KVM stack. Thanks to Google pushing pKVM in its mainline android distribution,
pKVM is enjoying first adoption outside of Google. Maybe Qualcomm, too, will eventually see the light and drop their DIY
hypervisor for pKVM, which is both audited better and supported by more software.
Getting KVM to run on QRB2210
With the above background, we can see why using kvm on the Arduino Uno Q's Qualcomm QRB2210 SoC would be difficult. The
CPU is old enough that its stock firmware runs QHEE. In the arduino firmware's eMMC layout, QHEE is provided in the
hyp partition with two copies for the bootloaders A/B fallback boot support. The boot chain on this board is rather
baroque, with Qualcomm assembling a Rube Goldberg machine of bootloaders. First up is PBL, the SoC's ROM loader,
which is burned into the silicon's ROM at manufacture and cannot be changed. PBL loads into XBL initializes the entire
secure world side of the stack, including QSEE (Qualcomm's TEE OS). It also initializes an auxiliary Cortex M3 core
running RPM, the Resource Power Manager, a firmware that takes care of low-level housekeeping tasks like talking to the
SoC's power management IC (PMIC). After finishing intialization, XBL drops from EL3 to EL2 and jumps into QHEE. QHEE
initializes itself, then drops from EL2 to EL1 and jumps into ABL. ABL is a UEFI-compatible android bootloader, and is
likely a vestige of the platform's android lineage. ABL does pretty much nothing interesting, then loads into a u-boot
instance through an android-compatible boot mode. u-boot does nothing interesting either, then loads into systemd-boot
through UEFI. systemd-boot then finally loads the linux kernel and device tree.
Now, the first thing I tried was to simply null out the hyp partition in the hopes that that would make Qualcomm's EL3
bootloader load into uboot at EL2. As it turns out, this doesn't work on at least three levels: First, the earlier
bootloader simply barfs with an error message when it sees an empty hyp partition, subsequently dropping into emergency
download (EDL) factory recovery mode. Note that the bootloader expects a signed hyp image. On this board, secure boot
is disabled and all fuses are left factory unprogrammed, so we're able to use qtestsign to furnish a firmware image
with a header that looks sufficiently like a signature to the bootloader to let us pass. I proceeded to replace hyp
with a dummy partition consisting of nothing but a jump to the next bootloader stage (a Qualcomm UEFI loader), but that
didn't work either. As it turns out, QHEE apparently does some initialization work, likely talking to the closed
source and publically undocumented EL3 firmware, without that the platform simply will not boot. Every attempt at this
ended up with something apparently taking the CPUs into reset milliseconds after loading into the UEFI loader.
Since reverse-engineering the obsolete QHEE binary on this old platform isn't a good use of my time, I proceeded by
simply patching QHEE such that it would leave for the next stage in EL2 instead of EL1. Unfortunately, this didn't work
either. Qualcomm's ABL simply does not support EL2 boot, and just crashes with some (in EL2) invalid accesses to CPU
registers.
Patching QHEE
We found that we can't remove QHEE, but we also can't simply boot through to linux from QHEE at EL2. I wanted to avoid
tampering with the boot chain too much, even though I think most of it is redundant and can be deleted. What I ended up
with was the following: We boot from XBL through QHEE up to the kernel much like the stock firmware does, reaching
u-boot in EL1. However, inside the QHEE image, we hide a small patch that adds a new hypercall. Until this hypercall is
issued later in the boot, the patch stays dormant and QHEE initializes as usual. We then add a second, smaller patch to
the tail end of u-boot that does nothing but issue this hypercall right before jumping into systemd-boot. u-boot invokes
the hypercall at EL1, and the hypercall arrives in QHEE at EL2. Our patch then activates and does a few things. First,
it tears down all runtime configuration QHEE did, basically resetting the CPU's EL2 back to its reset state. It then
returns to the caller, but instead of dropping to EL1, it leaves at EL2. Since the call happens right before u-boot
jumps to systemd-boot, systemd-boot and subsequently the kernel get invoked at EL2. When the kernel finally starts, it
finds itself at EL2 with a clean EL2 state and proceeds to initialize KVM normally, as it would on any other AArch64
platform.