As part of the Ashen project, I'm writing a linux kernel patchset that quickly erases secret key across the kernel when
a kernel panic is triggered. To verify that I actually erased all copies of the key material, I did some end-to-end
tests where I'd have a test program stuff the kernel full of keys containing a canary pattern, then I'd panic the
system, take a RAM dump and search it for any occurrences of parts of the canary pattern.
During these tests, besides finding a bug or two in my code, I saw the canary in an unexpected place. Even when both the
userspace stuffing tool and the kernel properly wiped all of their copies, I'd still see canary copies in kernel memory.
After some debugging, I found that these bits of key material ended up there in a rather convoluted way. What happened
was that my stuffing tool memcpy'ed a larger buffer with key material from one place to another. My memcpy came from
glibc. When glibc memcpy is run on a CPU that supports vector extensions such as x86-64 AVX or ARM64 NEON, it will use
those vector extensions to accelerate the copy process. The issue arose because after memcpy is done, it leaves some of
the data it copied inside the CPU's vector registers. Since nothing else in the code used these registers, they would
stay unchanged even long after the tool scrubbed the secret data from all buffers.
Now, if the data only stayed in these CPU registers, we still wouldn't see it in a post-mortem RAM dump taken after the
system crashed. What ended up happening for it to show up there was that while the stuffer tool was waiting with its
buffers scrubbed for the system crash, a context switch happened and the kernel switched to another task. When the
kernel switches out one task for another, it saves all CPU state of the old task---including those vector registers---to
the task's management data structure in kernel space. After a while, this copy would get freed, but since that happens
with a plain kfree instead of a kfree_sensitive that also wipes the buffer, I would later find it in the RAM dump.
Background on Vector Extensions
Vector extensions are tightly coupled accelerator cores bolted on to most modern application CPUs that are meant to do
math on many operands in parallel. The acronym SIMD, for Single Instruction Multiple Data, describes the basic approach
of these accelerators: You issue a single instruction, which then instructs the accelerator to perform multiple
calculations on multiple sets of operands in parallel. Common applications for these include accelerating math like for
signal processing or AI inference, and accelerating data-intensive operations like glibc's memcpy used here. Compared
with an external accelerator like a DMA controller or a GPU, these accelerators are controlled through instructions
embedded inside the CPU's regular instruction stream, and the CPU and its vector accelerator can interact for instance
by directly moving register contents between the two, or even sharing certain registers.
The vector accelerator usually has a few wide registers (e.g. 128 bit). The accelerator's operations usually act on the
whole register, treating it as a list or vector of shorter values (e.g. 16 bit). For instance, when doing an ARM
NEON mul v0.4s, v0.4s, v1.4s instruction, it will interpret the v0 and v1 NEON vector registers as containing four
independent 32-bit values, multiply each pair of values from v0 and v1, and store the four results in v0.
On both ARM64 NEON and x86-64 AVX vector extensions, the vector registers are a completely separate register file from
the base CPU architecture's registers. This is important in our case since that means when most of the code isn't using
the vector extensions at all, it won't touch these vector registers either. The platform's Calling Convention
describes how registers are handled during function calls, and it turns out that across calling conventions, most of the
content of vector extension registers is just left alone across function call boundarys. This means that when glibc's
memcpy leaves part of the copied data inside the vector extension registers, when it exits, the compiler won't clean
up this data, neither before leaving memcpy, nor when re-entering its caller.
Vector extensions are only used in performance-critical code. As a result, it's not unlikely that some program doing a
memcpy will not touch the vector register contents leaked by memcpy for a long time.
How vector registers end up in kernel space
The figure above shows the failed scan result that triggered my curiosity. After testing my code, the scan found half of
a key's 32 bytes still intact somewhere in the RAM dump. We can examine where this leak happened by loading the RAM dump
into the crash utility. crash provides a gdb wrapper with bunch of automations for dissecting RAM images and parsing
kernel data structures contained in them. First, we load the RAM dump and locate the leaked data. The constant passed to
the search command here is an 8-byte section of the full 32-byte canary encoded as a little endian integer.
KERNEL: /home/jaseg/proj/ashen-driver/crash-wipe-exercise/ramdumps/qdl-ramdump-20260812-135028/vmlinux
DUMPFILES: /var/tmp/ramdump_elf_xCPstF [temporary ELF header]
DDRCS0_0.BIN
DDRCS1_0.BIN
CPUS: 4 [OFFLINE: 3]
DATE: Wed Aug 12 15:50:30 CEST 2026
UPTIME: 00:26:33
LOAD AVERAGE: 0.07, 0.03, 0.09
TASKS: 277
NODENAME: uno-q
RELEASE: 7.2.0-rc5-next-20260730-test-test-00015-g6e9f56682cc3
VERSION: #103 SMP PREEMPT Wed Aug 12 15:02:35 CEST 2026
MACHINE: aarch64 (unknown Mhz)
MEMORY: 4 GB
PANIC: "Kernel panic - not syncing: sysrq triggered crash"
PID: 0
COMMAND: "swapper/0"
TASK: ffffbdcca6003ec0 (1 of 4) [THREAD_INFO: ffffbdcca6003ec0]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
WARNING: reported panic task ffffbdcca6003ec0 not found
crash> search 0x13799b659e107d38
ffff0000c3604dc0: 13799b659e107d38
crash>
The result is a single hit inside our 4 GiB RAM dump. Next, we check what crash can tell us about the memory allocation
that contains this hit. We can already tell we're somewhere in kernel space, since the address starts with ffff. User
space virtual addresses start with 0000.
crash> kmem ffff0000c3604dc0
CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
ffff0000c0004400 4096 352 368 46 32k kmalloc-cg-4k
SLAB MEMORY NODE TOTAL ALLOCATED FREE
fffffdffc30d8000 ffff0000c3600000 0 8 8 0
FREE / [ALLOCATED]
[ffff0000c3604000]
PID: 1177
COMMAND: "exercise"
TASK: ffff0000c3604000 [THREAD_INFO: ffff0000c3604000]
CPU: 3
STATE: TASK_INTERRUPTIBLE
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
fffffdffc30d8100 103604000 dead000000000400 9 0 bfffe0000000000
As we can see, we're looking at a live kernel object in a cache containing a few hundred objects. The object's
allocation size is 4 KiB, and it starts at address ffff0000c3604000. Next, we'll hexdump the entire object. We'll tell
crash to dump data as 64 bit integers because that makes it easy to recognize memory addresses, which are 64 bit long
and which are usually aligned to 64 bit boundaries on this platform.
crash> rd -64 ffff0000c3604000 0x400
ffff0000c3604000: 0000000000000000 0000000100000002 ................
ffff0000c3604010: 0000000000000003 0000000000000001 ................
ffff0000c3604020: ffff80008fcc8000 0040010000000001 ..............@.
ffff0000c3604030: 0001000000000000 0000000000000000 ................
ffff0000c3604040: 0000000000000030 0000000000000015 0...............
[ ... many lines of pointers interspersed with zeros ... ]
ffff0000c3604bc0: ffffbdcca4d3b458 0000000000000000 X...............
ffff0000c3604bd0: 0000ffff94bbd740 0000000000000000 @...............
ffff0000c3604be0: 0000000000000000 0000000000000000 ................
ffff0000c3604bf0: 656767697254202e 6170206568742072 . Trigger the pa
ffff0000c3604c00: 6e61702065687420 0a2e776f6e206369 the panic now..
ffff0000c3604c10: 6f68203a65736963 657320676e69646c cise: holding se
ffff0000c3604c20: 0000000000000000 ff00000000000000 ................
ffff0000c3604c30: f000000000000000 0000000000000000 ................
ffff0000c3604c40: 00000000c0000003 0000000000000000 ................
ffff0000c3604c50: 2020202020202020 2f20202020202020 /
ffff0000c3604c60: 2f62696c2f727375 2d34366863726161 usr/lib/aarch64-
ffff0000c3604c70: 0000000000000000 0000000000000000 ................
[ ... zeros ... ]
ffff0000c3604da0: 0000000000000000 0000000000000000 ................
ffff0000c3604db0: 0000ffff94bf5000 0000000000000000 .P..............
ffff0000c3604dc0: 13799b659e107d38 9ed66841099b83ba 8}..e.y.....Ah..
ffff0000c3604dd0: 0000ffffc55328b0 ffffff80ffffffd0 .(S.............
ffff0000c3604de0: 0000ffffc55328e0 0000ffffc55328e0 .(S......(S.....
ffff0000c3604df0: 0000000000000000 0000000000000000 ................
[ ... zeros ... ]
ffff0000c3605ff0: 0000000000000000 0000000000000000 ................
What we see is clearly some large struct containing a bunch of pointers, mostly into kernel space, at the beginning
and containing bits of userspace data towards its end. Not only do we see part of our secret towards the end
(13799b659e107d38), we also see a mangled part of a string that the userspace binary prints when it's done (Trigger
the pa the panic now). The pointers starting with ffff 0000 all point into the kernel linear map, i.e. the part of
the VA range where the kernel's dynamic memory allocations live. However, interspersed are some pointers that are likely
to point to addresses in the kernel's code itself. We can recognize these since they start with ffff (the kernel space
half of the 64-bit VA space), but afterwards come four random-looking digits. These random-looking digits are caused by
our kernel having Kernel Address Space Layout Randomization (KASLR) on by default, which randomly shuffles where kernel
code gets put on every boot to make life harder for attackers. Luckily, crash handles this for us, and since my startup
script provides it with the random KASLR offset it is able to reverse this for us. If our kernel had KASLR turned off,
these addresses would look like ffff 8000 instead.
With this info, let's grep the rd output for addresses looking like they point to kernel code and lookup what's behind
them using the sym command.
crash> rd -64 ffff0000c3604000 0x400 | grep -o '\Wffff[89a-e]\w*\W'
ffff80008fcc8000
ffffbdcca3b1bd90
ffffbdcca3b18f24
ffffbdcca5bb8538
ffffbdcca3ad4108
ffffbdcca3b8e210
ffffbdcca60100c8
ffffbdcca3b07250
ffffbdcca3b06c8c
ffffbdcca5f10040
ffff80008fccb800
ffffbdcca4d3b458
crash> sym ffffbdcca3b1bd90 ffffbdcca3b18f24 ffffbdcca5bb8538 ffffbdcca3ad4108 ffffbdcca3b8e210 ffffbdcca60100c8 ffffbdcca3b07250 ffffbdcca3b06c8c ffffbdcca5f10040 ffffbdcca4d3b458
ffffbdcca3b1bd90 (t) dl_task_timer /home/jaseg/proj/ashen-driver/linux/kernel/sched/deadline.c: 1211
ffffbdcca3b18f24 (t) inactive_task_timer /home/jaseg/proj/ashen-driver/linux/kernel/sched/deadline.c: 2150
ffffbdcca5bb8538 (D) fair_sched_class
ffffbdcca3ad4108 (T) do_no_restart_syscall /home/jaseg/proj/ashen-driver/linux/kernel/signal.c: 3186
ffffbdcca3b8e210 (t) posix_cpu_timers_work /home/jaseg/proj/ashen-driver/linux/kernel/time/posix-cpu-timers.c: 1192
ffffbdcca60100c8 (D) init_nsproxy
ffffbdcca3b07250 (t) task_numa_work /home/jaseg/proj/ashen-driver/linux/kernel/sched/fair.c: 4122
ffffbdcca3b06c8c (t) task_cache_work /home/jaseg/proj/ashen-driver/linux/kernel/sched/fair.c: 1867
ffffbdcca5f10040 (D) runqueues
ffffbdcca4d3b458 (T) __switch_to+248 /home/jaseg/proj/ashen-driver/linux/arch/arm64/kernel/process.c: 777
This provides us a bunch of information. Most interesting are the (D) entries, which are pointers to global, static data
structures. Grepping the kernel for where the first of these is assigned to something, fair_sched_class and checking
what it is assigned to already tells us what we're looking at:
bigdata~/p/a/linux <3 git grep -p ' = &fair_sched_class'
kernel/sched/core.c=int sched_fork(u64 clone_flags, struct task_struct *p)
kernel/sched/core.c: p->sched_class = &fair_sched_class;
Thus, we're looking at a struct task_struct, which the kernel uses to track running processes and threads. We can now
let crash show us the memory contents interpreted as struct task_struct by running struct task_struct
ffff0000c3604000 with the memory address taken from the base of the memory allocation output by the kmem command
above. The resulting screenfuls of text contain everything the kernel tracks about this process such as the name of the
process in the comm field:
System Message: ERROR/3 (<stdin>, line 212)
Error in "code" directive:
maximum 1 argument(s) allowed, 8 supplied.
.. code::
crash> struct task_struct ffff0000c3604000|grep comm
comm = "exercise\000\000\000\000\000\000\000",
The leaked data we observed can be found in task_struct.thread.uw.fpsimd_state. Note that we have to add -x here to
force hexadecimal display. Our search value can be found in the third last entry.
crash> struct task_struct.thread.uw.fpsimd_state.vregs -x ffff0000c3604000
thread.uw.fpsimd_state.vregs = {0x6170206568742072656767697254202e, 0xa2e776f6e2063696e61702065687420, 0x657320676e69646c6f68203a65736963, 0xff000000000000000000000000000000, 0xf000000000000000, 0xc0000003, 0x2f202020202020202020202020202020, 0x2d343668637261612f62696c2f727375, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xffff94bf5000, 0x9ed66841099b83ba13799b659e107d38, 0xffffff80ffffffd00000ffffc55328b0, 0xffffc55328e00000ffffc55328e0},
Looking at kernel source in arch/arm64/kernel/fpsimd.c, we can see that the fpsimd_state field is used to store SIMD
register state while the task is not running.
Plugging the leak
With the diagnosis confirmed, what remains is preventing the data from leaking in the first place. The leak
happens through registers being used as they should be under our system's calling convention. As a result, it is tricky
to fix at the source code level. One possible fix is to insert an asm volatile block of assembly instructions that
overwrite all SIMD registers with zeros in places where we know data may leak. This is a tedious and fragile mitigation,
as it requires porting to every target platform, and we need to know in advance which functions can leak data this way.
A leaky memcpy call could be buried several nested function calls deep into a third-party dependency.
Luckily, there is a compiler feature we can use to have the compiler zero out these registers:
When compiling the stuffing code with -fzero-call-used-regs=leafy (or =all), the leak disappears because the
compiler will now insert assembly code to zero unused registers, including SIMD registers, after calling into functions.
A disadvantage of -fzero-call-used-regs is that it comes at some performance penalty. While on x86, zeroing out all
AVX512 registers takes only a single VZEROALL instruction, on ARM64, each of the 32 NEON vector registers has to be
zeroed out in a separate instruction, which will take several cycles even on CPUs with advanced NEON implementations
that can process multiple instructions in parallel. One solution to this is to use the function attribute variant of the
feature, __attribute__((zero_call_used_regs("all"))) specifically on functions where such leaks could happen.
Conclusion
In conclusion, be aware that secrets can leak from memcpy calls to registers long after the original copies are gone,
and that these registers can easily find their way into kernel memory. As a defense in depth measure, consider using
either the -fzero-call-used-regs compiler option or the corresponding function attribute, both of which are supported
by gcc as well as clang.
If you want to replicate this research, hit me up. I'm currently using a hacked up crash version modified to work with
my QRB2210 ARM64 target and a recent 7.2.0 kernel. I'm going to publish this stuff in a nice way eventually, but I can
definitely give you an advance copy if you want to replicate some of these findings.