Most Linux EDR products now lean on eBPF for their view of the kernel, and a growing number of Linux rootkits are written by people who read the same documentation. Both camps picked the same tool, which sounds like a fair fight until you remember that only one of them gets to decide what the other one sees.

cover

In May I wrote about eBPF rootkits and the blind spot they open in Linux memory forensics, and I closed with a five-step checklist. Checklists are comforting. A few months and some very instructive malware reports later, the more useful question is the one that comes after the checklist: once you accept that your sensor can be lied to, what can you still trust, and how do you get it out of the machine before somebody edits it?

Answering that is harder than it should be, because the word eBPF is currently doing three different jobs, and most of the confusion I see in Linux incident response comes from mixing them up.

In brief

  • Linux EDR products use eBPF for kernel visibility, and the same primitives now power rootkits, which puts the sensor and the implant at the same privilege level.
  • The word eBPF is doing three different jobs: defensive telemetry, offensive abuse and forensic capture, and most confusion comes from mixing them up.
  • Rootkits such as LinkPro and VoidLink intercept the very eBPF functions sensors rely on, so live tools like bpftool and ss cannot be trusted on a suspect host.
  • A memory image recovers the kernel’s raw state, including the prog_idr inventory of loaded programs, bypassing the syscall interface a rootkit filters.
  • Do not reboot before capturing memory, because eBPF implants live in kernel memory and disappear with it.
  • Hardening such as kernel lockdown can block memory acquisition, so the acquisition path must be settled before the incident.

Three jobs, one acronym

The first job is defensive telemetry. Tools such as Falco, Tetragon and Tracee, along with the sensors inside several commercial agents, attach small programs to tracepoints and syscalls, push what they observe into a ring buffer and let a user-space engine decide whether it looks bad. The LinuxSecurity overview walks through that pipeline, and the Falco documentation describes the same model. The promise is attractive: you watch the kernel doing things as they happen, before any user-space trick can tidy up. Defenders even aim the machinery at rootkits, as bpf-hookdetect does when it counts the kernel stack frames between syscall entry and handler to catch Diamorphine-style hooks.

The second job is offensive abuse, and it uses exactly the same primitives. In its taxonomy of Linux rootkits, Elastic Security Labs explains the appeal: no kernel module to load, so no module signing to satisfy and nothing in /proc/modules, and scanners such as rkhunter and chkrootkit walk straight past. The price is that the loader needs CAP_BPF or CAP_SYS_ADMIN and that the programs do not survive a reboot on their own. Ai miei tempi a rootkit announced itself the moment it called insmod. Today it can arrive through a syscall that half the observability stack on the host is already making.

The third job is the one this article is about: forensic capture, meaning the ability to take the state of the machine out of the machine and examine it without asking the machine for its opinion. It is a different question from the other two. Telemetry tells you what happened, as reported by something running inside the suspect kernel. A memory image, if you acquire it carefully, tells you what that kernel looked like at one instant. Telemetry is testimony, a memory image is a photograph, and testimony can be coached.

The sensor and the rootkit live in the same house

The structural problem is that the sensor and the implant run at the same privilege level, inside the same kernel, reading the same data structures. The LinuxSecurity piece reports research on rootkits that intercept the function eBPF programs use to hand events to the ring buffer (bpf_ringbuf_submit), dropping records before the user-space engine ever sees them. The agent keeps running, the dashboard stays green, and the activity simply never happened. An experiment called SPiCa, tested against the Singularity rootkit, proposes the countermeasure that makes sense: a second observer built on CPU non-maskable interrupts, whose heartbeat is compared with what the software telemetry claims. This is experimental work reported by a secondary outlet, so take the details as indicative. The architecture of the problem, though, does not need a research paper to be believed.

Some blind spots do not even require a rootkit. The same Elastic taxonomy describes how io_uring lets a process batch file, network and process operations into a handful of syscalls, which leaves a sensor keyed on per-syscall events looking at a thin slice of the activity. Projects like RingReaper and research from ARMO have already turned that into working tooling.

The deeper gap, I think, is conceptual. EDR is built to narrate events, while eBPF abuse leaves behind state: programs, links and maps sitting in kernel memory long after the event that created them. Nobody inventories state, because nobody was asked to. Loading is the one event that does get recorded. Since a patch by Daniel Borkmann in 2019, the kernel can emit an audit record on every program load and unload (record type 1334), which makes the claim, still repeated in some write-ups, that auditd cannot see eBPF activity out of date. But that record says who loaded program number 76 and when, not what program 76 does. Aqua’s researchers noticed the same limit while building Tracee’s detection, where a bare load event lacked the context to judge it. Datadog’s detection rule exists for that reason: it correlates tampering helpers such as bpf_probe_write_user and bpf_override_return with where the program came from, for example a world-writable directory.

Then there is timing. VoidLink, which I come back to below, waits three seconds after its kernel module loads before installing any hook, which is long enough to outlast a synchronous check of the new module, as Elastic’s VoidLink analysis points out. Elastic also makes an observation that deserves more attention than it gets: eBPF activity is conspicuous on hosts that never use eBPF tooling. On a Kubernetes node running Cilium and a runtime sensor, the same activity is camouflage.

Why the live system makes a poor witness

When a host looks suspicious, the instinct is to log in and ask it questions. That is the weakest option available. LinkPro, analyzed by Synacktiv, intercepts sys_bpf so that its own programs vanish from bpftool prog list. VoidLink goes further. A kernel timer fires every five seconds and walks the process list looking for names such as bpftool, bpftrace, strace, gdb and perf, and Elastic reports that one variant can pause its hiding or self-destruct when it spots them. A vfs_read hook scrubs the rootkit’s name from /proc/kallsyms, /proc/modules and the kprobe list, which is why Elastic’s own detection advice says to validate checks from a trusted environment. Running bpftool on a suspect host is itself a knock on the door.

The eBPF half of VoidLink is a good example of how inventive this tradecraft has become. Its component hooks recvmsg and, when ss queries sockets over netlink, uses bpf_probe_write_user to lengthen the previous message so that hidden entries are swallowed as padding, and ss never parses them. Cross-view checks, such as comparing ss against /proc/net/tcp, can still catch it when the hide lists drift out of sync. They are cheap and worth running, but when the views agree they prove very little.

# cross-view: the netlink view and the procfs view should tell the same story
ss -tnlp
cat /proc/net/tcp /proc/net/tcp6

There is also an uncomfortable trade-off with hardening. Microsoft’s AVML, a popular userland acquisition tool that reads from /proc/kcore, /dev/mem or /dev/crash, states plainly that it cannot acquire memory when kernel lockdown is enabled. Lockdown is precisely what Elastic recommends against this class of malware, and it has been available since Linux 5.4. The better a server is hardened, the harder it becomes to image at three in the morning. The way out is to decide in advance: a signed acquisition module prepared for each kernel, or acquisition from the hypervisor, which is how the Ariel University researchers kept the rootkit away from the acquisition process. In May I mentioned cloud provider memory snapshots. That holds where the platform exposes them, but as far as I know the large public clouds mostly give you disk snapshots rather than RAM, so the acquisition path has to be settled before the incident.

Whatever route you choose, do not reboot first. Datadog’s guidance on its rule makes the point bluntly, since eBPF programs live in kernel memory and disappear with it, and it is the same lesson as the F5 BIG-IP rootkit that lives in memory.

What a memory image can say about eBPF, and what it keeps quiet

The current Volatility 3 documentation lists a mainline linux.ebpf plugin that walks the kernel’s prog_idr and prints address, name, tag and type for each program. That is an inventory and nothing more: no load time, no helper analysis, no JIT dump. The plugin I described in May, Asaf Eitani’s, with helper-based scoring and program dumping, is a community plugin. The same 2023 contest also produced a family of plugins by Valentin Obst for listing programs, links, maps and attachment graphs, as the Volatility Foundation timeline records, and a project called BPFVol3 whose README describes it as alpha and warns against using it in real investigations. The tooling is a patchwork. Validate it against known-good dumps before you rely on it, and keep symbol tables for every kernel build in your fleet, because the mainline plugin gives up on kernels where it cannot find prog_idr.

What you get back, and what limits it, looks like this:

Artifact What it gives you Where it falls short
Programs Inventory of name, tag and type, plus helpers and JIT code with community plugins Names are chosen by the attacker, and nothing says who owns the program
Links Where each program is attached, for example a kretprobe on getdents64 Mostly community plugins, with uneven kernel coverage
Maps The state the program works from The layout is specific to each implant
Companion components Hidden modules and Netfilter hooks that sit next to the eBPF part Separate plugins, separate caveats

Maps are my favorite part. A map is the configuration of the rootkit, sitting in memory. VoidLink’s ss component consults a map called hidden_ports, and its boot script hides port 8080 by default. Dump that map and you hold a list of what the operator cared most about concealing, which is a better lead than many threat intelligence feeds.

Hybrids are the other reason not to stop at the eBPF listing. VoidLink pairs its eBPF program with a kernel module that unlinks itself from the module list and masquerades as the AMD memory encryption driver, so an eBPF-only look finds half the implant. A DFRWS 2025 paper by Roland Nagy on cross-view detection of hidden kernel modules, tested on 55 infected dumps across 27 kernel versions, is the kind of work that fills that half. Volatility already ships linux.hidden_modules, linux.modxview, linux.check_syscall and linux.netfilter, and the last one matters for VoidLink in particular, because its ICMP command channel rides on Netfilter hooks.

Some questions the image will never answer. It cannot tell you whether an unnamed program belongs to a sensor or to an implant without a baseline and an owner to compare against. It cannot tell you when the program was loaded, at least not with the mainline plugin. It cannot tell you what the rootkit did last Tuesday, because that is event history, which is exactly what the sensors were supposed to keep. And if the attacker went after the kernel’s own data structures, the image shows what the kernel believed, which is a better witness than a live shell but not an infallible one.

What I would put in place on Monday

Start with a baseline. For every class of Linux host, record which eBPF programs legitimately belong there, ideally at build time on a known-good kernel, and keep the Volatility symbol tables next to it. An inventory plugin only becomes a detector when something exists to compare it against.

# on a known-good host, at build time, capture the legitimate inventory
bpftool prog list > baseline/ebpf-programs.txt
bpftool map list  > baseline/ebpf-maps.txt

Next, instrument the loader, knowing these are tripwires for the careless rather than walls. Auditd records of type 1334 give you a timeline of loads and unloads, Elastic ships a prebuilt rule for bpftool loads, and the kernel logs a warning when a program using bpf_probe_write_user is installed, which Elastic turns into a syslog detection for VoidLink. A patient implant will route around all three, but plenty of intruders are not patient.

# enable the bpf() syscall audit trail on x86_64 hosts
auditctl -a always,exit -F arch=b64 -S bpf

Then restrict what you can. Setting kernel.unprivileged_bpf_disabled=1 stops unprivileged users, though not the root access most intruders already have, so also constrain CAP_BPF and CAP_SYS_ADMIN, and pair module signing and lockdown with an acquisition path that still works under them. Hardening that blocks your own forensics is a debt you will pay at the worst possible time.

Finally, rehearse capture before containment. Taking the image before pulling the network cable is a decision that costs minutes and is easy to argue about at the wrong moment, which is why I think it belongs in the playbook rather than in the on-call engineer’s judgment, in the same spirit as my argument that incident response is a business decision. With a trusted image in hand, the analysis can happen on a clean workstation:

# on the suspect host: trusted acquisition first, nothing else
# AVML works without lockdown; a signed LiME module covers hardened hosts
avml /evidence/suspect.lime

# on the analysis workstation:
python3 vol.py -f suspect.lime linux.ebpf
python3 vol.py -f suspect.lime linux.modxview
python3 vol.py -f suspect.lime linux.netfilter
python3 vol.py -f suspect.lime linux.psscan

# on hosts with auditd, the load and unload timeline
ausearch -m 1334 --start today

Check Point’s research found that VoidLink was largely produced through AI-assisted development, by a single developer in under a week, which suggests the price of this tradecraft is falling faster than the price of the tools that hunt it. None of the steps above closes the gap. They move the question from whether your sensor saw something to whether you can prove what the kernel held when it stopped being honest, and that second question is one you can prepare for. Keep the sensor, because it remains a perfectly good witness on honest systems. Just make sure that on the day you need it to be believed, somebody has already taken the photograph.

FAQ

Why can a Linux EDR sensor be lied to by an eBPF rootkit?

The sensor and the implant run at the same privilege level inside the same kernel and read the same data structures, so a rootkit can intercept the same eBPF functions the sensor uses to report events.

What can a memory image recover that live system tools cannot?

A memory image captures the kernel’s raw data structures, such as the prog_idr inventory of loaded eBPF programs, bypassing the syscall interface that a rootkit can filter or hide behind.

Should I reboot a suspect host before collecting memory?

No. eBPF programs and the implants that use them live in kernel memory and disappear with it, so memory must be acquired first, before any containment action.