Some thoughts about Windows Userland Rootkits
Rootkits are tools and techniques used to hide malicious modules from being noticed by system monitoring.
Usually this kind of techniques involves kernel modifications, but (especially on windows systems) appear also in user-mode context, but still enabled to hiding their processes, injected modules, registry keys, files, window, handles etc.
User-mode rootkits are not as stealthy as kernel-mode, but due to their simplicity of implementation they are much more spread: that’s why it is good to know how they works.
The Protection Rings
Protection rings, are mechanisms to protect data and functionality from faultsand malicious behaviour.
A protection ring is one of two or more hierarchical levels or layers of privilege within the architecture of a computer system. This is generally hardware-enforced by some CPU architectures that provide different CPU modes at the hardware or microcode level.
Rings are arranged in a hierarchy from most privileged to least privileged: on most operating systems, Ring 0 is the level with the most privileges and interacts most directly with the physical hardware such as the CPU and memory.
Userland rootkits runs on Ring 3, where user apps run, and since this is where every untrustworthy program runs, operating systems give this layer the least privilege that makes detection much easier using techniques based on heuristic, signatures and anomaly detection.
However, this does not mean that it is simple to detect the userland rootkits: the main goal of a rootkit is hideing itself and sustaining the administrator privileges for it's functioning.
Indeed rootkits need elevated privileges, but they are not the tools that provide the attackers with administrator privileges: this means that before a userland rootkit is entering the system, the attacker have already breached into the system and have performed privilege escalation and finally installed the rootkit, which retains the elevated privilege.
IAT Hooking and Inline Hooking
Userland rootkits uses hooking techniques in order to hide itself, usually IAT Hooking and Inline Hooking.
The Import Address Table (IAT) is comprised of function pointers, and is used to get the addresses of functions when the DLLs are loaded. Applicationa are usually designed so that all API calls will not use direct hardcoded addresses but rather work through a function pointer.
IAT hooking is a technique that malware uses to change the import address table. When a legitimate application calls an API located in a DLL, the replaced function is executed instead of the original one.
In contrast, Inline Hooking modifies the API function itself: the general idea is to redirect a legitimate function to another, so that the malware can perform processing before and/or after the function does its.
The hooks are placed by directly modifying code within the target function, usually by overwriting the first few bytes with a jump: this allows execution to be redirected before the function does any processing.
Hooking Detection
The most used technique in order to hunting userland rootkits is (obviously) the hooking detection: hooking is the main vehicle used by userland rootkits for hiding their presence on a system, so it seems only natural that looking for system hooks could itself be used to identify the presence of a rootkit on a system.
A lot of standard antimalware solutions already support this kind of protection, however sometime could be useful a specific tool that allows the analyst to deep-dive into a process' hooks, like GMER or HookExplorer.