Rootkit Threats: Evolution and Mechanism

A rootkit is one or several programs that use stealth methods to hide an intruder or malware in the system. As Windows is the most common operating system, Windows rootkit is an application that hides elements from other programs or the operating system. Also, a rootkit can modify Windows data structures the way to adjust the kernel behavior to achieve the specified goals of masking files and processes.

Despite numerous antivirus-related questions like “Is Norton better than Avast” or “What is the best antivirus,” one can say for sure that modern security applications succeed in rootkit detection and neutralizing. However, lately, rootkit danger has become more significant due to new build-in rootkit schemes within malicious software.

In this regard, one should know what the threat is and how to manage it.

Rootkit Evolution

The term rootkit originated from the Unix systems, where there was a name of a set of utilities used by hackers to hide their presence after gaining initial access. These are, typically, hacker instruments replacing principal Unix utilities. A rootkit lets a hacker take root in a compromised computer system and cover its tracks.

Windows rootkits had a more functional predecessor than UNIX ones. They were stealth viruses for DOS. Stealth viruses showed up in the early 90s. In contrast to UNIX rootkits, whose main task is to let an attacker into the system and mask his actions, DOS stealth viruses just hid from the user and antivirus programs by infecting files.

Early 21st century rootkits fit closely into the widely acknowledged but obsolete definition that a rootkit can operate at the user or the kernel level, based on the modification of the system call chain (Execution Path Modification) or the basis of direct modification of the system data (Direct Kernel Objects Manipulation).

In the mid-2000s, HacDef and Haxdoor accounted for about three-quarters of all existing rootkits. The versatile backdoors Rbot and SdBot were the malware pioneers, considering it was during the implementation of rootkit technologies.

As a result, the rootkit is a relatively old issue that developed together with the utilities and paths. The only thing which gave this malware a future is the mechanism of its work

The principle of operation of the rootkit

To hide some objects in the system, rootkits either intercept kernel system services, similar to how user-mode DLL rootkits act, or modify kernel data formations. According to their principle of operation, rootkits are divided into the following two groups:

• rootkits changing execution path;
• programs that directly manipulate kernel objects.

The first group contemplates using the following techniques:

• interception of functions using the system service dispatch table;
• interrupt interception;
• changing the first machine instructions of the intercepted process;
• changing pointers in kernel objects.

Rootkits of the first group force the thread that called the system service to execute its code. When the rootkit code gains control, it runs the system service, filters the data, and returns the result.

The rootkit of the second type is less noticeable because its code is not directly executed. A rootkit modifies one (or several) kernel data structures, forcing the system to operate on a dummy object (or object queue). These rootkits affect kernel performance indirectly.

Detection and use

Any rootkit technology can be used for both positive and negative purposes. It can be protecting intellectual data, protecting programs from viruses, protecting system files from erroneous deletion, and introducing backdoor programs, spyware, viruses, or Trojans. Fighting malware rootkits is not an easy task with a variety of detection technologies used:

• Search for hooks. It is done by comparing the function address with the range of addresses occupied in the memory by system processes.
• Signature search. This is a search for a chain of bytes in a scanned file specific to a malicious program. It facilitates the identification of the threat so that it can be managed.
• Heuristic or behavioral analyzer. This technology is based on searching for changes in the system settings, the Windows registry, suspicious behavior of processes and modules, etc.

Also, the integrity control feature of checking a checksum or digital signature of system files can help understand if the file has been modified. Nevertheless, specialized security programs and modern antiviruses have enough robust solutions to help users keep their devices rootkit free and clean.