Table of Contents
CHAPTER 1 …………………………………………………………………….. 4 INTRODUCTION………..……………………………………………………….4Aim & Objective of Research …………………………………………………..4
Research Questions …………………………………………………………….5
Hypothesis ………………………………………………………………………..5
CHAPTER 2 ……………………………………………………………………...6
LITERATURE REVIEW …………………………………………………………6
Overview of Rootkit Technology ……………………………………………….6
How a Rootkit Functions ….……………………………………………………6
Functionality Approaches of Rootkit ……………………………….…………..7
Types of Rootkit ………………………………………………………………….8
CHAPTER 3 …………………………………………………………………….11
DISCUSSION ………………………………………………………………….11
Rootkit Detection & Prevention ……………………………………………….11
Rootkit Prevention & Removal Approaches …………………………………12
Rootkit Detection & Prevention Tools ………………………………………..13
CHAPTER 4 …………………………………………………………………….14
CONCLUSION……………………………………………………………..……14
CHAPTER 5 ………………………………………………………….…………15
REFERENCES …………………………………………………………………15

EXECUTIVE SUMMARY
This research paper is divided into 5 main chapters like introduction, literature review, discussion, conclusion and references. The major points of this detailed research are summarized as below. * Rootkit technologies cause severe security attacks in today’s cyber world. This research discusses the significance of effective security strategies that should be formulated against security attacks. * Detailed analysis of rootkit technologies and functionalities are done. Different rootkit types, attack tools and approaches are analyzed. * Several rootkit detection & prevention systems are evaluated for preventing these security attacks effectively. Various rootkit prevention tools and methodologies analyzed as well. * This research paper concludes with the significance of effective security tactics against rootkits. Utilization of rootkit technology for legitimate purpose is also recommended.

A CRITICAL ANALYSIS OF ROOTKIT TECHNOLOGIES
CHAPTER 1: INTRODUCTION
Majority of people make use of computer & internet technologies in day to day life. Around 80% of world population use several gadgets and online services nowadays. Obviously, security plays an important role in such a complex hi-tech world. There are several security attacks & malware functions that may causes severe damage to entire computer network. Most of the attackers use highly sophisticated approaches to conduct attack on vulnerable resources (McGraw, 2006). Hidden or undetected attacks are most complex ones since the attacker may use some stealth software here. Rootkit is similar software that helps to carry out cyber attacks in hidden mode. This software can comprise a system through its stealth mode capabilities and backdoor operations (Vieler, 2007). The longer rootkit can comprise a system, the larger the attacker can make damage.
It is not an easy task for any security professionals when they are unaware of the malware techniques. Every cyber security professional should be alert about security related threats & vulnerabilities. In-depth analysis of various rootkit technologies helps them to devise some effective protection mechanisms against these attacks (Carrier, 2005). This research paper discusses about some technological aspects of rootkits, diagnostic approaches towards rootkit attack and its prevention techniques.
Aim & Objective of research
The primary aim of this comprehensive project is to analyse some rootkit technologies for preparing successful security defence mechanisms. This research paper helps to evaluate attacker’s skills & motive and thereby improving the overall system and network security.
Major objective of this research are below. * Study of rootkit technologies and tools. * Analysis of rootkit approaches like stealth mode, data modification, etc. on Linux and Windows platforms. * Study of rootkit detection & preventions tools and methods. * Analysis of rootkit utility for legitimate programs.
Research Questions
Following is a list of research questions that helped in the progress this project during each stage of research. * How a rootkit works? * What are different types rootkit and how do they affect the system? * Which are the main rootkit detection & prevention tools? * What are the different types of approaches for rootkit detection & prevention? * How to increase the system security against rootkit?
Hypothesis
Rootkit technologies have sophisticated hidden attack modes that may result in severe damage. Through proper use this technology can be used for some legitimate applications & programs, and thereby improve the protection strategies.

CHAPTER 2: LITERATURE REVIEW
Literature review discusses about overview of rootkit technology, and its operational aspects in detail.
Overview of Rootkit Technology
Rootkit is the term that indicates a set of software tools that gain the administrative root privilege in an illicit manner (Butler & Hoglund, 2005). Administrative root of a system has greatest significance since it is the most authoritative component of a computer that deals with operating systems. An attacker exploits system vulnerabilities through rootkit so that he/she gets an unauthorized access to the targeted system (Skoudis & Zeltser, 2004). This unauthorized user can modify the security spectrum as well, and hence authorized user cannot understand this security breach easily. Rootkit can gain access to high privileged system processes or files, and modify them. Rootkit replaces the system executable files, and thereby hiding the modifications. Rootkit accesses computer hardware, and eventually take control over operating system. Obscure nature of rootkit functioning makes it more difficult to trace its visibility. Most of the OS security settings and antivirus settings cannot detect any unusual modification caused by these tools (Blunden, 2009).
How a Rootkit Functions
Understanding of rootkit functioning is significant aspect of this research. As the definition says rootkit is a set of software codes or programs which make some undetectable modifications in a system (Butler & Hoglund, 2005). Most of the rootkit follows penetration & modification strategy. Modified files and processes generate an incorrect output. When we scan a system using some antivirus or antispyware software, then system generates an API (Application Programming Interface) call (Vieler, 2007). API calls generally request specific network or OS services. When rootkit access a system, it acts as a filter, and returns a list of unaffected files or services. Hence most of the infected files remain hidden. Several rootkit tools use IAT/EAT hooking functions to create undetectable processes. This helps the attacker to access data without getting noticed.

1.1 Rootkit Infected System

Above picture shows the response of rootkit infected system while an antivirus software scans it.
Functionality Approaches of Rootkit
Rootkit has capability to install ‘backdoor’ in the targeted system so that it can easily gain access (Butler & Hoglund, 2005). It compromises the login mechanism through sneaky login combination. Rootkit mainly uses 3 stages of functionality approaches. 1. Gaining & maintaining access 2. Attacking system and its resources 3. Destroying evidence
Gaining & maintaining access is the first stage in rootkit exploitation. It can be either through local or remote backdoor installation. Remote backdoor installations work in LAN connection only. Following are commonly used remote tool approaches. * Telnet or shell on a TCP port * CGI shell * Secure shell (SSH), regular or trojaned on high port * ICMP telnet * UDP listener * Reverse tunneled shell * Reverse shell/telnet
Rootkit software package consists of some attack tools to conduct successful attacks. Mainly 3 types of attack tools are used in rootkit. * Local attack tools – Local attack tools are used in LAN connections mostly. This attack recaptures the administrative root privilege of a compromised system, and modifies it (Blunden, 2009). Commonly used local attack tools are password sniffers that have capacity to capture text protocols in a LAN. The attacker creates chaos in targeted network, and analyses the transmitted data over network. Detained data is converted into decrypted format using some decryption tools. * Remote attack tools – Remote attack tools include some network noise creating tools like autorooters and scanners. AWU is such a mass exploitation remote tool that creates several noises over LAN perimeters (Blunden, 2009). It identifies the vulnerabilities of a system by utilizing the IP classifications. * DoS attack tools – This tool uses flood attacks to gather sensible information. Reflexive DoS attacks, spoofed flood attacks, DNS response floods, etc. are main tools here.
Destroying evidence is a critical stage of rootkit functioning. This stage eliminates pre-attack evidences, and prevents the formation of any new evidence (Carrier, 2005). It includes destruction of affected log files, audit traces, shell histories and application traces. The simplest method of evidence destruction is log file elimination and editing. Following is an example of rootkit technology that remains hidden while working.

1.2 Hidden Rootkit Program in a System

Types of Rootkits
Rootkits are divided into two categories according to its functionality. User-level rootkits and kernel-level rootkits. Kernel level rootkits are more functional and destructive in nature (Vieler, 2007). Rootkits are categorized as 3 types according to their behavioural changes. They are following. 1. Binary rootkits – These are the first rootkits used to replace system binaries like /bin/login. Generally binary rootkits are used for local or remote access and evidence removal. This rootkit exploits Trojan infected executable files to perform malicious activities. 2. Kernel rootkits – Kernel rootkits are set of malicious modules in kernels that are capable of several system call modifications. It seriously affects the UNIX operating system, since this OS has separate user & kernel mode, and hence it is critically vulnerable to kernel rootkits. 3. Library rootkits – These rootkits are known as Library Trojan rootkits since it uses Trojan infected library functions to conduct the attack (Altholz & Stevenson, 2006). T0rn 8 is a well known rootkit in this category. Library rootkits use some library functions such as libproc.a, glibc/libc, etc. to modify system’s main library functions.

CHAPTER 3: DISCUSSION
Rootkit Detection & Prevention
Rootkit detection is a tedious task. Generally there are 8 types of detection approaches used mainly. 1. Execution Path Analysis (EPA) Approach – EPA technique traps the profile system calls and analyses them. During learning period of EPA, system stores the exact system call counts & code paths. So it can detect the unusual system calls or instructions when rootkit affects. 2. Anti-virus Software Approach – This is the most effective rootkit detection approach. In anti-virus approach, new code paths are stored as new variants. Hence anti-virus software detects the presence of some new variants while rootkit start to operate. This detection approach starts to work at early stages of infections. 3. Differential Testing Approach – Differential testing is a new detection approach, which uses query based handling to detect rootkits (Skoudis & Zeltser, 2004). Similar query manipulations are taking place at each level, and occurrence of dissimilar results indicates the chance of an attack. 4. Host Intrusion Detection Approach – Host intrusion detection uses two layers of detection system against rootkit technologies. Two defensive layers can detect the attacks and prevent further kernel attacks. 5. Signature Based Detection Approach – This detection mechanism make use of a set of already known signatures and patterns for checking unusual behaviours (Altholz & Stevenson, 2006). Signature based approach is fast since it need to analyze a limited no. of signatures only. 6. Anomaly Based Detection Approach – Anomaly based detection approach uses continuous evaluation of system behaviours to find out affected rootkits. It observes behaviour patterns of system calls & files, and compares these results with earlier ones. 7. Virtual Machine Technology Approach – In this approach virtual machine technology is used to facilitate VM introspection capacity. There is a hardware independent host OS to monitor every event in the guest OS. This monitoring process is known as introspection. Presence of virtualization layer in this technology makes it inaccessible to attacker. 8. Automated Defence Approach – This detection approach uses a set of 3 defence mechanisms such as Prevention & Detection, Tracking and Containment (Skoudis & Zeltser, 2004). Automated defense approach consists of protected zone to prevent unauthorized access; efficient tracking systems to maintain steady OS & object relationships; and control mechanism to prevent system damage.
Rootkit Prevention & Removal Approaches
Kernel rootkits act as a biggest threat to technology since they access high privilege administrative root without effortless detection. A successful rootkit prevention approach should take place before the rootkit start to work (Butler & Hoglund, 2005). Obviously, it is a time consuming task that evaluates rootkit execution from its beginning. Following are three significant types of rootkit prevention & removal methods. 1. Signature Based Prevention Approach – This approach is similar to fingerprint comparison method in which each sequence of bytes in a system file is compared with another sequence of bytes in a malicious program (Altholz & Stevenson, 2006). It uses a simple principle; however, it is a tedious task. Signature based approach is quite effective in determining known attacks. Still it has several loopholes for new or unknown attacks. 2. Heuristic Based Prevention Approach – Heuristic based prevention approach uses system behavior comparisons to prevent rootkit attack (Skoudis & Zeltser, 2004). Here evaluates different system behaviors and attack behaviors by comparing the execution paths. Compared to signature based approach, this prevention method has more possibility for false positives. However, this approach is quite effective against latest and unknown threats. 3. Integrity Check Prevention Approach – This approach works best with digital signatures. Here continuously evaluates the trusted base line of some specific system files to ensure security. This prevention approach can effectively prevent some complicated rootkit techniques. System virginity checker and tripwire are two effective prevention tools in this category. 4. Cross View Based Prevention Approach – It is a well known rootkit prevention approach used by several anti-root kit programs. It compares both high level and low level view of a system. If compared results have mismatch content, then it indicates the presence of some malicious activity (McGraw, 2006). This approach can detect hidden programs, files, modified registry keys, etc.
Rootkit Detection & Prevention Tools
There are several rootkit detection & prevention tools available today. Some of them are listed following. * Tripwire * Hijacking anti-rootkit software * Memory shadowing software * VICE * System virginity verifier * GMER anti-rootkit * NICKLE prevention software

CHAPTER 4: CONCLUSION
Rootkits are biggest threat of cyber security world today. Most of the standard detection mechanisms have insufficient capabilities to trace them. Majority of people depend on online services for several transactions and other data services. It is not possible to take some baggy security procedures in such circumstances. Rootkits seriously affects a multinational company or an end-user in similar way. Rootkit technologies like Stuxnet are greatest threat to industries in every way.
A comprehensive knowledge of rootkits is necessary for successful detection as well as prevention. Good understanding of rootkit functionalities enables a security professional to prepare some effective counter measures against these cyber attacks. It helps in successful data leak prevention, efficient data filtering, and effective system monitoring. We can utilize this multi-usage technology for some legitimate programs. Rootkits have several backdoor utilities and monitoring programs that can be used for high-end uses.

CHAPTER 5: REFERENCES 1. Altholz, N., Stevenson, L. (2006). Rootkits for Dummies (pp.142-217). New Jersey: John Wiley & Sons. 2. Blunden, B. (2009). The Rootkit Arsenal: Escape and Evasion (pp. 37-113). United States: Jones & Bartlett Learning. 3. Butler, J., Hoglund, G. (2005). Rootkits: Subverting the Windows Kernel (pp.27- 251). Boston: Addison-Wesley. 4. Carrier, B. (2005). File system forensic analysis (pp. 431-464). Boston: Addison-Wesley. 5. McGraw, G. (2006). Software Security: Building Security In (pp.267-308). Boston: Addison-Wesley. 6. Skoudis, E., Zeltser, L. (2004). Malware: Fighting Malicious Code (pp. 198-312). United States: Prentice Hall Professional. 7. Vieler, R. (2007). Professional Rootkits (pp. 56-97). New Jersey: John Wiley & Sons.