Mac OS Vulnerabilities: Buffer Overflow
Michael Andrews
POS/355 Introduction to Operational Systems
May 1, 2014
Professor Christopher Warner

Mac OS Vulnerabilities: Buffer Overflow
The risk of a buffer overflow to an operating system is high, especially to operating systems using C language. Based on the UNIX architecture, Mac OS uses C, Objective C, and C++ code as its foundation, primarily for its speed and efficiency. However, this leaves them vulnerable to attacks. Despite its reputation for security, this fundamental operating system security flaw leaves the Mac OS open for external attacks, which can result in theft of user information and corruption of internal systems. However, measures exist to prevent attacks on buffer overflows through code as well as systems built into the Mac OS architecture.
A buffer overflow occurs when more data is put into the buffer than it can hold. This happens when there is not enough room allocated in the buffer and vital program information is overwritten by the new data. Attackers are able to exploit this by taking advantage of a program that is waiting on user’s input. In order to do this, an attacker must know the weaknesses in a program and understand how information will be stored in memory in order to alter the programs execution and gain access to a user’s system. Malware can also be specifically written in order to compromise the integrity of the system. Buffer overflows are the most common way for an attacker to gain access to a user’s system, with the two most common forms being stack and heap overflows.
Stack overflows are the most common form of buffer overflow and subsequently, the most likely for an attacker to exploit. Stacks are divided into units called stack frames, which contain data related to a call for a specific function. The data includes essential information related to the function, like its parameters, and link linking information, like the address that the function will use to execute when it returns. When the stack pointer exceeds its allocated amount of space, essential data in memory can be overwritten. Normally, this would simply cause the program to crash, which can be a problem, but not a security threat. To take advantage of a buffer overflow on the stack, an attacker will intentionally return information longer than the buffer’s size, so the attackers data can overwrite other data on the stack. If the attacker overwrites the return address with their address code, the attackers code can take over the user’s permissions. This attack can allow the attacker to gain control of the user’s permissions, account, and possibly control of the entire computer system.
Buffer overflows on the heap are more difficult for an attacker to exploit, but are also more difficult to detect. Since the heap does not store a return address value and data on the heap varies indistinctly while the program is running, it is less obvious how an attacker can exploit a buffer overflow on the heap. Attackers exploit the overflow on the heap by overwriting critical data. This allows the attacker to crash a program or change a value, like overwriting a stored user ID. The attacker can also overwrite the pointers on the heap when a buffer overflow occurs, allowing the attacker to potentially substitute different or change data. Overflows on the heap are more difficult for attackers to exploit, but preventative and protective measures must be put in place to defend against all forms of buffer overflow attacks.
The first line of defense is for programmers to carefully inspect their code and rewrite unsafe code in order to strengthen the system. Code can be written to check the stackframe for evidence of corruption to avoid buffer overflows on the stack. There are automatic range checks that can be inserted to modify compilers to protect against unsafe pointer references for buffer overflows on the heap. Programs are also available that can do this by placing a value code below the old frame pointer address. If the value code has changed when the transfer returns, the program will be aborted to prevent a potential attackers code from executing. The Mac OS incorporates some defense mechanisms to help prevent buffer overflow attacks. For years, Mac OS has used processors that support the NX bit, which allows the operating system to mark particular parts of memory space as non-executable. The Mac OS uses this by marking the stack and heap as non-executable, so if an attacker places executable code on the stack or heap and the code attempts to run, the program will terminate. The Mac OS also uses Address Space Layout Randomization, which chooses different locations for the stack, heap, libraries, and executable code each time a program is run. This defense mechanism makes the buffer difficult to locate by changing its location in memory. Libraries and other code also change location whenever possible, making it harder for attackers to exploit buffer overflows.
Attacks on computer systems are becoming more frequent and clever, but so are the methods used to defend against such attacks. Despite the inherent vulnerabilities of the Mac OS coding architecture, there are many methods in place to help protect against attacks. While these measures will never be completely effective, when combined with a user who is informed and vigilant, they can help provide the safest computing environment possible.

References
Apple. (2014). Secure Coding Guide. Apple Inc.
Retrieved from: https://developer.apple.com/library/ios/documentation/Security/Conceptual/ SecureCodingGuide/Introduction.html#//apple_ref/doc/uid/TP40002477-SW1
Gagne, G.; Galvin, P.B.; Silberschatz, A. (2013). Operating System Concepts (9th ed.). Hoboken,
NJ:Wiley & Sons, Inc.
Stallings, W. (2014). Operating Systems: Internals and Design Principals. (8th ed). Saddle River,
NJ: Pearson Education Inc.