Quantifying Hardware Selection in an FTK 4.0 Environment
Introduction and Background The purpose of this analysis is to evaluate the relative effectiveness of individual hardware component selection in the FTK 4.0 environment. While it is useful to document the individual hardware components which result in maximum performance, it is also important to identify those components which provide the best value. This effort is part of an ongoing commitment by Digital Intelligence to assist customers in making educated choices when selecting individual components for their forensic workstations. Approach Four basic steps were used to evaluate the application’s resource requirements. Step 1 (Establish Test Environment): A suite of tests was developed for the application. These tests were intended to represent the demands of a typical forensic examination. These tests were then automated in order to provide accurate and repeatable recording of results. Step 2 (I/O Channel Evaluation): The automated test suite was then used to determine the basic configuration of the I/O channels. As a starting point, the application manufacturer recommends up to 5 I/O channels: 1.) 2.) 3.) 4.) 5.) Operating System Casework Database Cache/TempDB Evidence
A demonstrated ability to combine two or more of these I/O channels could easily result in a less expensive and more manageable configuration. Evaluation of the I/O channel requirements would be essential in determining an optimal I/O configuration. A baseline system configuration can then be established using this information. Step 3 (Resource Evaluation): Using the baseline configuration, individual components were identified for modification. These components consist of the general hardware options available for system configuration. By limiting baseline modifications to individual components, the relative importance of the associated resources can be evaluated. Step 4 (Potential System Configurations): The final step was to identify and test several cumulative upgrades to the baseline configuration. The value of individual resource modifications, as identified in Step 3, would be essential in determining the hardware combinations to be tested. These hardware combinations would be good candidates for effective workstation configurations.
Methodology A test disk was created with the following attributes: 1.) Contains data which is generalized and varied. 2.) Contains data which is representative of what might be encountered in a typical examination. 3.) Contains data which is significant enough to result in a meaningful processing time. A test suite was developed with the following attributes: • Pre-Processing (factory defaults except as follows): o (Disable SHA-256) o (Enable Duplicate Files) o (Disable dtSearch)
Additional Processing Functions: • dtSearch/Entropy • Data and Meta Carving • Known File Filter A scripting tool was selected and implemented in order to automate the test suite. This tool not only allowed for the automation of testing but also ensured that the individual test times were accurately recorded. AutoIT™ was the tool selected to perform this task (http://www.autoitscript.com/autoit3/). Before beginning each test, an imaging tool (Ghost™) was used to restore disks to their baseline state. This would ensure that all residual data from the previous test would be eliminated including any file-system fragmentation, database fragmentation, or file relocation. The test suite would be run utilizing both compressed (E01) and un-compressed (DD) evidence since both formats are supported by the application. Step 1 (Establish Test Environment) A Windows7(x64) workstation was installed and utilized to create the test disk. Files from the public domain Enron dataset were used to provide email and attachment content. A number of messaging programs were installed and used to simulate “chat” with other users. Additional emails were created and sent with both browser-based and locally installed clients (Outlook). Web browsing was performed. All of these activities were intended to generate content similar to what might be encountered during a typical investigation. The resulting disk image (created with Tableau Imager) consisted of approximately 240 GB of uncompressed data.
The test disk was also imaged using the E01 compressed format the resulting dataset was approximately 60GB. The reduction in overall data output during the imaging process reduced acquisition times significantly. Step 2 (I/O Channel Evaluation) The baseline test system for this analysis was a core i7 system with 16GB memory. Five identical 7200 RPM SATA drives were attached and FTK was configured with 5 separate channels as follows: Separate Channel Configuration Drive Letter Contents C: Win7 x64 Operating System F: Evidence G: Database H: Case L: dbTemp (A sixth MS-Dos partition, Drive E, was also present on the C: disk but was not employed during testing. Drive D was the CD ROM.) The following charts were obtained from Windows Performance Monitor. An analysis of the disk activity indicated that both the O/S and Cache/dbTemp channels, as well as the Evidence and Case channels, could be combined with negligible performance impact. Tests were then run in the Consolidated Channel Configuration to validate this finding. The results showed that the five original channels could be reduced to three channels while only incurring a 2% loss in performance. Reducing the number of I/O channels results in a reduction in complexity, a reduction in cost, and the ability to combine case specific information on a single drive. The resulting configuration is shown below: Consolidated Channel Configuration Drive Letter Contents C: Win7 x64 Operating System and dbTemp files F: Evidence and Case G: Database
The following graphs compare the I/O usage of each of the processing steps with the “Separate Channel Configuration” on the left and the “Consolidated Channel Configuration” on the right.
Pre-Processing: dtSearch/Entropy:
Legend: Lt. Blue (L:) – dbTemp Blue (H:) – Case Green (G:) – Database Red (F:) - Evidence Black (C:) – O/S
Green (G:) – Database Red(F:) – Evidence and Case Black (C:) – O/S and dbTemp
Data and Meta Carving:
Legend: Lt. Blue (L:) – dbTemp Blue (H:) – Case Green (G:) – Database Red (F:) - Evidence Black (C:) – O/S
Green (G:) – Database Red(F:) – Evidence and Case Black (C:) – O/S and dbTemp
Known File Filter:
Legend: Lt. Blue (L:) – dbTemp Blue (H:) – Case Green (G:) – Database Red (F:) - Evidence Black (C:) – O/S
Green (G:) – Database Red(F:) – Evidence and Case Black (C:) – O/S and dbTemp
Step 3 (Resource Evaluation) The results of Step 2 indicated that only three I/O channels would be needed. This helped define the testing matrix for resource evaluation. The following resources were to be evaluated: • • • • • CPU/Processor Memory O/S & Temp Drive DB Drive Evidence and Case Drive
In order to effectively compare two different architectures, a baseline was established for both an Intel i7 and an Intel Dual-Xeon system as follows: Component Processor I7 Baseline I7-3820 3.6 Ghz Quad Core 10MB Cache X79 16 GB 10K RPM SATA 7200 RPM SATA 7200 RPM SATA Dual-Xeon Baseline E5-2609 2.4 Ghz Quad Core (8 cores total) 10MB Cache C602 16 GB 10K RPM SATA 7200 RPM SATA 7200 RPM SATA
Two systems were built utilizing the baseline configurations in the table above. The performance test was run on each and the results recorded. The entire suite of tests would then be run, modifying a single component, in order to quantify the impact of the associated resource on overall system performance. Both compressed and un-compressed evidence was processed. Each system was installed with Microsoft Windows 7 Ultimate (64 bit version) and all patches applied. The Windows Firewall, Search Service, Scheduled Defragmentation, and Windows Update were turned off or disabled. The Auto-IT (scripting environment) was installed and configured. FTK version 4 (4.0.2) and the Postgres SQL database was installed and configured per the manufacture’s instructions. The following tables identify the associated hardware permutations and the resulting impact on system performance:
Resource Utilization Analysis With the Step 3 tests completed, the quantitative impact of hardware selection becomes more obvious. • • • • • Increasing throughput for the Database I/O channel significantly improves performance An increase from 16GB to 32GB of memory improves performance Increasing throughput for the Case and Evidence I/O channel improves performance CPU selection or architecture does not significantly affect performance Increasing the throughput for the O/S and dbTemp I/O channel does not significantly improve performance
Most notably, increasing throughput to the Database showed significant improvements in performance. Additionally, further incremental improvements in throughput appeared to scale accordingly. Increasing memory from 16 to 32 GB resulted in a large performance improvement. This suggests that the application can take advantage of additional memory resources. Increasing throughput of the Case & Evidence I/O channel also showed a performance improvement. However, this performance improvement appeared to be somewhat limited regardless of the storage device employed. This could be an indication that other resources had become the limiting factor. The application performs comparably regardless of the architecture, speed, or number of processors. This strongly suggests that the application is in not processor bound. (Previous tests have also indicated that the forensic process, in general, is not processor bound). Increasing throughput for the Operating System I/O channel did not yield a performance improvement. This suggests that the I/O requirements of this channel can be met by storage device of modest architecture. Step 4 (Potential System Configurations) With a better understanding of application resource utilization, it becomes possible to develop several relevant system configurations. As little benefit could be demonstrated with the upgrade of CPU speed, count, or architecture, further testing would be performed using only an i7 processor system. The single most significant improvement in performance resulted from the increase in throughput of the database I/O channel; specifically with SSD architecture. To further explore this we evaluated:
• •
Standalone SATA Solid State Disk (SSD) PCIe-based Solid State Disk (SSD)
For the Case and Evidence I/O channel, additional storage devices to be tested included: • • • • USB 3.0 Hot-Swap Drive Network-based storage RAID-5 Array PCIe-based Solid State Disk (SSD)
Ultimate performance should not overshadow reliability - especially for the Case and Evidence channel. It should be noted, while unprotected storage environments (like RAID-0) might deliver marginally better performance, a RAID-5 volume is proven to provide critical data protection with only a very small decrease in performance. The same can be said for individual hard drives (including SSD), when considered for use in other storage positions where long term data preservation is also critical.
Description
RAM
OSDrive
DatabaseDrive
EvidenceDrive
Pre-Processing
dtSearch
Carving
KFF
Total
% Improvement from baseline
Base System 32GB + SSD Database + USB3 SATA DATA 64GB + PCIe Database + Raid5 DATA 32GB + SSD Database + Raid5 DATA 32GB + PCIe Database + Raid5 DATA Faster Processor + 32GB + PCIe Database + Raid5 DATA
Optimization Runs E01 Evidence
1600 1400 1200 Time in minutes 1000 800 600 400 200 0 Base System 32GB + SSD Database + USB3 SATA DATA 64GB + PCIe Database + Raid5 DATA 32GB + SSD Database + Raid5 DATA 32GB + PCIe Database + Raid5 DATA Faster Processor + 32GB + PCIe Database + Raid5 DATA
KFF Carving dtSearch Pre-Processing
Analysis of Combined Components An analysis of the results of Step 4 testing confirmed the following resources continued to benefit from further enhancement: • • Increased throughput on the Database I/O channel: PCIe SSD showed improvement over SATA SSD for the Database I/O channel Additional memory: increasing memory from 32 to 64 GB showed improvement with DD Evidence but a slight degradation in performance with E01 Evidence Increased throughput on the Case and Evidence I/O channel: RAID-5 showed improvement over SATA and USB3 SATA Hot Swap for the Case and Evidence I/O channel
•
Accessing the Case & Evidence via a network file server was tested to assess the relative performance of storing the casework and evidence in a centralized location. There are many benefits such as backup management, redundancy, and shared storage, which could mitigate a small reduction in overall performance.
Final Results With over 70 different configurations tested (between the 2 architectures and 2 evidence formats), the relative value of hardware selection becomes more obvious. Significant improvements can be obtained with an optimized choice of components. FTK 4.0.2 benefits from increasing memory, improving the I/O frequency (IOPS) to the database channel, and improving the I/O throughput to the case and evidence channel. There is no significant return on investment in utilizing more than three I/O channels for an FTK 4 system. Additional processor speed, number of cores, or processor cache only improves performance at the very high end of I/O channel improvements. It should also be noted that the Dual-Xeon architecture did not distinguish itself in these tests. Processing compressed evidence (E01) files was significantly faster than processing un-compressed evidence (DD) files. This was likely due to the reduced I/O requirements on the Case and Evidence I/O channel. Systems which will be processing un-compressed evidence will benefit from maximizing memory while systems processing compressed evidence will find memory above 32 GB to be slightly detrimental. In all other aspects, processing compressed or un-compressed evidence had similar resource requirements. With significant reductions in acquisition time, processing time, and reduced storage space requirements, E01 files might be considered a preferred format for evidence.
The following three configurations represent a range of component choices for the FTK 4 environment when processing un-compressed (DD) Evidence: Economy core i73820@3.6Ghz Quad - 10MB 32 GB 10K SATA SATA SSD SATA Mid-Range core i73820@3.6Ghz Quad - 10MB 64 GB 10K SATA SATA SSD USB3 SATA High-End core i73960X@3.3Ghz Hex - 15MB 64 GB 10K SATA PCIe SSD RAID-5
CPU
Memory O/S and dbTemp I/O Channel Database I/O Channel Case and Evidence I/O Channel Time (in minutes)
1078
734
438
The following three configurations represent a range of component choices for the FTK 4 environment when processing compressed (E01) Evidence: Economy core i73820@3.6Ghz Quad - 10MB 16 GB 10K SATA SATA SSD SATA Mid-Range core i73820@3.6Ghz Quad - 10MB 32 GB 10K SATA SATA SSD RAID-5 High-End core i73960X@3.3Ghz Hex - 15MB 32 GB 10K SATA PCIe SSD RAID-5
CPU
Memory O/S and dbTemp I/O Channel Database I/O Channel Case and Evidence I/O Channel Time (in minutes)
545
464
323
Other considerations It should also be noted that raw PC performance is not the only factor to be considered when working to minimize case processing times. Functional convenience can also play a large part in minimizing overall case processing requirements. Little value can be demonstrated if a relatively expensive hardware selection generates a small performance gain but also brings with it significant administrative overhead. Although it has been demonstrated that higher cost fixed disk systems can provide measurable performance benefit, these fixed resources must be re-imaged or recreated each time the contents are to be replaced or updated. Depending on the amount of data involved, this reimaging, recreation, or copying can take a significant amount of time. The resulting managerial overhead might easily be displaced through the use of removable media. As a result, any time advantage seen in the relatively high-cost, high-end solution might quickly be overcome through thoughtful management of casework data. This could easily include the use of paired sets of removable database and case/evidence drives as benchmarked. Similarly, although the location of the case/evidence on high speed network storage resulted in slightly lower performance, the administrative benefit is even higher. Observations and Summary With the completion of over 70 iterations of FTK 4.0.2 benchmarks, a number of interesting observations have been recorded. While many of our observations might be as expected, some were more interesting than others. The following observations appear to be the most relevant when selecting hardware components for processing in the FTK 4.02 environment: • I/O Channel Configuration: The analysis of bandwidth utilization for the 5 identified areas of I/O (O/S, dbTemp, Case, Evidence, and Database drives) supported a reduction in I/O channels to a consolidation of 3 (O/S and dbTemp, Case and Evidence, and Database). Testing demonstrated relatively insignificant change in case processing times while resulting in a much less complicated and expensive solution. Additionally, this configuration also lends very well to simplified case management as it maintains both Casework and Evidence on the same storage device. I/O Component (Drive) Selection: o The Database Drive: Careful selection of the Database drive is proven to be the most important drive choice with respect to performance. Selecting a Database drive capable of supporting a very high level of I/O Operations per Second (IOPS) results in significant performance gains. Solid State Disks are most beneficial in this position with further gains delivered by the PCIe based implementations.
•
o The Case and Evidence Drive: The second most important drive selection was proven to be the Case and Evidence Drive. The Case and Evidence drive is served very well by a storage device of very high I/O Bandwidth (MB/Sec). RAID arrays appear to work very well in this position as they provide very high throughput as well as increased storage capacity. Specifically, RAID-5 volumes delivered performance almost as high as RAID-0 volumes, but have the significant benefit of data protection in the event of drive failure. o The O/S and Database Temp Drive: The choice of the O/S and dbTemp drive appeared to have the least effect on system performance. A relatively inexpensive 10K RPM SATA drive performed well in this position. • System Memory: Increasing the system memory significantly reduced case processing times. However, it should also be noted, that when working with compressed evidence files (E01), an increase beyond 32 GB was actually detrimental to performance. This is particularly noteworthy, as there are significant performance gains which can be had by using compressed evidence drives (as discussed below). CPU: CPU clock rate, number of cores, or multiple CPU architectures did not have a significant impact on processing times until the I/O subsystems were fully optimized. This is due to the I/O bias of case processing tasks. Ultimately, Dual-Xeon systems do not justify added expense over the i7 processor based systems. This is likely a result of newer i7 systems having much more capable I/O subsystems when compared to the more “mature” implementations typically found on Xeon based platforms. Evidence File Format: The difference in performance between processing Compressed (E01) and Uncompressed (dd) file formats was quite significant. As we have seen that the ultimate limitation on processing performance is often the I/O throughput capacity of the system, lessening I/O requirements can be of obvious benefit. By using a compressed data source, we are able to trade some CPU activity in lieu of I/O demands. Additionally, testing has demonstrated that Image decompression is one of the few processing activities which places any significant demands on the CPU. Using a compressed image format quite simply helps offload a portion of the very busy I/O demands onto a much less used CPU resource.
