White Paper on Memory Usage

6/19/2008

Abstract:
This paper was written to compare customer environments with regards to memory usage for the Interactive Application running on Metaframe servers in the Hosting environment.

Data Information:
A sample group of servers was chosen at random and the following information was obtained from these servers using two methods. Calculations were made from the both groupings of data, each section will show the calculation methods used. As a best practice on physical machines memory utilization should not exceed 90%. The remaining 10% of unused memory is to manage runaway processes, unexpected errors, less utilization of memory paging swap file, and memory available to avoid a system abort.

1. HP OpenView data collection occurred on 6/17/2008 a. Physical Memory as reported by HP OpenView b. Max Memory used at time of sample c. Memory (MB)Avail from Peak d. 90% of available memory for process use e. Total users at time of sample

2. Hyena System Reporting data collection occurred on 6/23/2008 a. System Processes and memory utilization for each b. Application process requirements for and c. Total memory used for system and application processes d. Total users at time of sample

3. Calculations a. Total memory available after 90% ruled applied b. Total memory available after 90% rule applied with system and application processes subtracted c. Average memory used for system processes d. Average memory used for application per customer e. Potential additional user capacity per customer per server f. Potential customer average per server with 90% of memory used

As of this writing its unknown if HP OpenView has the capability to monitor multiple individual processes for reporting purposes.

Data Integrity:
Since both of the data categories were collected at different times, the data may not reflect complete memory usage for the executable processes. The findings in the white paper will provide data for future considerations on deployment of servers in the Hosting environment.

Sample Servers:
Sample servers were chosen at random and for no particular reason other than they host Interactive. Two other servers were chosen in this study because they run the application. The purpose of choosing these servers was to compare the memory footprint of the different applications. All but one of the servers are physical, a virtual production server was chosen to view the memory configuration and memory consumption. The servers are listed in the following section. Also, two Cigna NetworX servers were chosen to compare the NetworX footprint. At the time of the sample it was unknown which processes were required to measure the NetworX footprint. The servers are still in the study but have no application information attached to them. No charts will be made available for the Cigna servers.

This paper can be used in conjunction with the Metaframe usage review written on 4/21/2008 to make recommendations for future considerations about reduction of server farm for particular customers, and increase of virtual server deployments given the technology gains seen in recent versions of VMware ESX server.

Cursory Review of Data:

1. Users per server data was gathered over a 14 day time period with 5 minute intervals set for reporting. Due to limitations with HP OpenView the actual time intervals were 10 minutes and an average number of users within the intervals were reported. Each user count was rounded up to get a better picture of an actual number.

2. Application and System processes were gathered from Hyena system reporting tool.

3. System configuration memory as reported by HP OpenView is not displaying the entire 4 gigabytes of memory which is the standard for Hosted Metaframe servers. There could be a variety of reasons why this is occurring. First, a system with 4 gigabytes of memory should be reporting the number 4096 in several windows of the operating system. The sample group of servers reported the following amount of physical memory in the systems. 3071, 3327, 3583, and 3903, without a physical inspection of these systems though it’s difficult to say that each system has the appropriate amount of memory installed. A possible explanation follows:
The following information was obtained from the following website: http://askleo.com/why_doesnt_windows_show_all_4gigabytes_of_memory_i_have_installed.html *(A quick over-simplification and aside for some: a "bit" is simply a single digit that can contain either a 0 or a 1. Thus when we talk about a "32 bit" operating system or processor, we're talking about systems that operate natively on 32 bits at a time.)
If you look at all possible arrangements of a collection of 32 1's and 0's, you'll find that there are 4,294,967,296 possible combinations.
Computer memory is arranged in bytes, so when you order 4 gigabytes of RAM, you're actually getting 4,294,967,296 bytes of memory. And yes, each byte of memory is assigned its own unique number or "address" - that's how the processor tells the memory hardware which bytes of RAM it wants to operate on.
All's well and good, and you would expect that while a 32 bit operating system would be able to address at most 4 gigabytes of RAM, it seems like it should be able to address all 4 gigabytes of RAM.

Enter the concept of "memory mapped" hardware.
The best, and often the largest example, will be your video card. It typically includes video memory of its own. That memory is "mapped into" or made visible within your PC's 4 gigabyte address space. Say I have a 512 megabyte video card; the memory layout might look something like this:

What you'll notice is that since the video card must place its 512Meg video memory somewhere into the 4 gigabyte address range that your computer can access, it "gets in the way of" 512Meg of your system RAM. That 512Meg of system RAM becomes inaccessible.
Windows works very hard to minimize the impact, and on any system that has less than 4Gig of RAM you'd never notice, since Windows will make sure to put the video and other memory mapped hardware in places that don't conflict with physical RAM. But as soon as you put in 4Gig of RAM that's the maximum a 32 bit system can address and as a result there's no place the memory mapped hardware can hide. It will have to obscure some of that RAM.

4. HP OpenView was unable to obtain utilized memory information from csn-tzg-met-05 which runs the application, the EMS team has been notified and a fix is in the works. Information was gathered from Hyena reporting and will be used in place of any HP OpenView statistics for this study. The application requires the following processes to run for each user: csrss.exe, icpadmin.exe, sfserver.exe, ntvdm.exe, macrun.exe, wfshell.exe, ssonsvr.exe, winlogon.exe.

5. The Interactive application required the following processes to run for each user:

6. All data gathered is from Production systems located at ’s TCS datacenter location. For data comparison purposes, values of High, Low, and average will be used in the comparison charts. All data for this study has been saved and could be made available upon request.

Conclusions: 1. A review or findings study needs to be done to explain the disparity in physical memory reporting through HP OpenView. 2. A complete accounting of all processes running in support of the Windows OS and all installed tools needs to be completed to understand how much baseline physical memory is available to running processes for application use. 3. Further review of HP OpenView’s reporting capabilities needs to be explored so all data gathered is done at the same time, or at a minimum within a 30 minute window. 4. Prior to installation of a new version of , a baseline of process memory used by Interactive needs to done. A calculation at that time could be applied for a starting point for environment build outs for number of users per server. 5. System monitoring needs to be implemented and reviewed each month by support teams for maximum memory utilization. 90% memory utilization should be used as the ceiling. If memory utilization passes the 90% threshold a review of the customer environment should be done to understand the potential impact. Any customer environment that uses substantially less than 90% of the physical memory should also be reviewed as this may indicate an environment that could be collapsed to fewer Metaframe servers. 6. It appears through the data gathered that each customer uses interactive in a unique manner through customization and add on products. Memory utilzation varies from client to client and difference is greater than expected. 7. Using 40 users per server for a client initial build is a good starting point. Had used a higher number (50-55) for the build out phase of the UAT or URN project, more servers would have been needed to be added to accomadate for lack of memory available on the system. • Once a customers implementation is completed, a review should be done to see what memory process utilization is peaking at. At that time a decision should be made to decrease or increase the customer Metaframe environment. 8. Virtual Machines (VMs) are configured with the minimum amount of resources needed to sustain their load and appropriate performance. This eliminates excess resources being allocated allowing those resources to be utilized by other VMs driving up consolidation rates. A minimum of 768M of memory and a single virtual CPU is allocated to a VM. A maximum of 64G of memory and four virtual CPUs can be allocated to a VM. • With less than 768M of memory the Windows OS will tune its caches in a way that leads to bad performance. • A single virtual CPU allocated to VM still has the ability to “float” around any of the physical CPUs and multi-cores. The ESX resource scheduler makes the determination which physical CPU/core the single virtual CPU of a VM executes on. The scheduling happens on a regular basis. • Two or four virtual CPU VMs are only utilized for VMs that run applications that are SMP aware, multi-threaded, and consistently needs more than 100% of a CPU. Configuring a VM with two/four virtual CPUs incurs an overhead with the SMP OS as well as reduced efficiency of the ESX resource scheduler. The ESX resources scheduler has a much harder job of placing a two/four virtual CPU VM on a set of physical CPUs/cores than it does a single virtual CPU. This is due to simply less combinations of CPUs to use when placing a two/four virtual CPU VM to execute on as well as it is harder to find two/four CPUs/cores at the same moment in time available for utilization. The overhead associated with an SMP OS is well documented that each CPU added does not scale in a linear fashion, each incurs an overhead, and leads to reduced efficiency. In addition, the ESX scheduler incurs an overhead that increases with each virtual CPU added. It is therefore an industry best practice as well as a VMware’s best practice to only use multi-virtual CPUs in a VM for applications that are SMP aware, multi-threaded, and consistently need more than 100% of a CPU. This combination offsets the overhead associated with a multi-CPU VM.