UAA Case Study
Subaru
In the document “Outage at UAA: A Week Without Critical Information Systems” the Executive Summary describes that it is rare for a disk failure to result in an outage at a system level. The term “rare” is subjective and cannot be measured without measuring impact. Is it rare if it only occurs once every year? The answer depends on how much pain is inflicted on the organization when the event occurs. If an organization experiences once a year an enterprise wide service interruption that lasted between a day, then the organization would not consider it a rare occurrence. For those who are accountable to the service outage, they may explain the outage as the result of a rare occurrence to save their credibility (and perhaps their job). The impact resulting from disk failures at UAA had a harsh impact on the organization. Some organizations might not survive a similar outage. In the case of the University of Alaska at Anchorage, comprehending the potential causes and impact of an EVA failure before the fact would have led to better planning for the potential of an EVA failure. A BIA and subsequent BCP would have reduced the effect EVA outage on the entire UAA organization. To further comment on the perspective of a disk failure being rare please note that from an IT operations perspective, I reject the notion a disk failures are rare. Although a hard drive manufacturer may quote meat-time-between-failure (MTBF) in the thousands of hours, hard drive MTBF can vary significantly in a storage array. Drive failures can increase or decrease due to the next firmware revision. When drive failures increase, the storage array vendor/manufacturer ALWAYS recommends a firmware upgrade. This underscores the importance of an experienced IT operations team. Although the Outage document discusses that the CIO is challenged by limited technical personnel available in Anchorage, a more experience SAN engineer would have made better decisions in the days leading up to the EVA failure. The case study states references at article titled “Up in Smoke” (Ross, Tyran, Auer, Junell & Williams, 2005.) that “Many universities in particular lack the both the resources and foresight to develop a disaster plan.” Disaster plans, in its simplest form, can be the tactical situational decisions that a group of IT people make in order to avoid a disaster. If you are a university and lack the foresight to avoid disasters, then you should find the resources to close the gap because the disaster is coming. If you are a CIO for a university in Anchorage running a complex infrastructure, and you have been managing through years of staff cutbacks, then risk of a major outage is high. An experienced IT operations team will go a long way to compensate for weaknesses in infrastructure and planning. The business impact analysis of a failure is ideally planned well in advance of potential failures. However, if you do not have a business impact analysis and continuity plan in advance of such a problem, then do not make configuration changes or upgrades on a mission critical system without taking time to understand the impact of the failure. Despite the EVA being in a fragile state, there was time to step back, comprehend the impact, determine if there were steps that would allow a reduction of the impact. For example, coordinating completion of backups with the start time of reboot or system changes would reduce the recovery point objective. This could have improved their position of the Blackboard recovery. A statement of business impact and back-out plan is a best practice in change management, and if time allows, even during emergency changes. The Outage document indicates that the UAA ITS team had implemented ITIL processes to improve efficiency. Change Management is an ITIL process, thus I question the UAA ITS change management processes and/or the execution of these processes as a probable contributor to the impact of the outage. It’s also notable that the review panel recommended the escalation of incident reports to a more holistic view. This is indicative of poor ITIL problem management processes. On a larger scale, if the UAA had performed an advanced, thorough business impact analysis on a failure of the EVA, they would have been in a better prepared to react to the failure of the EVA. Assuming the BIA would have revealed that the maximum acceptable outage was less than 24 hours, then the outcome of the BIA may have been to purchase the infrastructure redundancy, including a redundant array. However, I agree with Levine (cited in the document) that “a good disaster plan can enable a company to recover despite of financial straits.” Note that redundant infrastructure adds significant complexity to the infrastructure, the expertise to maintain that infrastructure and to fail over elegantly when an outage occurs. A thorough BIA and subsequent BCP would have revealed numerous gaps in maintenance process that, once mitigated, would have reduced the overall recovery time of the failure of the EVA. The subsequent BCP would have required testing of the recovery plan. Ideally this would include regular testing of the EVA, including reboot and test restore. In the case of the UAA, this type of testing would have been performed during school breaks and holidays. Regular testing is also an opportunity to update firmware and apply patches. Recovery testing would also have revealed that the parallel recovery method was not as effective as serial, forcing more focus on business impact and the priority of recovery of services (BIA). An better understanding from the stakeholders would have helped the UAA ITS prioritize the order of service recovery. Through a BIA, the UAA ITS would have had stakeholder agreement that recovery email was the highest priority and should occur as soon as the EVA was restored. An advanced comprehension of a BIA would have revealed to the UAA ITS that they did not have an adequate communication capability to compensate for the potential loss of the email system. CDW used to run a television commercial that joked about sending a company wide-email to notify the “email server is down.” There are numerous ways to compensate for emergency communications in the event of an email service outage, including call trees, text broadcasts and Twitter. IT teams usually improve as an outcome of a disaster. It’s like the old saying, what doesn’t kill you makes you stronger. Comprehending the impact of outages forces better processes for both avoiding and managing disasters, which makes the organization stronger. Experienced IT operations gain critical experience through disaster recovery. IT teams learn the most about the impact of outages because of the outages. The most thorough BIA/BCP will not foresee all that is possible to cause an outage. Consequently, BIA/BCP should improve as the result of outages. We use the anecdote that “always take advantage of a good outage” which means that the pain of an outage will help loosen purse strings and/or make policy modifications to ensure the pain is not felt again. When CIO Whitney meets with the Provost and the Deans, I do not think he should recommend redundant infrastructure at this point in time. His plan for avoiding a similar disruption of service should begin with the following: 1) A thorough BIA/BCP for all services provided by the UAA ITS. The outcome of these may indicate that there is cost benefit to adding additional redundant infrastructure. 2) Improvement in maintenance processes including recovery testing of the EVA and backups. 3) Implement a formal policy for software patching and firmware upgrades. 4) Identify gaps in personnel technical expertise and staff for those gaps. The UAA ITS needs at least one experienced SAN administrator. 5) CIO Whitney should also be prepared to articulate how outages will be communicated to stakeholders and the user community. Good communications during an outage always diminishes the impact of an outage.