Business running again – after 47 hours from the airplane impact at the WTC.
The reason: * A well conceived DRP; * A skillful execution of the DRP, with people working on adrenaline and reacting on instincts.

DRP – based on several mirror sites.
Triangular architecture: WTC – Rochelle Park – London.
The significant loss of lives made recovery from this event especially difficult.

Sources (moving forward): http://www.baselinemag.com/c/a/Business-Intelligence/Pop-Culture/

From day one, Rochelle Park was seen as a concurrent system, not a disaster-recovery site. The shift was driven by eSpeed's role as the largest player in electronic bond-trading, which meant uninterrupted service was an imperative. The nondescript building in a blue-collar town was perfect—a former telecom facility across from another telecom building. Systems alternated between the trade center and the mirror site, with particular products (e.g., zero coupon bonds) running live for a month at one location and then switching to the other; about half of the company's approximately 40 products were live at each location at any given time. "In that sense we had run our disaster-recovery tests the day before," says Noviello.

The mirror site and the World Trade Center were connected by a high-speed optical line, over which eSpeed linked the storage area networks at each site. Sybase data-replication software mirrored critical databases between the sites. Half of the company's Microsoft Exchange e-mail servers were also located full-time in Rochelle Park.

Some DRPs are approved and put in place with the awareness that the plan itself will not work, only for audit purposes.

On Wednesday, Cantor Chairman and CEO Howard Lutnick told him the bond market would reopen the next morning, and asked if the system would be ready; Noviello spoke to Claus and his lieutenants and came back with a yes.
"We never considered not being there on Thursday," he says. "There is too much dedication and enthusiasm in this group. We said we will be there for ourselves and our friends."
Working in the cold, crowded data center, which normally houses no more than a handful of workers, eSpeed's people relied on their knowledge of the systems and procedures instead of following a written plan.
"We had some major hurdles to cross, but we approached them systematically, and each step worked so there was no need for a plan B," says Noviello. "For days it looked like chaos, but people knew what they needed to do." Senior people worked in shifts, and workers took charge and stepped up as needed. It helped that as a small company, people had had lots of exposure to different systems.
!! The impact of not having a disaster recovery plan is clear: two of the four companies that did not have a business continuity plan have not regained their business potential.

Introduction

“Common natural disasters area a leading cause of data loss…” says Bud Stoddard, AmeriVault President and CEO. Events such as the 8.3 earthquake that struck Hokkaido Japan September 25, 2003, and the firestorm in San Diego just weeks ago are just two examples of how natural disasters are devastating millions of unprepared businesses around the world. These are not the only events that must be considered, however. The terrorist attacks against the US on 9/11/01 and the biggest blackout in North American history in August 2003 are examples of man-made disasters.

A 2002 U.S. Bureau of Labor study showed that 93 percent of companies that lose a significant amount of data fold within 5 years. Another survey found that only 60% of businesses have a “credible disaster recovery plan that is up-to-date, tested, and executable”. But just any plan won’t do. “A business recovery plan is a live document, it need to change, evolve, and mature,” says Joe Richardson, executive vice-president of operations and administration at CIT.

In this paper, we will explore the pros and cons of basic prevention, outsourced prevention, and advanced prevention. We will also report on Implementation strategies, how to choose a method, return on investment, planning, and testing. We have also included a case study as an example of why disaster recovery planning is so important to businesses today.

Basic Prevention – Off-Site Tape Back Up

We are going to share a few different ways to help insure that data is safe in case of a disaster. Unforeseeable problems such as fire, floods, viruses, theft, or corruption are just a few of the disasters that can a strike a business. And because data equals money, they need to have their data protected, somehow, someway either backed up and/or replicated off site. According to Rick Lacroix of EMC in Hopkinton, Mass. “Information protection has taken on new importance over the past several years and customers and companies of all sizes are looking for ways to safeguard their information.”

One way to preserve the safety of data is tape backup. There are several software and off-site facilities that will provide a service to customers needing this protection and it is not unusual for an enterprise to spend 25% of its information technology (I.T.) budget on disaster recovery. Many companies have tape backup as their entire disaster recovery plan. For some businesses this is all they need or can afford although this is the least costly of the options presented in this report, it is a risky way of ensuring your data is safe. Unless the media is moved off-site, a disaster has the potential to destroy your backup tape along with your original data. Obviously, this is the least expensive way to protect your data, but one must way the cost of storing data off-site as opposed to the importance of the lost data. Research has shown that more than 80% of the businesses suffering from catastrophic data loss have gone out of business within 12 months. This shows the importance of data and computers in businesses of all sizes.

Outsourced Prevention – Offsite data center

There are many alternatives to data recovery systems. No matter the size, all organizations need to consider having some type of a data recovery system in case a disaster strikes. According to Claude Brazell, U.S. program manager for business-recovery services at HP in Santa Clara, “Responsibility for disaster-recovery planning still falls to I.S. 99% of the Time.” The I.S. organization needs to evaluate and analyze the impact of a loss of company data. This evaluation leads to a plan to protect the company information so that they will be able to recover from a major disaster.

Offsite disaster recovery vendors offer many services and full security of their client’s data. On a recent visit to the SBC disaster recovery center in Irvine, we were able to see first hand some of the different options offered to companies to meet their individual needs. For example, options include office space rental so that a company would have somewhere to work and address critical business issues in an emergency. Clients are allowed 24-hour access to their data and technical support is on available around the clock. Back-up generators supply power in case of emergency and the electricity runs through a power conversion system to ensure its purity. The data center utilizes the latest in fire suppression technology; systems constantly sample the air, monitoring for possible fire or flooding. Depending on severity, either dry pipe suppression or temperature reducing measures will be enacted. SBC also offers service in other geographical areas, allowing a company to distribute risk in multiple areas. It is in a company’s best interest to not keep all of their eggs in one basket and research all options available in order to protect their business data. Prevention is better than cure.

Advanced Prevention – “Hot Site” plus tape

All organizations’ main objective is to resume business as usual within a reasonable amount of time. But why should a company invest in a costly disaster recovery plan? “Nearly one in three companies operate without a formal DR strategy in place” according to Imation Corporation. Natural disasters only account for three percent of incidences reported by BI-Tech. However, since 9-11, the power outages across the nation, viruses, and many natural disasters recently have changed many organizations opinion of having such a plan.

So how does an organization implement a strategy? First management with I.T. must work together and agree that a “hot site” is necessary. The second is to determine a geographic area for this location. The “hot site” must be equipped with servers, networking capabilities, tape and disk storage. This steering committee must be held responsible for the follow through of the DR plan. Frequent tests must be performed to ensure the integrity of the data, software, and hardware. Lastly, document the plan and maintain a copy onsite and offsite the organization.

How much would a plan cost to be implemented? There are several different types of products that can be purchased such as firewalls and external tapes recorders which all vary in cost. Equipment may also need to be purchased to activate the secondary site. In addition to the hardware and software, communications such as cellular phones, satellite phones, and fax, must be taken into account. A secondary power source, such as a diesel generator, is necessary considering the purpose of the structure. There will be several on-going costs like rental space and consulting.

Maintenance is crucial in a successful implementation of a DR plan. Updating software and hardware, periodic testing of the system, procedures, and constant training of employees must continually be visited. No one can predict what may happen but it is always safe of an organization to be prepared for the worst case scenario.

Implementation Strategy

Specific disaster recovery strategies vary by company; the primary reasons are the differences in reliance upon I.T. and the time required to recover in case of a disaster.

One measure of reliance is determining the I.T. operating budget as a percent of the total operating budget. Another measure of reliance is to simply evaluate operations to determine how long it can run without a given computer system. Tape recovery yields the longest amount of time following a disaster, while a hot site would provide the fastest recovery time.

Some organizations spend a great deal of money on business continuity; the nature of the problem prevents any organization from achieving a 100% foolproof solution. However, there are diminishing returns when spending money on disaster recovery. A company can gauge an appropriate amount by doing the following:

1. Listing disasters that might occur based on the geographic location of the I.T. systems, proximity to major cities, and business type

2. Multiply the number of days of downtime based on a given disaster times average revenue lost per day

3. Average those figures

4. Compare against costs of each proposed recovery method

Finances may limit a company to a given budget for disaster recovery; the method above gives a rough idea on the costs of a disaster; since disaster recovery is effectively an insurance policy, managers can weigh the return on investment for each method.

After choosing a given disaster recovery method, I.T. departments must plan for its implementation. This plan should address the concerns of a business in the case of a disaster, plus provide expectations for recovery given a disaster. A good plan should be tested at least annually. Per Sarbanes-Oxley regulations, there are requirements for public companies to test their disaster recovery plans. Testing, while painful, is a necessary part of a good plan. A false sense of security can oftentimes do more damage than not having a plan.

Case Study: Cantor Fitzgerald and eSpeed

The World Trade Center is a complex of seven commercial buildings in New York City that was demolished by a terrorist attack on September 11, 2001. Two companies in particular, eSpeed and Cantor Fitzgerald, L.P., had operations on the 101, 103, 104 and 105 floors. Approximately one thousand people are employed by the two companies at the World Trade Center. The impact at the World Trade Center's North Tower took the lives of 733 Cantor Fitzgerald LP staff, including 150 IT workers. The north tower impact also took out eSpeed's operations on the 103rd floor, which included some 250 Compaq and Sun servers running Windows NT, Alpha/OpenVMS and Solaris. Noviello (eSpeed’s CIO) estimated that 1,000 workstations were destroyed. The floor was also the center of software development activity, and many developers used more than one desktop system, all of which were lost.

Business: eSpeed provides business-to-business electronic marketplace services, primarily in bond trading. The company operates a private trading network, connected to more than 700 financial institutions. Under a joint operating agreement, it also provides all technology services for Cantor Fitzgerald.

Chief Information Officer: Joseph C. Noviello

Financials in 2000: $110.6 million in revenue, with a net loss of $60.4 million

Business: Cantor Fitzgerald, L.P. is a financial services firm with operating units that are involved in a variety of market-based business initiatives including portfolio trading, investment banking, financial spread betting, market advisory, energy brokerage, CO2 emissions trading and electronic trading technology. Cantor benefits institutional investors and traders around the world.

eSpeed's systems were built on a dual architecture that replicated all machines, connections and functionality at the World Trade Center and at a Rochelle Park site, with a third facility in London. The mirror site and the World Trade Center were connected by a high-speed optical line, over which eSpeed linked the storage area networks at each site. Sybase data-replication software mirrored critical databases between the sites. Half of the company's Microsoft Exchange e-mail servers were also located full-time in Rochelle Park. Systems alternated between the trade center and the mirror site, with particular products running live for a month at one location and then switching to the other; about half of the company's approximately 40 products were live at each location at any given time. "In that sense we had run our disaster-recovery tests the day before," says Noviello.

No firms suffered worse fates on Sept. 11 than Cantor Fitzgerald and its electronic marketplace unit, eSpeed. More than 700 employees of the two companies died in the destruction of the World Trade Center's north tower, where Cantor and eSpeed shared their headquarters and a vital computer center. Yet eSpeed was up and running when the bond market reopened at 8 a.m. on Sept. 13, a little more than 47 hours after the disaster. That was possible in part because of some lucky timing. But the rapid response was due to careful planning and help from other companies. Their size was a factor, as well. Different people had had lots of exposure to different systems.

Thanks to planning, the company can keep operating, even if something should happen to Rochelle Park. In the future, its data center in London will serve as the mirror site. Going forward, the company's systems should be even more resilient. “We are learning a lot of lessons as we are restoring the system,” says Noviello, including how to automate more aspects of bringing systems back up. "And we are not restoring our bad habits."

There's no luck to Cantor Fitzgerald's being in business today. It was a matter of planning and hard work. "People plan for disaster recovery. No one ever plans for the loss of equipment, facilities and resources. To bring up our systems in 47 hours demonstrates our planning toward building concurrent systems," said Noviello, adding, “we did not lose any data”.

Conclusion

In summary, careful planning and diligence in plan execution are necessary to implement a business continuity strategy. The expenses associated with carrying out DR plan vary with each organization; as long as companies rely on computers, there will be a need to implement a disaster recovery plan.

• Computer Weekly, 9/9/2003, Anthony Adshead.

• http://encarta.msn.com, © 1997-2003 Microsoft Corporation

• http://interchange.novastor.com/datasheet/tapecopy.html

• http://www.computerwork.com/securitytopics/security/recovery

• http://www.horizons.bc.ca/support/disasters.html

• http://www.simplesan.com/benefits/dr.htm.

• http://www.swc.com/new/articles/disaster.html

• Information Management Journal, November / December 2003.

• Richardson,,Joe: Executive Vice President of Operations and Administration at CIT.,The Secured Lender, Disaster Recovery Planning are you prepared?, Mark Elmerick.

• Stoddard, Bud: AmeriVault President and CEO, Information Management Journal, November / December 2003.

• The Secured Lender, “Disaster Recovery Planning are you prepared?”, Mark Elmerick.

• “World Trade Center,” Microsoft® Encarta® Online Encyclopedia 2003