Business continuity no matter what

Leading insurance company Migdal chooses the MedOne[1] data center for hosting its disaster recovery site

When Migdal Insurance Company Ltd. formulated its strategy for business continuity covering all scenarios, with no compromise and no half-measures, it decided to locate its backup site at the MedOne data center[2]  in northern Israel: “Migdal’s strategy is to create a competitive advantage based on the highest possible levels of reliability and availability —levels that generally exceed regulatory requirements — because of our concern and deeply felt responsibility for the rights of the insurees and with the intent of maintaining a high level of service even under disaster conditions,” says Shai Bason, Deputy CEO and head of Technologies and Organizational Infrastructure at Migdal Insurance.

Risk management is the home territory of insurance companies. Their expertise in minimizing exposure to operational, financial, and environmental dangers is what enables them to “invest” in the safety of individuals and of other companies at a price that is worthwhile for their clients — who are unable to handle such risk management similarly by themselves — while still coming out ahead. One significant element in the risk management strategy of insurance companies is strong protection of their IT systems for the sake of ensuring business continuity regardless of what occurs, including major natural disasters, warfare and terror, crime, and human error as well. Regulations set the bar for protection of the insurance companies’ IT systems high enough to ensure continuity of service to the insurees in such cases, but insurance companies that are guided by a strategy of operational excellence are not satisfied with a mere passing grade. “The companies make a big investment of resources in creating trustworthiness and in availability that enables them to continue providing their clients with services under any scenario,” says Ronnie Sadeh, CEO of MedOne. “Our hosting installations in the center and in the north are the only ones in the country that answer the growing need for business continuity assurance and IT system protection at the highest international standards.”

Risk management and black swans

“From the client’s point of view,” says Bason, “the reason for insuring the business is the vital need to prepare a safety net as protection against the possible results of destructive events. Consequently, the most important feature of the safety net is clearly its ability to remain fully functional and continue to work even when such an event damages all the infrastructures. For example, an earthquake insurance policy is worthless if the insurance company’s IT system itself is vulnerable to earthquakes. For that reason, Migdal’s management decided several years ago to overhaul the company’s business continuity program and adapt it to scenarios that are extreme — even if they are also so-called ‘black swans.’ That is, low in probability.”

Experience provides an object lesson: Companies that failed to return to operation within 72 hours of the September 11 emergency (because their offices and their backup systems collapsed in the same complex of buildings) failed to return to normal operation even after two or three years.

“In the 21st century, any business continuity plan must be based on disaster recovery planning. Or for short, every BCP needs DRP,” says Sadeh. “It needs an IT infrastructure in a protected, disaster-resistant environment — an alternative data center built especially to withstand every challenge, natural or manmade. The alternative site can take over production from the main site speedily and safely, with no significant disturbance to ongoing work. Building such sites, operating them professionally, and hosting their main IT systems — the production systems — as well as the businesses’ DRP infrastructures: that’s our specialty at MedOne.”

Obviously, the information processing and storage systems are the “shop floor” of companies in the financial sector as a whole and of insurance companies in particular.

“For us, the survival of historical information in the storage systems is even more critical than for the banks,” says Bason, “because we deal not only with recent transactions. We have policies 50 years old and older that are still a significant part of our clients’ protective deployment, and our clients are our business partners in the fullest sense. We can’t imagine letting them down. We experience history as alive and active in our day-to-day operations, and that’s why it’s supremely important to preserve the information at all costs. On that understanding, Migdal’s management set its goal particularly ambitiously with respect to disaster recovery, with a special focus on storage protection. The model selected was replication and full duplication of the system infrastructures, using two physically separated sites — more than 70 kilometers apart (43 miles) — that operate independently. Each of them can provide full service at a high level of performance without depending on the system that it mirrors.”

A zero-malfunctions operational culture

The program got under way at the start of 2011. During the first half of that year, the requirements were investigated and specified, and in the second half tenders were issued and the secondary site — the one that could take over for the main site on the shortest of notice — was chosen. “Our requirements for the DR site were strict and exacting, because we understood there was no compromising on the quality of the site that would be our refuge in disaster,” says Bason. “We not only demanded resistance to the full spectrum of natural disasters, from earthquakes all the way to floods, but we demanded protection and security against hostile attacks and the dangers of war, we demanded independence from external power sources for at least 72 hours, we demanded real-time 24/7 monitoring, administration and control, and no less important, we demanded a zero-malfunctions operational culture. We chose the site that turned out most suitable, and it was the MedOne hosting site in the north.”

MedOne builds its installations as special-purpose data centers. The company focuses its business on hosting IT systems for clients like Migdal, who are aware of the benefits of complete trustworthiness in business continuity. “Our hosting facilities enable an organization to implement a solution which guarantees business continuity at the highest quality and which will answer the organization’s changing needs in the future,” says Sadeh. “The installations were constructed according to international standards and they are available today to host main production systems in addition to hosting DR sites. Many companies have concluded that economically it’s preferable to rent space and purchase services at our data center sites rather than building their own facilities.”

Even on a small scale, already a tie-breaker

MedOne operates not only a northern site but also two secured data center sites in central Israel (Gush Dan). Altogether, MedOne offers its clients 17,000 square meters (180,000 square feet) of secure, protected underground floor space with electrical backup. The installations can operate autonomously for up to 72 hours with no reliance on the electric company or on other outside infrastructure suppliers. The storage space is divided into “cages” and into “hosting rooms,” which are locked and are set up to host servers and storage systems in all configurations. The sites offer hosting under 24/7 administration, under an outsourcing model; the clients benefit economically from the expertise, the advantage of scale, the improved focusing of their business, and the operational flexibility. “The changeover to a model of renting managed data-center services has been gaining strength dramatically in recent years,” says Sadeh. “Constructing, maintaining, upgrading, and managing data centers is something far removed from the business focus of all our clients. Generally they have no interest at all in owning real-estate assets and certainly not in special-purpose assets for IT storage. Our advantage is great, and the matter of specialization tips the scale in our favor even for a relatively small-scale enterprise because if a business wants to be quick and flexible, a massive investment in a high-quality IT installation is a heavy financial weight to be dragging at its feet.

“The outstanding advantages of having the main IT system hosted at a secure managed data center are reflected in the overall cost of an uncompromising business continuity plan,” Sadeh continues. “If you intend to protect your computer room in all scenarios, including natural disasters on the national level or all-out war, the cost of doing everything internally, by yourself, is very high. For that reason most organizations in Israel can’t set up a BCP solution. Cost/benefit analysis will show them that locating even their production site at an external data center, not only their backup site, is preferable.”

Changing airplane engines mid-flight

Migdal’s IT serves all the company’s 3400 employees, of course, but in addition it has a community of direct and indirect users that is 50 times as large. They include more than 10,000 insurance agents who rely on Migdal’s services for every day-to-day operation from suggesting a policy to determining maturity values, collecting a premium, or making a payment, or handling someone’s temporary disability, plus some 1.7 million individual and corporate clients. “In the total re-hosting of the systems, we managed to return to normal operation within 10 hours, which is less than half the time that the company’s management had specified,” Sadeh recounts.

While the data centers were being constructed as part of the DRP project at Migdal, all the storage, infrastructure, and communications systems were significantly upgraded. A significant part of the solution was the transition to virtualization for 98% of the IT environment, and the transition involved upgrading and replacing the storage and backup systems. At the same time, a robust communications system between the installations was set up, based on dark-fiber connections on two independent channels. “Taken in all, this project changes everything from the ground up. It touches on every aspect of Migdal’s operations,” says Bason. “We haven’t left a stone unturned. We replaced legacy servers with blade servers, we converted the storage farm (totalling 1.3 petabytes) to modern high-end solutions, we moved from an Exadata database platform to Oracle, and we transferred all our application servers from a Unix environment to a virtual Linux environment. What amazes the Migdal management to this day is that it all came and went with zero disturbance to the organization’s ongoing business activity! As an analogy, imagine exchanging two motors on passenger planes, while they were flying, while leaving the passengers — and the pilots — with no feeling that anything unusual was happening outside.”

In November the re-hosting operation concluded for the secondary site, at the MedOne data center in the north, after completion of a series of ultra-realistic fitness tests. “For a DR site, there are two aspects to operational fitness: One is the immediate backup, the mirroring, of every one of the systems by a replication mechanism; and the other is the ability to transfer the company’s operations from the main site to the secondary site in a short time (less than one workday) and almost without losing current information. That is to say, transactions that take place during the hosting operation shouldn’t fall through the cracks,” says Bason. “In our case, that’s while re-hosting a number of separate work environments that all are vital to operations and all dependent on one another. They include VMware on Wintel servers, Solaris on Sun servers, Exadata on Oracle servers, and AS/400 on IBM computers. Our tests proved that a move from site to site can be very quick and the users can remain unaware of any difference at all in performance.”

MedOne Data Center Services

Migdal’s investment in information infrastructures to guarantee business continuity in all scenarios, even the most extreme, did not escape the notice of the company that rates survivability levels in the financial sector. Migdal is Israel’s only company to date that has been rated AAA, the highest level on the scale. “The triple-A rating could be said to reflect the strength of the three pillars in our investment,” says Bason. “One:  We put together a holistic BCP approach of satisfying not merely the regulatory requirements but also the promptings of our own organizational DNA — the drive for excellence in customer service. This approach is expressed in every detail of the solution that we developed together with MedOne. Two: We were the first in Israel to implement innovative technologies that enable a company’s business continuity to be defined as a complete operational solution for uninterrupted functioning of the organization in every challenging scenario. And three: We upgraded and updated the basic hardware and software to simplify administration at the overall system level, by coordinated transfer onto standard virtual platforms that ensure undisturbed business agility and scalability. I’d like to point out that the MedOne engineers were partners in the successful completion of the project. In this success story, we see a combination of high-level professional abilities with technology. Also, in our case, the physical specifications of the data center and the vendor’s business commitment to occupying a central position in the data center sphere were what led us to choose MedOne as host for the Migdal site. But in order to carry out a plan for significantly changing infrastructures, with the IT manager taking responsibility for completion that meets a strict schedule and that doesn’t disturb ongoing work, we needed partners with a well-developed culture of service, not only with appropriate physical resources. I found those partners at MedOne and I can’t but share with them the credit for succeeding.”

“The culture of service that Mr. Bason mentions reflects the basis on which MedOne went into business,” says Sadeh. “We are positioned where four forces meet and intensify the demand for services that host organizational IT systems:

  • Trends in regulation worldwide
  • Israel’s unique geopolitical position
  • Financial value added (replacing capital investment with operational expenses)
  • Increased managerial awareness of crucial dependence on IT systems as a precondition to ensuring business continuity in ordinary times and in emergencies.

“In order to maintain the highest levels of trustworthiness and availability, and to enable our clients to function uninterruptedly in all situations and under all conditions, we constructed the data center to the strictest international standards. Those standards are embodied in everything from the structure itself, through the accessory installations and power system, through to the work procedures and strict security. Of course the human factor is central to the company’s ability to stand by its SLA — its promise to the clients as to availability. We invest in people, not only in equipment, and we develop human resources with a view to the long term. Good, dedicated employees, devoted to service and committed to professional excellence, are what make the difference in quality — and clients like Migdal will testify to the effectiveness of our service-oriented approach.”