Implementing DR and BCP

After incidents like 9/11, Tsunami and Mumbai Floods-people have understood
in no uncertain terms, the catastrophic consequences of disasters, either man
made or natural. However, to prevent losses from compounding, CXOs have to
figure out ways and means of keeping their core businesses intact in such
situations. So, today if you go for any compliance certification, one of the
pre-requisites is the kind of disaster recovery measures you have in place for
your business. And just because of this reason, most businesses are opting for
DR and BCP policies without even completely understanding the complexities
involved. And still there are lots of misconceptions and myths surrounding DR
and BCP deployments. In this article we try to demystify DR and BCP with some
live case studies and implementation scenarios. But before we begin let's try to
understand why we really need DR and BCP.

Preparing for disasters

Disasters could be natural, in the form of earthquakes, tornados, floods,
etc or they could be manmade such as wars, militant attacks, accidents, etc.
Plus, there is one more category of disasters which occur frequently-biological
pandemics such as Bird Flu, Chicken Gunia, Plague, etc. Now all these disasters
have different characteristics and affect one or more of the four pillars of
Business Continuity-Workplace, Infrastructure, Data, and People. But today most
of us understand DR as just Data Recovery, ie, if your data is corrupted or
lost, you can recover it from some remote storage device. But that's not the
only thing in DR. The consequences of a disaster could be more than just data
loss. So, whenever disaster strikes, it can take away any or all of the four
pillars of your business and your core business could come to a stand still.
Nobody would even want to imagine facing such unfortunate events, but that
doesn't mean they can't occur. A DR and BCP deployment is like a medical policy,
where you keep investing certain amount of money, and you only see the benefits
of the policy when you fall ill. Otherwise, you don't see anything, except money
going out of your pocket as premium. As we don't mind spending money for our
medical insurance and plan for future, we should do the same for our core
business. In a nut shell, a DR and BCP strategy is a medical insurance for your
business. We'll now use a few examples to explain how disasters attack the
various pillars of your business and how you can take effective precautions.
Some of these may sound completely illogical or impossible, but then a disaster
warn you before striking. So do take these examples with that

in mind.

Let's assume that a large banking company runs its core business from a major
city in India. One fine afternoon its network is attacked by cyber terrorists or
there's a virus outbreak. In such a situation, the data integrity is lost. The
easiest way to maneuver this disaster would be to immediately isolate the cyber
attack on the branch and transfer the core job to a DR datacenter hosted at some
other location. This would help users to immediately connect to remote DR
servers and get back to work.

Take another scenario. One day the same city where the bank was operating
from, encounters an epidemic. The Bird Flu virus hits the city, and being an
airborne virus, infects anybody walking out in the open. So a city wide red
alert is sounded, a curfew is enforced, and nobody can come out in the open. In
such a scenario, all your pillars that constitute Business Continuity remain
intact except human resources. So your data, equipment and workplace are intact
but no one can come to the office and operate from there. So, the strategy to
overcome such a problem should be different. Here you must have a DR site with
not only data, but also with a backup of employees who can take over the charge
of the center and finish the tasks from some other city.

Now let's take another example where an earth quake destroys the entire
building, with the data center and all the equipment. Here, even though peoples'
lives might be saved, everything else would get destroyed. In such a situation,
a remote DR site is required where you have all the necessary equipment, seating
arrangements, data and even a recreation zone, where you can fly in your staff
and let them get back to work in as less a time as possible. Such a DR site
should not be in the same geographical location as the site in question, so that
the calamity does not affect both sites at the same time. On the other hand, it
should not be too far away so that it takes a lot of time to fly out people.

Outsourced or In-house?

After reading all these you must be wondering whether you should have a DR
site with redundancy of all for all four pillars of BCP? The answer here should
ideally be yes, for at least the first three pillars of BCP. Human resource is
the only component for which a 100% redundancy is not feasible. There are
ideally two ways in which you can get your DR and BCP site ready. The first and
the traditional one is the in-house model where you have your own premises,
equipment, and data kept offsite, so that in case a disaster strikes, you get
back to that offsite center for DR.

The other and more contemporary way is to outsource Business Continuity to third
party DR and BCP sites. This approach has some real benefits and can save you
huge money and hassles, but both trends have positives as well as negatives. In
the following section we try to discuss the pros and cons of both approaches.

In-house DR

The biggest problem with an in-House model is that if you plan to build a DR
site with 100% redundancy of equipment, data and seats, then it's like having
investing double the capital for the same task. And in case you don't encounter
any disaster for years at a stretch, the ROI for the investment comes out to be
zero. And then, it's not only about building a DR site initially and forgetting
about it completely. You have to regularly monitor, manage and test equipment,
and data kept at the site, so that in case of an emergency you know everything
is working fine. Such a kind of monitoring and management and the associated
drills require a huge amount of investment. So in a nut shell, it's like buying
your own hospital instead of getting a medical insurance done.

There are some good things about an in-house DR site though. For instance, if
security and data theft is a major concern, then you might not even think of
opting for an outsourced solution as you might have reservations in sending your
data regularly to places that are not under your complete control. For
businesses such as stock exchanges, BFSI, etc an in-house model alone can work.

Outsourced DR

On the other hand, in a third party managed model you just have to pay for
the number of seats you hire from the managed service provider (MSP). And you
don't even have to bother about the management and monitoring of the equipment.
All that is taken care by the MSP. So, you just paying for the number of seats
and the investment is pretty less. Now depending on your requirements, you can
even acquire shared or dedicated seats. In case of dedicated sets, you purchase
a particular number of seats and those seats are isolated for ever unless and
until you meet with a disaster and come to the MSP's DR site to use those seats.

However, in case of a shared model, the same seats are sold to more than one
organization. As the number of organizations that hire seats grow, the
investments decrease. If you get, let's say n number of seats for a 1:3 ratio,
this would mean that the same seats would be sold to three organizations. And
obviously, the investment for acquiring those seats would be around one third as
compared to that for a 1:1 ratio. But there's a catch. Let's say three
organizations from Mumbai acquired n number of seats for a 1:3 ratio, at a
nearby third party managed DR site. In case of a disaster, the whole city would
be affected all three companies would get affected. All three companies would
try to reach the same DR site and acquire the n seats. This means they would
require thrice the number of seats actually available. In such a situation all
companies would have to make do with only one third of the resources allocated
to them. This is the biggest disadvantage of a shared model. However, if you
want to keep your investments low then with proper planning and selection of
third party DR sites, you can solve most of these problems and have a fully
functional DR site.

In Citrix XenCenter you can view status of all XenServers running in the resource pool as well as VMs through a single console

Now as we have already talked about what are the reasons and different ways
of having DR and BCP policies. Let's now go a step further and make our hands
dirty to give some of the DR and BCP related tools a try.

Virtualization and BCP

Virtualization has made its way in most of the enterprises, in one way or
the other. Virtualization is also starting to change the way companies do
Business continuity planning. Now, when a server crashes it doesn't necessarily
means that we have to recover whole of the data to a physical server. You can
easily recover a virtual server and continue all your business processes. Many
enterprises are also opting for strategies where they have running replica VMs
of the servers running their critical business applications; so in case of a
server failure, users are automatically switched to the replica VM.

There are many virtualization solutions available which have gone a few steps
further to provide BCP in their unique ways. One such solution is Citrix
delivery center which delivers applications right on your desktop anywhere in
the world. So, the applications can be delivered in India while being run on
some other part of the world. The advantage here is, in case a disaster happens
in the region/country where application was delivered and the whole site gets
wiped out, the users can still access their business applications from their
homes and another office. This also means that you need to create a mission
critical DR site only at the place where you have the application server
running. While at the local site you only worry about the workplace DR. This
means you just need to have separate office building with dumb terminals which
can connect to the main site in case of a disaster at local site. This not only
makes implementation of DR site easy, but also saves good amount of money as you
don't have to take care of data and equipment redundancy at the local site.

Citrix XenServer 4.1 beta

Citrix recently released beta version of Citrix XenServer 4.1, which is a
part of Citrix delivery center. It is a server virtualization platform which
uses Xen hypervisor to enable servers to host virtual machines simultaneously.
XenServer allows users to create a Resource Pool by combining multiple Xen-enabled
servers. Memory, storage, CPU and other resources of this storage pool can be
dynamically controlled and allocated to either running or new virtual machines.
With XenServer administrators can create multiple clusters of Resource Pools and
manage them through a single console. This creates a virtualized data center
environment which not only reduces the complexity of managing server, but also
ensures that you use each server to its optimum level.

Another interesting feature of XenServer is live relocation capability called
XenMotion. When a XenServer host in the pool requires maintenance, VMs running
on that host can be easily relocated to other servers in the pool without
disturbing the state of VM. Similarly, in case of hardware failure, VMs can
automatically move to new resources ensuring that Business critical Applications
are running smoothly at all times. If you are planning to run VMs with Windows
you will require an AMD-V or Intel VT x86-based server with one or up to 32
CPUs. Recommended RAM is 2 GB and disk space of 60 GB. When XenServer is
installed, it creates two 4 GB partitions on the machine; rest of the space can
be used for virtual machines.

Citrix XenServer comes with Xentools, through which you can contantly monitor resources used by VMs running on the pool

Deploying XenServer

For running XenCenter you need a Windows 2003,XP or Vista with .NET
framework 2.0 or above and min 1 GB of RAM. To prepare a resource pool and
enable XenMotion, you will need a shared storage on the network. Installing
XenServer host is simple; it is made of stripped down Xen-enabled Linux OS, VM
templates, a local storage repository for VMs and a management agent. To install
just boot the server with XenServer installation CD, the installer will
automatically detect all hardware present and will ask for the basic information
depending on the configuration of the server. You have to provide things such as
setting root password, DHCP, hostname etc. Once installed the server will reboot
and is ready to be managed through

XenCenter.

After this, you need to install XenCenter on a Windows machine on the same
network. Install XenCenter and then launch it from the Programs menu.

Now, before creating resource pool remember that for a resource pool each CPU
has to be from the same vendor i.e. if you have one AMD-V CPU and second one as
Intel VT then both of them cannot be used together. Resource Pool can be created
on XenServer Host through CLI as well as through XenCenter management console
running on any other host. To create a Resource Pool launch XenCenter from the
programs menu and click on 'Connect to New Server' option. In the window which
pops up, provide Hostname of the XenServer with username and password. Once
connected to the XenServer, click on New Pool option on the menu bar, this will
launch a New Pool wizard where you've to provide a name for the pool and select
the Master XenServer and the slave servers. Once you finish this, the Resource
Pool has been created and you are ready to host virtual machines on this pool.
You can also add Storage Repository to this pool by right-clicking on the pool
you just created and selecting the option New Storage Repository.

To create a Virtual Machine on the pool, from the menu click on New VM
button. This will start New VM wizard, wherein you will found 26 templates for
VMs; if you are planning to run any OS from those templates select that
otherwise select Other Install media option. Further you'll be asked to provide
name, location of source files, and how much CPU and memory should be used for
the VM. Once the wizard gets finished, it will automatically start the VM and
look for installation media on the specified location. After the VM has been
created, you have to install XenServer tools on it to be able to measure its
performance. To install these tools, simply click on the VM name and select the
option Install XenServer tools. Now an installation wizard will start inside the
VM. Just follow the on screen instructions to install the tools. Once installed,
VM will reboot, and now you can monitor performance of this VM from anywhere in
the network through XenCenter.

In CAXosoft you can easily create scenarios for replication of data from man site to DR site

CA XOsoft Replication File Server

CA XOsoft Replication File Server is a real-time data replication solution
for file servers and applications like MS Exchange, IIS, SQL, Oracle etc. It
transfers changes to files instantly to the replica site which can be hosted
locally or over WAN. This solution supports Automatic synchronization i.e. if
replica or the master server reboots, the soln automatically resynchronizes
master and the replica server. It comes with a single management console for
performing all the operations and monitoring the replica as well as master
servers. In case of a disaster, users can either be diverted to replica site or
data can be easily restored to the master through easy wizards. Using the CA
XOsoft Assured Recovery feature of this soln one can perform automated disaster
recovery testing.

Deploying CA XOsoft

The CA XOsoft has five main components namely CA XOsoft Control Service,
Engine, Management Center, PowerShell and CA XOsoft CDP Repository. Deployment
of these components can vary according to the architecture of enterprise
networks and DR needs. When deploying this solution for high-availability
purposes, its control services should not be deployed on master or replica
server. Through this module administrator can control operations of CA XOsoft;
it communicates with engines as well as the managers. It accepts requests from
the Manager module and converts them into commands which are then passed to
engines to perform a particular task. All critical scenario files are maintained
on the server running control services. The soln is also responsible for
authentication of users. Installing part of CA XOsoft is simple; first you need
to install XOsoft Control Service, followed by XOsoft core Engine and XOsoft CDP
Web Server.

After installation, open a browser and type http://host_name:port_no/start_
page.aspx to open CA XOsoft Manager. It will ask for the username and password;
provide the credentials you chose during installation. Now from the Quick Start
pane, click on Scenario Management option. This will start CA XOsoft Manager. To
create a new scenario go to Scenario menu and click on the New option, which
will open Scenario Wizard. The wizard will then ask you to choose the server
type for which you want to create the scenario, and then specify the master as
well as replica hosts. Now, the wizard will verify the connectivity with replica
and master hosts specified and whether the CA XOsoft engine is running on these
hosts or not. If the engine is not detected, you will be asked for the
credentials of the remote machine so that it can install Engine service on the
host. Next choose the files on the master which Replica host has to replicate
and also the location on replica host where data must be backed up. Configure
properties of replica and master and click on 'Run Now' to start the data
synchronization process. After this process finished, a final report will
displaying time taken, files modified, directories created etc. This scenario
can be run any time later on by clicking on Run button and could also be
modified later on as per the needs.

To recover data from a replica, first stop the scenario. Then from the
scenario folder select replica host and from the tools menu select Restore Data.
This will start Recovery Method wizard, provide the information required and
data will get restored on the master side. By default after successful
synchronization XASoft will show a report.

5 Steps to an Effective Disaster Recovery Strategy

The following five strategies can help enterprise IT organizations implement robust high- availability and disaster strategies that maximize system availability for day-to-day operations. Strategy One: Solve problems faster Traditionally, one of the key challenges in executing timely disaster recovery was a delay in alerting IT staff to an outage, and subsequent problem diagnosis. Advanced clustering technology notification and reporting capabilities can pinpoint when an outage occurs, and immediately notifies administrators about the problem. Clustering technology then takes immediate action by starting up applications at the secondary data centers and connecting users to the new data center. Administrators can then use configuration management tools to diagnose the cause of the downtime, such as identifying a change that may have been made by another administrator. The tools can display the nature and time of the change, speeding problem identification, and resolution. When the change is reversed, the normal operating environment can then be restored. With configuration management tools, data center administrators can be confident that their systems can prevent similar outages in the future. Strategy Two: Automate recovery processes For many organizations, system recovery is a manual process. It often requires time-consuming trouble- shooting to identify and solve the problem, and then administrators must rebuild the infrastructure step by step, including restarting servers, installing software, mounting data, starting up and configuring the software, and reconnecting users to the secondary site. Pressure builds on administrators as time, revenue, and customer loyalty slip away, and the potential for human error rises. An automated approach, such as high-availability clustering, eliminates vast amounts of downtime compared to the traditional manual recovery process. If a system fails in the primary data center, the software can restart the application automatically on another server. The administrator may be notified by a text message or via an email, and has visibility into problems at all times, but the series of activities required for maintaining business continuity is handled by the software; with limited action required by IT employees. If a disaster threatens to cripple an entire data center, an automated approach can eliminate human error and reduce downtime by triggering failover of the critical applications to the secondary site. The failover solution should determine which replicated data the application needs to continue operations. Then a single-click starts an automated procedure that restarts the application and connects the users to the secondary site. Automated failover also addresses a common weakness in many disaster recovery plans — the assumption that key employees will be available to physically enter the data center and manually restart applications. If the employees are unavailable, business continuity suffers. Automation helps reduce this potential point of system failure. Strategy Three: Test your DR plan Recent studies have shown that few companies test their DR plans on a regular basis, and as a result, most companies have little faith that their DR plans will work when needed. Companies have been reluctant to conduct DR testing because testing often involves bringing down production systems, mobilizing a large segment of the work force, thus taking them off of more urgent projects, and forcing employees to work during inconvenient hours such as weekends or nights. Anand Naik Director, Systems Engineering, Symantec India With automated failover capabilities, IT organizations can test recovery procedures using a copy of the production data — without interrupting production, corrupting the data, or risking problems upon restarting a production application. This capability means that tests can be run during business hours instead of over the weekend, hence reducing staff overtime. As an added benefit, automated tests run during peak production periods can re-create and approximate the conditions that would occur during a true failover situation. Configuration management tools can also give more confidence to IT managers that their DR plans will work by ensuring that servers at DR sites are consistent with those in production sites. Server builds change over time as patches are implemented or as application dependencies change. This can prevent clustered servers from working properly, as stand-by servers may have not received the latest patch or configuration updates. The latest configuration management tools can run consistency checks that will alert administrators that servers have drifted from the standard build. Action can then be taken to make the appropriate changes and ensure that HA/DR technology will work when called upon. Strategy Four: Extract value from secondary sites For most enterprise IT organizations, secondary sites are viewed strictly as cost centers, sitting idle much of the time. New advances in server provisioning software allow more value to be extracted from secondary sites, enabling them to be used for test development, quality assurance, or even less critical applications. If a disaster strikes and the primary data center goes down, administrators can use provisioning software to automatically re-provision server resources to match the production environment. Advanced clustering software also reduces high cost of the traditional condition that applications must be failed over to the identical hardware that the production applications run on. The most sophisticated clustering software permits failovers between different storage and server hardware within a data center or at remote sites. With the flexibility to dynamically reconfigure and reallocate resources, the secondary site becomes a resource that can be used for multiple purposes the majority of the time, but can be quickly reverted to its backup designation when needed. This underscores the value a secondary data center can deliver, making it more accessible to more companies. Strategy Five: Achieve High Availability and Disaster Recovery in Virtual Environments Server virtualization has become mainstream technology in today's server-centric data center. Server virtualization employs virtual machine technology that allows multiple operating systems to be run on a single server, each functioning independently of the others with its own operating system. Restarting virtual servers at secondary sites has traditionally been a manual process, requiring personnel who may not be available during an actual disaster. New clustering software allows companies to deploy server virtualization technology and receive the same automated disaster recovery benefits they can expect in their physical server environments. Furthermore, new high availability and disaster recovery tools are available that reduce the complexity of protecting and managing both physical and virtual server environments. With clustering software, administrators can fail over applications from physical servers to virtual servers, and manage physical and virtual resources from a single graphical user interface. The result is that, through effective management of physical and virtual servers, hardware costs at secondary sites can be significantly reduced.