Strategies for managing storage

Data is the lifeblood of any organization. It is, therefore, important that

this blood is kept clean and free of all impurities, so that it has a smooth

flow and is available as and when required. Sound like something that a healthy

human body should do? Well....isn't your IT infrastructure similar and

complex? It has many different applications that are constantly generating so

much data and working on it. It has a network that must ensure smooth flow of

data across all applications, no matter where they are located. It has various

storage devices to store data, so that data is always available whenever you

need it. If this storage infrastructure is not managed properly, it could have

serious implications for your business. What you need, therefore, is a good

storage management strategy not only to manage the data properly, but also to

ensure business continuity. Storage management should, therefore, be top

priority for any IT manager today. We arrived at this conclusion based on views

of 58% of the respondents of our



survey, who said that storage management was far more important than any other
IT infrastructure management area. Another 25% said that it was equally or more

important than other IT infrastructure management streams.

Key challenges



How difficult or challenging is the task of managing storage? For more than 50%
of our respondents, it was manageable, but another 38% found it to be either

difficult or very difficult.

We can divide the specific challenges themselves faced by the IT managers

them into issues related to data backup, user related problems, data

consolidation, availability and performance. Plus, issues related to managing

data spread across a large geography.

Issues related to backup include reliability of storage media, insufficient

backup windows and speed of restoration should the need arise to do so. There

were also worries related to rapid changes in storage technologies. As tape has

been around for a long time, the associated challenges of backing up data to

tape are fairly well known. There's no single solution to the challenges, but

there are a few guidelines that need to be kept in mind. For instance, when

going for a tape solution, you need to ensure that there's a pre-defined

roadmap for it so that five years down the line, your existing solution will

still work with whatever is available at that time. 

Research Methodology While researching this story, we interacted with 25 users and CIOs across large enterprises to understand their strategies for managing storage. We had futher in-depth sessions with a few others who have been . There are some very clear strategic trends that emerge from this survey, which have been presented in this story. The interpretations of what the CIOs have told us are ours and they are in no way responsible for the same.

How do you manage the ever growing volume of data in your organization?

Likewise, you need to ensure that the vendor will be able to support your

existing backup solution in future as well. It shouldn't happen that there are

no drives available to read the extremely critical data that you've backed up

to tapes today. On the data archival front, there were worries about retaining

data for compliance, identifying and distinguishing the right data that needs to

be archived, and finally retrieving it in case of an investigation. Moreover,

the challenge is not in data retrieval per say, but the speed and accuracy at

which it's done. Here again, policy plays a more important role here than

technology. You need to ensure that whatever you're backing up is checked for

accuracy. You need to perhaps test your backups at regular intervals to ensure

that you don't get a heart attack when you really need to restore the data.

What's your biggest hurdle in ensuring data compliance?

Another key challenge was managing data that's spread across a large

geography. There can be many issues to tackle in such cases. The data could be

residing on multiple platforms and being used by different appls. You might be

adding additional branch offices at a very rapid pace, for which additional

storage capacity needs to be provisioned for, etc. The amount of storage usage

and application performance due to growing data is the other issue faced by CIOs.

As data increases continuously, it's a challenge to ensure that the

applications that use them are able to do so with good response time.

One of the key reasons for so many storage challenges is that the there's

no end to the growing amount of data. It just keeps growing and growing. The

opinion on how easy was it to manage this growth was almost equally split

between being manageable to difficult to manage. 55% of the respondents felt

that this was manageable while another 41% felt that it was difficult. But by

and large, there were two ways in which a majority of the respondents were

managing the growing volumes. One was through storage consolidation, and the

other was to optimize storage utilization by following data management policies.

That is perhaps the best combination to follow today. There might be instances

where adding more storage capacity can help, but that is not the best method

beyond a certain point.

Major and minor pain points



Owing to its vastness, there are lots of management tasks that need to be
prioritized and taken care of. The task that seemed to keep the CIOs busiest was

adhering to statutory requirements for data retention and archival. This was the

issue that CIOs spend most of their time on. This is not surprising as more than

60% of the respondents said that their organization is required to store data

for legal requirements. 

The biggest hurdle to staying compliant appears to be consolidating multiple

islands of data. Some 52% respondents stated this as their biggest challenge. As

30% of the respondents to our survey were from BFSI, they had to follow various

guidelines like the banking regulation act, prevention of Money Laundering Act

as well as RBL and SEBI guidelines. Some CIOs also said that their companies

followed US based and international guidelines such as Basel II, Serbanes Oxeley,

etc. More than 50% of the respondents felt that the Indian IT laws on data

storage for compliance were actually sufficient for their industry. However, an

alarming 36% said that the Indian laws were insufficient. The key issue that

emerged here was a lack of clarity on the IT laws for data compliance. We were

also told that there were contradictions of data requirement under various laws.

There weren't any specific laws defined for data storage, and whatever

landmarks were set were insufficient. Therefore a consolidation of data laws

themselves is required, along with a clearly laid out policy. The next most time

consuming task was managing data security, with 40% of the respondents saying

that most of their time is spent taking care of this. Another 32% of the

respondents said that they spent moderate time on this issue. On the other hand,

email archival, regardless of its high importance in storage management, isn't

really a pain area for CIOs. A majority of the respondents said that they spend

the least time handling email archival woes. The other area that our respondents

didn't spend much time was migrating applications and data from one server to

another. While migration is definitely not to be taken lightly, it also isn't

something that's done frequently.

Existing vs future deployments



More than 60% of the respondents already had centralized their storage
management, and 23% wanted to do it in the near future. Email archival is also

being taken seriously with more than 50% respondents already using it, and

another 23% of the respondents wanting to deploy it in near future. Same

situation prevails for disaster recovery and business continuity planning

solutions, with more than 50% current ownership and another 23% wanting to

deploy it in the near future.

There were more fiber based SAN implementations than IPSAN based, but the

future holds more promise for IP based SANs. Our survey revealed that 27% of the

respondents already had IP based SANs, while another 23% said that they plan to

deploy it in the near future. On the other side, 36% of the respondents already

had fiber based SANs, but only 9% more displayed interest in deploying the same

in the near future.



The area of maximum interest amongst the CIO community was ILM (Information
Lifecycle Management). Though only 9% of the respondents had already deployed

it, another 32% wanted to deploy it in the near future. The next major area of

interest was disaster recovery and business continuity planning implementation.

Interestingly, 52% respondents had already deployed disaster recovery and BCP,

while another 28% said they plan to do so in near future.

Content management, document management, continuous data protection and

e-mail archival were the other three promising areas for the future. There were

24% respondents each for these four areas, who plan to deploy them in the near

future. Content addressable storage or CAS and snapshotting for all their hype

aren't very popular yet. There weren't too many takers for these two

technologies. In fact, there weren't too many current ownerships for CAS.

Storage virtualization for all its hype is still not a priority for most. In

fact, the number who already had it was higher than the number of CIOs who

planned to deploy it in the near future.

Manpower and vendor management



A good 44% of the respondents to our survey said they had dedicated manpower for
managing storage. There were another 36% who didn't have any dedicated staff

for the job, and relatively few had outsourced the same.

Managing such manpower is not easy and the key reason for this seems to be

the rapid advances in storage technology. Every other day, something new and

exciting comes up in storage, promising great benefits to your business. If you

deploy the same, your manpower must have the skills to manage it. Providing

training on the latest technologies therefore emerged as the key concern. The

next biggest concern is finding and retaining dedicated storage professionals.

As technologies change rapidly, storage professionals would want to learn the

same and if the organization isn't able to provide the necessary training,

they would look for better opportunities. The results on vendor related issues

were fairly interesting. Around 36% of the respondents were more than satisfied

by their primary storage vendors. However, another 44% who were moderately

satisfied. That is, they didn't have any major issues with their vendors, but

weren't exactly overwhelmed by their support. So when we asked for specific

issues related to vendor management, hardly any responses came out. There were

stray cases of inadequate onsite support or the technology of data storage being

provided. There's no end to challenges in storage management, as this is one

stream that's under constant development with new technologies evolving every

day. This is understandable considering that the growth in volumes of data in an

organization isn't going to stop or slow down. Organizations will therefore

continuously require new and better technologies to manage them. Therefore, you

need to constantly evolve your storage management strategies to manage this data

and the infrastructure that holds it in place.

View Point: Anand Bhatnagar, Vice President, The Bank of Rajasthan Limited

View Point: Ashish Dandekar, AVP Enterprise Management, NSE.IT Limited

Q: You'd mentioned that you plan to deploy ILM in the near future. What are the benefits you are likely to get by deploying it? If we can't define the life cycle of data, then it will be a never-ending process and data pruning will have to wait for dead end. While presently what we and mostly everyone in the banking industry has adopted is needs based management. While data is mostly backed up to magnetic media, sometimes either due to an application upgrade or non-availability of the specific platform to access that data, the archival (historical data) remains a thing of mandatory requirement only. Hence while accessing the desired data when really needed and also at the same time pruning and destroying the "not so useful" data will save the space and management effort to a great extent. Another dimension is falling of stored data in wrong hands will also be safeguarded, when there are policies and systems in place for the same. Q: You'd mentioned that your biggest challenge in storage management is keeping a balance between the application response time due to an increase data in the application and data availability in the application for making the analysis more meaningful. Can you please elaborate on the same? Basically, we would like to know how you are ensuring good application response time. There is an ever-increasing need of historical data in any organization for analytical purpose. But as the volume of the data increases, efficiency moves in reverse order. So on one side, there is always pressure to retain data that's at least eight to 10 years old, on the other side whatever indexes you have, backup time, data extraction becomes more time consuming That was the reason I mentioned that the biggest challenge is keeping the balance between response time and data availability. Currently we are managing the same by maintaining a parallel system for keeping an archival reporting system, which is accessed by users more frequently. This way we reduce access to such data by users and data also remains in the system for exigencies and analytical uses. Q: How do you ensure high data availability for your applications? High availability of data is ensured by making it available on common storage with servers that are attached to the same in clusters. The implementation of RAID 0+1, a redundant power supply, redundant capacity of hard disks and hot swappable hard disks are some of the measures we use to ensure high data availability in the application. Data replication to DR and also the "archival in making—redo logs" to near line site are also steps in the same direction. Q: What are some of the challenges you face in doing storage consolidation? How are you addressing them? Storage consolidation was considered necessary for ensuring backup at regular intervals and for reducing downtime. This is more complicated when applications are on different platforms and common storage has to integrate with them. We achieved this by making separate virtual volumes for each application on the central storage and while deciding for the backup, the same was taken from the snapshot of the database. Thus while data backup from a single storage could take a backup of multiple applications, application downtime was also reduced considerably. Another dimension to doing data consolidation is for making a common database for multiple islands of databases in the organization. Though the ideal solution for the same is data warehousing and implementing business intelligence tools on the same, but we did not have enough budgets for the same to implement it immediately. Therefore we found a middle path and skipped the data-warehousing step for the time being. We developed separate ETL (Extraction Transformation and Loading) scripts for extracting data from different databases and got the cube developed using SQL 2005 through one of our technology partners. Then by using reporting tools, we generated analysis from this cube and thus the data was utilized from multiple resources.

Q: What are some of the best practices to follow for doing storage consolidation? You need to examine the growth of your data by checking all your historical data. You also need to look at your current storage requirements and projections/analysis of requirements for next 3-4 years. Plus, you need to identify the real business need for doing consolidation. Check whether your organization is subject to Indian or foreign compliance norms. For storage consolidation, you need to do forward thinking and visualize your organization's IT strategy for next 3-4 years, keeping in mind the cost of technology obsolescence. You need to plan & strategize for business continuity, and don't mix it with the implementation time. That needs to be accounted for separately. Q: Suggest a few guidelines for optimizing storage capacity. Verify your business need and firm up on your data requirements. Often, server and storage consolidation become a joint activity. So look for alternates with a competitive techno commercial analysis. Ensure that the device is benchmarked before procurement whenever possible for ensuring smooth compatibility. Consolidate your data requirements and use business requirements as predictions for data growth. Q: What are some of the best practices to follow in storage management? Centralize data and keep the infrastructure in a data center under supervision of a storage custodian (an employee who is responsible for storage in the organization). Follow back-up policy and media redundancy. If tapes are used, ensure testing on a yearly basis as tape is prone to getting infected by fungus. Store the media in fireproof cabinets. If possible, take 2 copies of back up, and store one of them at an off-site location. Sensitive data can be encrypted. The media should be stored in non-condensing but controlled humidity conditions under constant temperature (OEM stipulates temperatures between 16 to 32 degrees C) for better life expectancy. When placing purchase orders, ensure better services by a formal Service Level arrangement/ agreement for ensuring better uptime and problem resolution. Q: What should you look for when choosing a disaster recovery partner? For disaster recovery, it's always preferred to examine what's best for an organization through independent organizations that will help you devise a proper approach and strategy. These organizations follow best practices and have certified professionals to help strategize on DR/BCP. Post strategizing the DR plan & strategy, implementation can be done by an OEM/vendor with the best matching solution.