Managing Patches & Updates

Regularly patching and updating software is right on top of

the list of unglamorous must-be-done activities in the otherwise glamorized

world of IT.

Given the ever increasing list of security holes and bugs

in software that is used, this concern of updating the software and then

managing it is actually valid.

Unfortunately, not enough is being done at the right time

to contain the issue, as is evident from the lower satisfaction scores.

Satisfaction with vendors on the issue too remained none to high.

The graph above best illustrates the ground realities when it comes to software patching and update management in Indian enterprises. The graph tracks the responses of CIOs to two questions: How concerned are you about Patching and updating software you run (red) and how satisfied are you with the update process (blue). The lag in satisfaction levels, as compared to the concern is clearly visible even to the disinterested, and encapsulates the challenges faced in keeping applications upto date

Problem statement



Software vulnerabilities have multiplied manifold in the last decade. In

1995, CERT recorded 171 vulnerabilities. In comparison, the number recorded ten

years later, in 2005 is 5990 or a growth of 35 times in ten years!

Vulnerabilities in custom created software are not covered in this count.

As the number of applications run by an enterprise goes up

and as the number of vulnerabilities in each of them grow exponentially, IT

teams are left with the thankless jobs of playing catch up on the job of

patching running systems.

An identified vulnerability leads to a patch. The patch

needs to be applied to all running systems without affecting work and without

the system breaking down. And that is where the problem comes.

Typically, software is customized and tuned to specific

requirements and hardware. Extensions and bridging elements may be added. These

could break when a patch is being applied. So, applying a patch



requires rigorous testing and is a tedious and time consuming task that has to

be undertaken when outside of working hours.

As the number of  vulnerabilities multiply, the IT

team may just not be able to cope up with the patching work, or some systems may

be too fragile to be patched.

Empirical evidence suggests that it is not the

vulnerabilities themselves, but the absence of patching that lead to attacks,

with most attacks and malicious software becoming available long after the patch

for the vulnerability is available.

As the graphic below from CERT shows, there is a lag

between the discovery of the vulnerability and the widespread availability of

exploit tools. Today, software developers have become nimble enough to bring out

patches before such availability of exploits. The onus thus rests on the IT team

to patch the  systems before exploits become available.

Patches come out not just for vulnerabilities, but also for

other software glitches and  minor feature and version



upgrades.

Given the sheer number of patches and the number of systems

on which they need to be applied, IT teams have no option but to resort to some

form of automation for patch management and roll out. This rollout can be done

either by the specific software themselves or by network wide management

software. Software like anti-virus software and Operating Systems, that by

default expect multiple installations, now come with their own deployment

managers that download a single local copy of the patch and then deploys them

over the network, increasing manageability, eliminating user intervention and

conserving bandwidth.

Network management software like IBM's



Tivoli



and Microsoft's SMS Server come with extensive patch deployment and

management capabilities. They also offer the additional ability of being able to

do management tasks like version control and software inventory management.

In this story we look at how Indian enterprises are going

about patching and updating their software and try to identify concerns and best

practices. As is usual with our implementation strategy stories, we polled

twelve CIOs across geographies and verticals for their views and practices.

Vulnerability Exploit Cycle

What to patch?



We identify nine distinct categories of software that need to be patched or



updated on a fairly regular basis. The complete list is given in the graph

below, which charts the priorities of the CIOs across these nine categories. Not

surprisingly, anti virus and firewalls are right on top of everyone's priority

list.  Hardware BIOS's, and appliances and desktop applications figure

lower down the list when compared to others. What seems to be emerging is that

the concern for patching and updating is higher for network centric enterprise

applications and is lower for personal workstations and network devices. While

it is unreasonable to expect the highest priority for every piece of software on

the network, the lower priority for desktop applications in some enterprises

should be a cause for concern, as there are many more applications running on

the desktop and they are thought to be more prone to bugs and vulnerabilities.

The lower priority on appliances is not so much an issue, yet, given the lower

proliferation of appliances. One does get a nagging feeling overall that

currently, desktop may be open to attack, and as appliances proliferate on the

network, more focus needs to go on to them too.

Priority in Patching

Key concerns



The most significant concern of the CIOs polled was version control of

software that they had running. Typically,



organizations end up running multiple versions of the same software because of a

number of reasons. As patches are version specific, and deploying the wrong

patch on the wrong version can have disastrous results, it is only

understandable that version mapping and management is the key concern in

deployment.

Patch Application Timelines

Another concern that was thrown up was impact analysis, or

what exactly will happen once you  deploy the patch. Will it work or will

it break existing systems? It is to understand this that patches are first

applied to test systems before deploying on to production systems. Obviously you

cannot test out every single patch before deployment. So you would need to do an

ABC analysis of running systems to work out which are the ones that will be

deployed only after rigorous testing, which



are the ones that would be deployed directly, but



in stages, and which are the ones that are going to be applied direct, straight

from the vendor's website.

Timelines



There are two issues here. How soon after a patch is released do you apply

it, and how much time do you take to roll it out? Like we saw earlier, there is

a window between the discovery of an exploit and it being widely used by adverse

forces. Within this window, patches have to become available and have to be

tested and deployed. In the case of patches that provide feature



upgrades, this time line is obviously not applicable.

Time taken to Patch

The enterprises we talked to, seemed to be on the ball,

when it came to how soon they applied the patch, with half of them applying

patches within a week of their release and a quarter, even faster!

Time taken to Patch

And for most CIO's, it was a week's work to get the

patch rolled out. That means, taken together, most patches were rolled out

within two weeks of their becoming available. Not bad at all, one must confess!

View Point

Ishwar Jha, Zee Telefilms Ltd Q.What are your key areas of dissatisfaction, with current patching mechanisms? A.Various hardware and software vendors work in hybernated manner and when they provide updates, they don't look holistically for the impact that update can have on hardware and software. Also some vendors do not provide a rollback feature which is a major cause of concern. Q.For custom developed applications and modules, how do you ensure updates and patches?  A.“We first install the software on test environment and check the functioning of custom software and also analyse the changes that needs to be made before deploying the final patch. We also take backup of existing environment before applying any custom patch.”

Whose job is it?



Who is responsible for administering the patches? Half the CIOs polled

stated that it was the responsibility of the user or administrator of the system

in question to keep it up-to-date with patches, and a quarter of them had given

that responsibility to outsourced IT service providers.

Operating Systems Managed

And the remaining had a mix of these two as well as the

vendor's of the respective solutions. Enterprises in this quadrant may have a

task on hand as their systems grow, in clearly identifying the process owner for

each system and in ensuring that everyone delivers up their responsibility.

Tracking which patch has been applied to which system is

critical, particularly in a multi solution or multi version environment. Network

management software like



Tivoli



automatically do this. Where such systems are not in use, the IT teams resort

to manual processes like registers.

Obviously, this is not a desirable situation and is prone

to errors.

Update Management Solution Handles?

Automation



Automated patch deployment is the recommended method of



applying patches in large environments. How many do automated



deployments?

Not enough, by the looks of it. Of the nine classes of

software that we identified earlier on (refer to the Priority in Patching graph

elsewhere in the story), only Operating Systems and anti-virus software have

more CIOs doing automatic patch deployments than the number who don't.

Hardware appliances and firmware figure at the bottom of

the list of automated patch updation. As we go forward, we hope than more and

more systems will come under automatic patch management processes, thereby

reducing the stress and burden on the IT teams. And in Operating systems, Only

Windows and to some extend, Linux deployments seem to be utilizing automatic

patch deployments. This needs to be read with the caveat that we did not query

the respondents on the specific operating system that they run. Three-fourths of

CIO's we spoke to had disabled the users' ability to patch or otherwise

update desktop applications in their organizations. This fits in with the other

trend of desktops getting more rigorously monitored and the users' ability to

install applications becoming strictly limited.

Anil Chopra and Krishna Kumar