Managing Patches & Updates

Friday, May 05, 2006

Regularly patching and updating software is right on top of the list of unglamorous must-be-done activities in the otherwise glamorized world of IT.

Given the ever increasing list of security holes and bugs in software that is used, this concern of updating the software and then managing it is actually valid.

Unfortunately, not enough is being done at the right time to contain the issue, as is evident from the lower satisfaction scores. Satisfaction with vendors on the issue too remained none to high.

The graph above best illustrates the ground realities when it comes to software patching and update management in Indian enterprises. The graph tracks the responses of CIOs to two questions: How concerned are you about Patching and updating software you run (red) and how satisfied are you with the update process (blue). The lag in satisfaction levels, as compared to the concern is clearly visible even to the disinterested, and encapsulates the challenges faced in keeping applications upto date

Problem statement
Software vulnerabilities have multiplied manifold in the last decade. In 1995, CERT recorded 171 vulnerabilities. In comparison, the number recorded ten years later, in 2005 is 5990 or a growth of 35 times in ten years! Vulnerabilities in custom created software are not covered in this count.

As the number of applications run by an enterprise goes up and as the number of vulnerabilities in each of them grow exponentially, IT teams are left with the thankless jobs of playing catch up on the job of patching running systems.

An identified vulnerability leads to a patch. The patch needs to be applied to all running systems without affecting work and without the system breaking down. And that is where the problem comes.

Typically, software is customized and tuned to specific requirements and hardware. Extensions and bridging elements may be added. These could break when a patch is being applied. So, applying a patch requires rigorous testing and is a tedious and time consuming task that has to be undertaken when outside of working hours.

As the number of  vulnerabilities multiply, the IT team may just not be able to cope up with the patching work, or some systems may be too fragile to be patched.

Empirical evidence suggests that it is not the vulnerabilities themselves, but the absence of patching that lead to attacks, with most attacks and malicious software becoming available long after the patch for the vulnerability is available.

As the graphic below from CERT shows, there is a lag between the discovery of the vulnerability and the widespread availability of exploit tools. Today, software developers have become nimble enough to bring out patches before such availability of exploits. The onus thus rests on the IT team to patch the  systems before exploits become available.

Patches come out not just for vulnerabilities, but also for other software glitches and  minor feature and version upgrades.

Given the sheer number of patches and the number of systems on which they need to be applied, IT teams have no option but to resort to some form of automation for patch management and roll out. This rollout can be done either by the specific software themselves or by network wide management software. Software like anti-virus software and Operating Systems, that by default expect multiple installations, now come with their own deployment managers that download a single local copy of the patch and then deploys them over the network, increasing manageability, eliminating user intervention and conserving bandwidth.

Network management software like IBM's Tivoli and Microsoft's SMS Server come with extensive patch deployment and management capabilities. They also offer the additional ability of being able to do management tasks like version control and software inventory management.

In this story we look at how Indian enterprises are going about patching and updating their software and try to identify concerns and best practices. As is usual with our implementation strategy stories, we polled twelve CIOs across geographies and verticals for their views and practices.

Vulnerability Exploit Cycle

What to patch?
We identify nine distinct categories of software that need to be patched or updated on a fairly regular basis. The complete list is given in the graph below, which charts the priorities of the CIOs across these nine categories. Not surprisingly, anti virus and firewalls are right on top of everyone's priority list.  Hardware BIOS's, and appliances and desktop applications figure lower down the list when compared to others. What seems to be emerging is that the concern for patching and updating is higher for network centric enterprise applications and is lower for personal workstations and network devices. While it is unreasonable to expect the highest priority for every piece of software on the network, the lower priority for desktop applications in some enterprises should be a cause for concern, as there are many more applications running on the desktop and they are thought to be more prone to bugs and vulnerabilities. The lower priority on appliances is not so much an issue, yet, given the lower proliferation of appliances. One does get a nagging feeling overall that currently, desktop may be open to attack, and as appliances proliferate on the network, more focus needs to go on to them too.

Priority in Patching

Key concerns
The most significant concern of the CIOs polled was version control of software that they had running. Typically, organizations end up running multiple versions of the same software because of a number of reasons. As patches are version specific, and deploying the wrong patch on the wrong version can have disastrous results, it is only understandable that version mapping and management is the key concern in deployment.

Patch Application Timelines

Another concern that was thrown up was impact analysis, or what exactly will happen once you  deploy the patch. Will it work or will it break existing systems? It is to understand this that patches are first applied to test systems before deploying on to production systems. Obviously you cannot test out every single patch before deployment. So you would need to do an ABC analysis of running systems to work out which are the ones that will be deployed only after rigorous testing, which are the ones that would be deployed directly, but in stages, and which are the ones that are going to be applied direct, straight from the vendor's website.

Timelines
There are two issues here. How soon after a patch is released do you apply it, and how much time do you take to roll it out? Like we saw earlier, there is a window between the discovery of an exploit and it being widely used by adverse forces. Within this window, patches have to become available and have to be tested and deployed. In the case of patches that provide feature upgrades, this time line is obviously not applicable.

Time taken to Patch

The enterprises we talked to, seemed to be on the ball, when it came to how soon they applied the patch, with half of them applying patches within a week of their release and a quarter, even faster!

Time taken to Patch

And for most CIO's, it was a week's work to get the patch rolled out. That means, taken together, most patches were rolled out within two weeks of their becoming available. Not bad at all, one must confess!

View Point

Ishwar Jha, Zee Telefilms Ltd Q.What are your key areas of dissatisfaction, with current patching mechanisms? A.Various hardware and software vendors work in hybernated manner and when they provide updates, they don't look holistically for the impact that update can have on hardware and software.  Also some vendors do not provide a rollback feature which is a major cause of concern. Q.For custom developed applications and modules, how do you ensure updates and patches?  A.“We first install the software on test environment and check the functioning of custom software and also analyse the changes that needs to be made before deploying the final patch.  We also take backup of existing environment before applying any custom patch.”

Whose job is it?
Who is responsible for administering the patches? Half the CIOs polled stated that it was the responsibility of the user or administrator of the system in question to keep it up-to-date with patches, and a quarter of them had given that responsibility to outsourced IT service providers.

Operating Systems Managed

And the remaining had a mix of these two as well as the vendor's of the respective solutions. Enterprises in this quadrant may have a task on hand as their systems grow, in clearly identifying the process owner for each system and in ensuring that everyone delivers up their responsibility.

Tracking which patch has been applied to which system is critical, particularly in a multi solution or multi version environment. Network management software like Tivoli automatically do this. Where such systems are not in use, the IT teams resort to manual processes like registers.

Obviously, this is not a desirable situation and is prone to errors.

Update Management Solution Handles?

Automation
Automated patch deployment is the recommended method of applying patches in large environments. How many do automated deployments?

Not enough, by the looks of it. Of the nine classes of software that we identified earlier on (refer to the Priority in Patching graph elsewhere in the story), only Operating Systems and anti-virus software have more CIOs doing automatic patch deployments than the number who don't.

Hardware appliances and firmware figure at the bottom of the list of automated patch updation. As we go forward, we hope than more and more systems will come under automatic patch management processes, thereby reducing the stress and burden on the IT teams. And in Operating systems, Only Windows and to some extend, Linux deployments seem to be utilizing automatic patch deployments. This needs to be read with the caveat that we did not query the respondents on the specific operating system that they run. Three-fourths of CIO's we spoke to had disabled the users' ability to patch or otherwise update desktop applications in their organizations. This fits in with the other trend of desktops getting more rigorously monitored and the users' ability to install applications becoming strictly limited.

Anil Chopra and Krishna Kumar

Page(s)   1