Previous Section
 < Day Day Up > 
Next Section


The Four-Phase Patch Management Process

As detailed earlier, the Microsoft-recommended patch management process is a four-phase process: Assess, Identify, Evaluate & Plan, and Deploy. The process is a continuous cycle, reflecting the reality of operations management. As the result of an alert of software update, the organization shifts from the Assess phase to the Identify phase and begins its journey through the process cycle. It’s important to realize that movement through the phases of the process is update- specific. This means that if an organization is in the Evaluate & Plan phase in response to one update when an alert is received for another, the organization will move to the Identify phase for the second update and be in two phases of the process simultaneously. This can pose both logistical and technical challenges to the organization, especially if the alerts are somewhat related or dependencies exist in the updates, and care should be taken.

The Assess Phase

The Assess phase is the first phase in the patch management process. In the Assess phase the organization is concerned primarily with the ongoing assessment of its production environment covered by the patch management process and with monitoring for alerts of software updates. Although many of the steps and tasks in the Assess phase are similar to the tasks undertaken during preparation for the implementation of a patch management process, they’re instead focused on optimizing and improving the existing patch management process.

Inventorying and Discovering Existing Computing Assets

As part of the operations of any production environment, IT assets will be added, updated, or retired. This task in the Assess phase is concerned solely with ensuring that the record of IT assets is accurate. As assets are added to the production environment, they should be inventoried and evaluated for inclusion into the patch management process. Assets that are retired should be removed from the record and from the patch management process. Existing assets should continuously be inventoried to ensure that their configuration hasn’t changed without a formal change management process or the patch management process. You can automate these tasks somewhat by using SMS 2003’s Active Directory and network-based discovery techniques, which are especially useful for discovering unmanaged or rogue systems that were deployed outside a management process.

Assessing Security Threats and Vulnerabilities

Although the IT assets within an organization might have the latest software updates applied to them, there might still be risk from security threats and vulnerabilities introduced by the addition, configuration, or removal of hardware or software components. The organization needs to remain vigilant and check for vulnerability using tools such as the Microsoft Baseline Security Analyzer (MBSA), available from http://www.microsoft.com/mbsa and the Software Update Inventory Tools for SMS 2003, which are described later. When vulnerability is discovered, you need to undertake a risk analysis to determine the best course of action. You can find details on how to conduct such an analysis in Chapter 3, “Understanding the Security Risk Management Discipline,” available at http://www.microsoft.com/technet/security/ prodtech/windows/secwin2k/03secrsk.asp. Typically, you’ll take steps to mitigate the vulnerability through a configuration change or possibly a software update.

Determining the Best Source for Information About Software Updates

The patch management team needs to remain informed about new software updates for IT assets under control of the patch management process. The team can remain informed by subscribing to e-mail notifications, visiting vendor Web sites, and through regular contact with representatives of software vendors.

Microsoft’s authoritative Web site for information about security bulletins is http://www.microsoft.com/security, where users can also sign up for e-mail notification of software updates and security bulletins issued by Microsoft.

Assessing the Existing Software Distribution Infrastructure

Although careful planning and implementation might yield a versatile patch management infrastructure that meets the needs of the organization at implementation time, it might not suffice as the organization changes. The patch management team needs to assess the patch management infrastructure continuously to ensure that it continues to meet the organization’s needs. SMS 2003 is an extremely flexible management solution and can be reconfigured as needed to support the addition, change in configuration, or removal of categories of IT assets or sites.

If the Assess phase was entered after the completion of the last phase in the patch management process, the Deploy phase, the organization should evaluate how successful the deployment of the software update was. If problems were encountered, the infrastructure should be examined for problems.

Assessing Operational Effectiveness

Lastly, in the Assess phase the patch management team needs to validate the entire patch management process. They need to ask questions such as the following: Does the patch management team have the necessary resources? Do key security stakeholders have the necessary training and understanding of the patch management process? Are formal processes in place for day-to-day operations that have an impact on security? You can find the Microsoft Operations Framework Self-Assessment Tool, designed to gauge an organization’s operational excellence, at http://www.microsoft.com/technet/itsolutions/tandp/opex/ moftool.asp.

As with assessing the software distribution infrastructure for problems after a deployment, the patch management team should assess the operational effectiveness of the process, including their own performance. Lessons learned in this assessment should be applied in order to improve operational excellence for future software updates.

Leaving the Assess Phase and Moving to the Identify Phase

When the patch management team receives notification of a software update that affects the production environment, the patch management process moves into the Identify phase.

The Identify Phase

During the Identify phase of the four-phase patch management process, the organization is focused on gathering information about the software update that triggered entry into the Identify phase, whether or not the update is relevant to the production environment; obtaining the software update itself; and categorizing the update as either an emergency update or one that can be dealt with routinely within the time frame set by the organization for software updates.

Discovering New Software Updates Reliably

When an organization receives an e-mail alert from Microsoft about a software update, the alert contains links to Microsoft’s Security & Privacy Web site, found at http://www.microsoft.com/security, where more detailed information is made available. Organizations might choose to subscribe to other sources of information about software updates or receive e-mail messages from account representatives or other Microsoft employees. Regardless of where the alert came from, the patch management team needs to verify its authenticity. For alerts detailing software updates to Microsoft products, the authoritative source of information is the Microsoft Security & Privacy Web site, which the team should visit. Of special note, Microsoft never releases software updates as attachments to e-mail messages. If the team receives an e-mail message with an attachment purporting to come from Microsoft and containing a software update, the safest course of action is to delete the message. Similarly, rather than clicking on links contained in e-mail messages that appear to take the reader to a Microsoft Web site, the reader should copy and paste the URL into their browser, as the true Web site that the user will be taken to can be hidden in the formatting of the e-mail message.

SMS 2003 Software Updates, when configured, will automatically poll Microsoft’s Web sites, looking for information about software updates in the products that it’s aware of. Using the SMS Administrator Console, you can view just about every applicable software update for the production environment, along with the number of systems that are in compliance and those that are not. Figure 13.5 shows a screen shot of software updates listed in the SMS Administrator Console. You can find details on how to configure SMS Software Updates to show this information later in this chapter.

Click To expand
Figure 13.5: Software updates listed in the SMS Administrator Console.

Determining Whether Software Updates Are Relevant

Benjamin Franklin wrote, “In this world nothing is certain but death and taxes.” Although this was written long before software was invented, today he would no doubt feel compelled to add software updates to the list of certainties faced in life. The fact is that software releases are made on a continual basis and can come from a variety of sources, such as operating systems and application vendors, independent software vendors (ISVs), original equipment manufacturers (OEMs), and hardware manufacturers. Not every released software update will necessarily apply to an organization’s production environment, even when it touches a software product in use. The patch management team will need to evaluate and determine whether or not patches are relevant to the production environment covered by the patch management process. For example, a software update to a word-processing package that’s deployed within the organization might apply only to a feature that isn’t installed or used, or perhaps the risk from not installing the update is considered so low that it can be safely ignored. A good starting point for evaluating the relevance of a software update issued by Microsoft is the detailed bulletin found on Microsoft’s Security & Privacy Web site at http://www.microsoft.com/security. The bulletin will contain details of the vulnerability addressed by the update, any mitigating factors that might affect the requirement to update a system, and any workarounds if available. Each software update should be assessed from a risk management perspective, with the organization making a determination about whether or not the update should be applied to its affected IT assets. When considering the risk from not installing an update, the organization must balance it against the risk from installing the update. Risk from installation includes incompatibilities with already-installed applications, system instability, and loss of functionality.

As described earlier, SMS 2003 Software Updates can be configured to poll Microsoft’s Web sites for details of available software updates and list them in the SMS Administrator Console. Only updates that are applicable to the production environment are listed. An update is deemed to be applicable when one or more SMS Clients scans itself and determines that the update should be applied and reports this information to an SMS site server. You can find full details on configuring SMS 2003 Software Updates, and details on how the feature works, later in this chapter.

Obtaining and Verifying Software Update Source Files

When assessing the relevance of a software update, it’s often desirable to have the source files for the update at hand. The patch management team should take care when obtaining software updates to ensure that they come from legitimate sources only. Every software update that Microsoft releases is digitally signed, and you can verify its authenticity by examining the signature. Although you can do this manually, you can also do it using SMS 2003. When a software update is identified as relevant to the production environment and listed under Software Updates in the SMS Administrator Console, it can be included in a package built by the Distribute Software Updates Wizard. You can instruct the wizard to download software updates automatically, and it will check the signature of each as it does so. If the wizard is unable to download a software update automatically, which might be the case if different updates exist for different platforms and configurations or if the patch management team prefers to download updates by hand, you can verify the software update’s digital signature through its properties using Microsoft Windows Explorer. Figure 13.6 shows the dialog box of a software update’s digital signature, accessible through Windows Explorer.

Click To expand
Figure 13.6: A digital signatures dialog box, accessible through Windows Explorer.

Until a software update identified in an alert has been successfully downloaded and verified, the patch management process can’t leave the Identify phase.

Determining the Nature of the Software Update and Submitting a Request for Change

Every software update released by Microsoft is assigned a severity rating. The purpose of the ratings is to provide guidance to administrators and patch management teams about the urgency with which the update should handled. These ratings, and their meanings, are listed in Table 13.2. You can find full details of the ratings used by Microsoft in TechNet at http://www.microsoft.com/technet/ security/bulletin/rating.asp.

Table 13.2: Software update severity ratings

Rating

Definition

Critical

A vulnerability whose exploitation could allow the propagation of an Internet worm without user action.

Important

A vulnerability whose exploitation could result in compromise of the confidentiality, integrity, or availability of users’ data or of the integrity or availability of processing resources.

Moderate

Exploitability is mitigated to a significant degree by factors such as default configuration, auditing, or difficulty of exploitation.

Low

A vulnerability whose exploitation is extremely difficult or whose impact is minimal.

The patch management team should take an update’s severity rating into account when determining its nature. Software updates with a low rating might be safely ignored until a predefined refresh cycle is begun, whereas an update rated as critical might need to be deployed as soon as possible. Mitigating factors should also be taken into consideration when determining the nature of an update. For example, with sufficient perimeter defenses and a secure VPN in place, a patch management team might consider not rushing an update for a newly discovered vulnerability that a worm is exploiting if the port used by the worm to propagate itself is blocked at the firewall and not allowed over a VPN connection.

Other considerations that the patch management team should examine are the impact of the update on the production environment. For example, does the update require systems to be restarted after installation and can it be rolled back if it’s determined that the update has a negative effect on the production environment?

Once the patch management team has completed determining the nature of the software update, it should submit an RFC to the Change Management Board.

Leaving the Identify Phase and Moving to the Evaluate & Plan Phase

The events that trigger leaving the Identify phase and the handover to the Evaluate & Plan phase are the successful acquisition of the software update and the submission of the RFC.

The Evaluate & Plan Phase

The Evaluate & Plan phase of the patch management process is where the organization needs to examine how it will respond to a software update, how it will release the update, how it will build the update, and how it will conduct acceptance testing for the release.

Determining the Appropriate Response

When an RFC is created, the initiator assigns an initial priority and category to the request. The patch management team should review the RFC and agree to, or change, the priority and category. The finally determined values will have an impact on the remainder of the patch management process, including how and when the software update is released. When the team reviews the initial priority attached to an RFC, it should consider what assets are impacted by the vulnerability the update addresses and whether these are critical systems, whether controls are in place (or can be put in place) to mitigate the vulnerability, and whether the update needs to be applied to as many systems as first thought. It’s recommended that the organization define priorities for release of software updates and time frames consistent with their needs, such as those in Table 13.3.

Table 13.3: Release priorities and time frames

Priority

Recommended Time Frame

Minimum Recommended Time Frame

Emergency

Within 24 hours

Within two weeks

High

Within one month

Within two months

Medium

Depending on availability, deploy a new service pack or update rollup that includes a fix for this vulnerability within four months

Deploy the software update within six months

Low

Depending on availability, deploy a new service pack or update rollup that includes a fix for this vulnerability within one year

Deploy the software update within one year, or you might choose not to deploy at all

The organization might wish to consider formalizing the criteria used when determining whether or not to adjust a release’s priority. You can find an example of formalized criteria in the form of environmental and organizational factors and the corresponding priority adjustments in Table 13.4.

Table 13.4: Release priorities adjustment criteria

Environmental/Organizational Factor

Priority Adjustment

High-value or high-exposure assets impacted

Raise

Assets historically targeted by attackers

Raise

Mitigating factors in place, such as countermeasures that minimize the threat

Lower

Low-value or low-exposure assets impacted

Lower

Emergency change requests, where vulnerability is being exploited within the production environment or system instability is affecting line-of-business applications, need to be handled expeditiously. These requests might cause other requests with lower priorities to be delayed or halted if already in deployment in order to free the necessary resources required to process them. You can find more information on emergency response later in the chapter.

Not all software updates are the same, and each might have a different effect when applied to a system. Some updates might require a system to be rebooted while others do not. It’s possible that some updates will rely on other updates, perhaps a service pack, and can’t be installed without them. Also, some software updates, once applied, can’t be removed from a system. The patch management team members need to assess the impact that a software update will have on the environment in order to plan for it. This is called categorizing the update.

Once the release priority and category of the software update has been determined, the release needs to be reviewed and authorized for deployment. Before authorization is granted, the patch management team will need to consider various factors, such as what is currently happening in the production environment, the release’s projected cost, the best means of distributing the release, the resources required to manage the release, and any dependencies the release requires. Once authorization has been granted, a member of the team should take ownership of the release. The release owner is responsible for assembling the necessary resources to guarantee the building of the release, its testing, and its eventual deployment.

Planning the Release

Once the release has been approved, detailed planning should take place to guarantee the deployment’s success. Although considered previously in the Identify phase and when determining the appropriate response in this phase, the IT assets that need to receive the software update need to be identified and recorded. The Software Updates component of SMS 2003 can help identify assets that need updating, but if SMS 2003 can’t discover the vulnerability that the update addresses, the patch management team might need to resort to mining through the information recorded by the SMS Client hardware and software inventory agents.

The patch management team will need to determine when to release the software update and whether to allow users to influence the release process. The team might decide, for example, to allow users a seven-day grace period during which they can choose to install the update contained within the release before it becomes mandatory. The team will also have to take into account the release’s impact on the production environment. For many organizations it might be easier to release updates over the weekend instead of during the week. The team will need to factor in the size of the update, as larger updates will take longer to download to clients, especially those that connect over slow links such as portable computers on a dial-up connection.

The team might also wish to consider a staged release, where some systems receive the update before others. For example, if the update is determined to be a high priority and is applicable to the patch management infrastructure servers as well as other IT assets, the team might decide to apply the update to the servers first to guarantee their availability while deploying the update even if an exploit has been discovered.

The team will also want to draw up a timetable for the release lifecycle, including packaging the software update, testing both the update and the package, publishing the package in the patch management framework, deploying the package to systems, and validating the deployment. As part of the preparation for patch management, the organization might wish to consider developing templates of release timetables, one for each combination of category of asset and priority level. A sample timetable for a release is shown in Figure 13.7.

Click To expand
Figure 13.7: A sample release timetable for servers for a high-priority release.

Building the Release

Once the release has been approved, the patch management team needs to build the release package, which will be distributed to affected IT assets using the patch management infrastructure. The package’s format will depend on the infrastructure tools. You can use the SMS 2003 Distribute Software Updates Wizard, described later in this chapter, to build a package containing the software update from updates available from Microsoft’s Web sites and which are discoverable by the Software Updates feature of SMS.

Conducting Acceptance Testing

When the release package has been built, it must be tested prior to deployment to ensure that no problems will result from application of the package in the production environment. The goal of testing should be to test both the package and the update(s) contained within it in an environment that’s representative of the production environment. At a minimum, the following tests should be conducted:

  • The package can be deployed successfully to assets within the production environment, including over slow links, such as those connecting remote sites and portable computers.

  • The system reboots correctly after the package has been applied.

  • The package can be uninstalled or rolled back.

  • The package, once installed, doesn’t prevent business-critical and infrastructure systems from functioning normally.

Most organizations will wish to conduct more than these minimum tests to ensure that line-of-business applications continue to function normally. The number, range, and detail of the tests will depend on the categories of IT assets in the production environment and the software installed on them.

Before they begin to test, the patch management team members should build test plans, which define what should be tested and what the desired results should be. As with release timelines, these test plans might be developed while preparing for patch management.

Building a patch test environment that mimics the production environment, complete with patch management infrastructure, can be expensive. An alternative to a test environment is to designate certain machines in the production environment as test machines that will receive a package prior to the rest of the production environment. If the package can be deployed and the update applied successfully, the package can be deployed in a general release. This testing strategy needs to be managed carefully, as end users will become the testers and might require additional training. This strategy is also likely to result in acceptance of a patch without the rigorous testing that would be achieved in a test lab.

Leaving the Evaluate & Plan Phase and Moving to the Deploy Phase

The build of a release and the success of the acceptance testing for a package are the triggers for the change to the Deploy Phase.

The Deploy Phase

The last phase of the Microsoft-recommended four-phase patch management process is the Deploy phase, where the patch management team prepares to deploy the packaged software update built in the Evaluate & Plan phase into the production environment. If problems develop during, or as a result of, the deployment, the team might have to roll back or manually apply the update on affected systems. After the package has been deployed, and perhaps rolled back, the team will want to conduct a review of the experience to look for ways to improve future deployments of software updates.

Preparing the Deployment

Once the package(s) containing the software update(s) to be applied to the production environment have successfully passed acceptance testing in the Evaluate & Plan phase and the organization has moved into the Deploy phase, the patch management infrastructure and the organization need to prepare for deployment. Among the steps that need to be taken to prepare the organization are communicating the rollout schedule to affected parties and configuring the patch management infrastructure to prepare to distribute the update.

Communicating information about impending updates to stakeholders such as system administrators and end users is extremely important. A simple and effective communication tool is e-mail. By informing the organization that a software update is pending or in progress, the team can accomplish many things, including reducing the number of calls to the help desk from users wondering what’s happening to their systems and giving users the opportunity to install an update when it suits them before forcing a mandatory update on them.

The steps that the patch management team will take to prepare the patch management infrastructure for the update will depend largely on the technology used and how it’s configured. With SMS 2003, the team will need to import packages containing software updates from the test environment or build new ones in the production environment, assign distribution points for the package, and stage the updates on the distribution points. In environments where there’s a hierarchy of SMS 2003 sites, care should be taken to ensure that each package is assigned an appropriate priority and is distributed to all the sites.

Deploying the Software Update to Targeted Computers

As with the preparation, the steps taken to deploy a package containing updates will depend largely on the technology being used and the way it has been configured. The relative priority of the release and the options chosen when building the package will also figure into the deployment. Essentially, there are three steps to deploying a package containing a software update: making clients aware of the update’s availability, monitoring the package’s deployment, and recovering from failed deployments.

How the advertisement containing information about a package ready for deployment is configured will depend largely on the update’s nature. For example, a relatively low priority update might allow the end user to choose whether or not to accept the package and install the update on the user’s system, whereas a high-priority update might be mandatory and leave the user with no choice but to have the update installed. The organization might also wish to create multiple advertisements for the same package, targeting tailored advertisements at different categories of IT assets to reflect different priorities across each.

During the deployment, the patch management team needs to monitor the production environment for problems that might arise on the systems to which the package is being delivered or within the patch management infrastructure itself. Among the variables that should be tracked are the number of systems that have received the advertisement and have successfully applied the update, the number that have received the advertisement but failed to apply the update, the number that have received the advertisement but have not yet attempted to apply the update (for whatever reason), and the number that have not yet received the advertisement. Depending on the technology and how it’s configured and used, the patch management infrastructure might be able to reliably report these figures. If the infrastructure is incapable of supporting such reporting, the patch management team might have to use custom scripts (perhaps contained within the package itself) to monitor the deployment. SMS 2003 does provide a reporting mechanism so that the patch management team can gauge the deployment’s progress.

If during the deployment’s monitoring the patch management team sees a number of systems receive the advertisement but fail while applying the update, they should begin an investigation into why. Of immediate concern should be the possibility that the package is somehow corrupted. A good indicator that this is the case is a high number of failures proportional to successful applications of the package early in the deployment process. If a determination is made that the package is responsible for the failures, the team should stop the deployment and resolve the problem before making the package available again. Not all failures will be attributable to a faulty or bad package, however. Some might simply be due to the system not requiring the update contained within the package, perhaps because the system was manually updated earlier by the user or administrator or because the system was incorrectly identified as requiring the update due to a vulnerability that doesn’t exist on the system.

In most deployments there will always be a few systems that can’t be updated using an advertised package, either because of their configuration or because they aren’t managed by the patch management infrastructure. In these cases the team will likely have to visit each system and apply the update manually. Care should be taken when deploying an update in this fashion to ensure that the correct installation options are used to guarantee the updated system’s integrity.

The worst-case scenario for the patch management team is a successful deployment of a package containing an update only to find that the update needs to be rolled back due to unforeseen circumstances, such as application incompatibility that wasn’t identified during testing. Complicating factors might be that the update performs as expected on some machines but not on others. If there are no initial reports that the update causes problems to systems and the patch management process leaves the Deploy phase, the subsequent action that needs to be taken to remedy the problem can be considered a software update in its own right and requires movement through the four-phase patch management process.

Reviewing the Implementation

The goal of the review process is to gather information and lessons learned from the deployment that can be used in future applications of the patch management process. Typically held a short time after it has been determined that a package was successfully deployed, the review should focus on ensuring that the IT assets in the production environment have been categorized correctly, that the vulnerability that was mitigated by the software update has been removed, and that the update will form part of the secure baseline configuration for new systems deployed into the production environment. The review should also examine the patch management team’s performance.

Leaving the Deploy Phase

Once the Post-Implementation Review has been completed, the organization leaves the Deploy phase of the patch management process and reenters the Assess phase.



Previous Section
 < Day Day Up > 
Next Section