Previous Section
 < Day Day Up > 
Next Section


Responding to Emergencies

Even with the best planning and resources, the chances are high that at some point, an organization will have to cope with an emergency, such as a patch with an emergency release priority, a worm exploiting an unpatched vulnerability in the production environment, or the realization that a previously applied update is causing systems to fail. An organization can react to problems and resolve them more quickly if it is well prepared for the eventuality.

The patch management team should attempt to anticipate the types of emergencies that it will have to deal with, and even conduct fire drills. In the event that the entire patch management team cannot be brought together quickly enough to address an emergency, there should be a core emergency response team that has the authority to handle the problem appropriately. By far the most common emergencies requiring the participation of the patch management team are releases with accelerated timelines and the rollback of previously applied updates.

Releases with Accelerated Timelines

There can be many reasons why a release is deemed to be an emergency and the patch management team needs to accelerate the release timeline. Regardless of the reason, there can be extreme pressure to ignore the patch management process in place and deploy a software update immediately. By circumventing the process, however, organizations risk the integrity of the production environment. The four-phase patch management process recommended by Microsoft is designed to cope with emergencies and should be followed when responding to one.

Note 

It is important to stress that just because a software update is given a critical rating by Microsoft, it doesn’t necessarily translate to an emergency release for the organization. A software update is designated with an initial release priority in the identify phase of the four-phase patch management process. This initial priority is confirmed or adjusted in the Evaluate & Plan phase. If the priority is confirmed as being an emergency release, the patch management team needs to make accommodations from that point on in the process to ensure that it is dealt with in a timely fashion.

The greatest difficulty in dealing with releases with accelerated timelines is that of completing the acceptance testing for packages and the updates contained within them. Often the patch management team is faced with the prospect of having to deploy a package that has not been fully tested in order to meet the accelerated timeline. In such cases the team should concentrate on testing core functionality to ensure that day-to-day operations are not affected by the deployment of the package and application of the update. The patch management team may have to continue testing the release through deployment and afterward in order to confirm that deployment was the best course of action and to satisfy themselves that a rollback will not be required in the future.

SMS 2003 packages containing emergency releases should be advertised as mandatory, forcing clients to install them. By default advertisements that are mandatory have the option Assignments Are Not Mandatory Over Slow Links checked. The patch management team should clear this check box, found on the Schedule tab of the Advertisement Properties dialog box. For large packages which cannot be feasibly downloaded to clients connecting over slow links, such as portable computers over dial-up connections, the patch management team might wish to resort to issuing a call for those systems to be brought in to a local office for updating or some other out-of-band patch management strategy.

When an emergency release affects the servers hosting the patch management infrastructure, the patch management team might choose to patch those systems manually or out-of-band, to ensure that any necessary reboots do not affect the deployment of packages. Another consideration for the patch management infrastructure is how packages are deployed between sites. With SMS 2003, the intersite senders should be configured to remove all restrictions on the intersite senders in order to distribute the updates to other sites as quickly as possible. When links between SMS sites have too little bandwidth to cope with a large package, the patch management team might wish to consider alternatives for distributing updates to the remote sites, such as using the SMS Courier Service.

Rolling Back Software Updates

Occasionally there might be the need to roll back a previously deployed update if it is deemed to be causing or likely to cause problems in the production environment. A rollback might not always be an emergency event, and the patch management team must determine the appropriate priority to place on the event. In all cases, the team should strive to treat a rollback as though it were a software update and follow the established patch management process.

Whether or not the rollback is deemed an emergency, any ongoing deployments of the update in question should be halted. To disable the program for an update which needs to be rolled back, the SMS administrator can select the option Disable This Program On Computers Where It Is Advertised on the Advanced tab of the Program Properties dialog box. For packages that were created by the Distribute Software Updates Wizard, the SMS administrator should re-run the wizard, select the package containing the update that will be rolled back, and clear the update to remove it from the package. This will update the contents of the package, removing the update in question and leaving other updates in the package intact. If there are no other updates in the package, the SMS administrator can delete the advertisement and the package.

The mechanism for rolling back an update is update-specific. Not all software updates, especially updates addressing security vulnerabilities or updates to core components of an operating system, can be rolled back. For those that can be rolled back, the administrator needs to determine what the command to initiate the rollback is, build a package with a program to invoke it, and deploy it. The SMS administrator has two options when deploying a rollback package. The first is to have the rollback program determine whether or not the update has been applied to a system before initiating the rollback and advertise the package with the program to the same systems that the software update was advertised to, and the second is to attempt to identify the systems which applied the update and target an advertisement for a package with the program that performs a rollback to these systems.



Previous Section
 < Day Day Up > 
Next Section