Linux Patch Management: Keeping Linux Systems Up To Date

1.1. Basic Patch Concepts

In the world of Linux, patches are more than just something you might apply to the source code of a kernel. They include the updates that can help you keep your systems secure, error-free, and updated with the latest features.

Before we continue, it's important to define the concept of a patch and note the variety of sources from where you can download patches for your computer systems.

1.1.1. What Is a Patch?

A patch is an update. It incorporates changes in source code. Patches are normally applied to specific software components, such as the kernel, or a service, such as vsFTP. Patches may fix bugs, address security issues, or incorporate new features. As an administrator, you're responsible for testing the new software, making sure that it addresses any problems before your users see them on their systems.

In general, when you patch a program, service, or system, you'll be upgrading a package. Some Linux distributions can be configured to warn you when patches for installed software are available. We'll show you how to configure this in later chapters.

Security Fixes

The most important patches address security problems. This is where the Linux development model shines. Developers start working on a patch almost immediately after a security issue is revealed. The process is public, which reassures those concerned about the quality of the patch. As a result, security patches are often available in hours.

If you're administering a Linux computer, you need to keep up to date on the latest security issues. If you've paid for a subscription to a Red Hat or a SUSE distribution, you can get email warnings about security problems with your installed services. Other distributions may make alerts available by email or through their mailing lists. Problems with services are often announced on major Linux news sites, such as www.linuxtoday.com.

Service Upgrades

Users will always demand upgrades. And Linux developers will respond. I suspect that after someone finds a way to use Linux to toast bread, another developer will start working on how to upgrade Linux into a bread maker.

More practically, if someone developed a way to make Samba on Linux emulate a Microsoft Active Directory domain controller (which is in work for Samba 4.0, per http://www.samba.org/samba/devel/roadmap-4.0.html), there would be a lot of demand for that service. You would likely find yourself downloading that package on a substantial number of computers.

In general, programs that are released with new features get a lot of demand. Unfortunately, bugs are most likely to be found when a program is released with new features.

Just remember, the developers of a service are usually different from the developers of a distribution. So if you have a problem with a service, the fault may not be with the developers at Red Hat, SUSE, Debian, and so on.

Bug Fixes

If software were perfect, I think there would not be so much work in computing, especially at Microsoft. A lot of work goes into diagnosing and repairing buggy programs. Fortunately, the same infrastructure which leads to quick security fixes also leads to quick bug fixes for open source Linux programs.

When users report problems, they're likely to demand quick solutions. The previous performance of Linux developers in finding quick solutions leads to increased expectations for quick bug fixes. As an administrator, you'll be expected to roll out patches quickly, reliably, and securely.

Kernel Patches

Patches to the Linux kernel are of a special kind. They include the changes in source code between consecutive versions of a kernel. For example, if you want to upgrade from kernel version 2.6.15 to 2.6.16, you should apply patch-2.6.16 to your current kernel.

There are special requirements associated with kernel patches. Generally, they work only with the kernel as released through www.kernel.org. The people behind some distributions build their kernels with different features. Native Linux kernel patches may lead to conflicts, lost features, or even kernel panics.

Standard Linux kernel patches require that you adjust perhaps dozens of settings and then compile that kernel in binary format. This process can take hours, and is therefore something that you may not want to repeat. Many Linux distributions include preconfigured kernels in their repositories that you can use to upgrade your systems without having to compile them.

Note

Debian-based Linux distributions make it easy for you to set up a binary kernel from recompiled code, which you can then use to upgrade other Debian-based systems on your network.

Kernel Upgrades

Because the kernel is so important to the operating system, you should not take chances. It's best to upgrade Linux kernels by installing them side-by-side with the current working Linux kernel. If there is a problem, you can still boot that system with the old kernel.

If your users add hardware, modify drivers, or otherwise experiment with the guts of Linux, there's a chance that a new kernel that works on your test equipment may not work on some of the computers on your network.

In recognition, most upgrade services treat kernel-* packages differently. Services such as Red Hat's up2date and SUSE's YaST Online Update do not install upgraded kernels, unless you authorize the installation.

Risks

Whenever you install a patch, there are risks. For example, many kernel patches may lead to situations where you're unable to boot Linux.

If you run into a situation where a patch leads to a crash, you may be able to restore your system from a backup. If you're unable to boot your system for this purpose, you may need to use a "rescue mode" for your distribution. Rescue modes are available using the first installation CD associated with Red Hat/Fedora and SUSE.

Alternatively, you may be able to use a CD/DVD-based distribution, such as Knoppix. When you boot from a Knoppix CD, you can start a full-featured Linux distribution. You can then connect to backup media and even networks to restore your pre-patch configuration. For more information, see www.knoppix.org.

1.1.2. Patch Sources

There are several sources for patches and upgrades. The best source is generally the upgrade repository preconfigured for your distribution. However, there is often a delay when distribution developers process updates from other sources, such as the kernel, or services, such as the Apache Web server.

If you're in a hurry, you can download packages from the Web site directly associated with your service. While not built for your distribution, it can help you get new features into service as quickly as possible.

Sometimes the developers of a distribution don't bother with every service or upgrade. In some cases, third parties, such as developers of related distributions or even independent developers, configure packages that you can use.

The Native Linux Distribution

When you're upgrading a service, it's usually best to upgrade using the package built by the developers of your distribution. That is the most reliable way to be sure that the package you install copies files into expected locations. It also ensures that the package you install becomes part of that computer's package database.

For example, if you want new features associated with a new version of vsFTP on Red Hat Enterprise Linux 3, you'll want to use the associated vsFTP RPM package built by Red Hat. There are several advantages:

Files associated with the package are installed in locations expected by other programs built for that distribution.

Dependencies for a service may vary between distributions.

Upgrades built for a distribution are included in the package database for that system.

Upgrades can be installed with the native update systems for that distribution.

However, there may be a delay before the employees or volunteers associated with a distribution are able to build and customize a new or updated service.

Service Developers

If you absolutely need the latest update as soon as possible, you can install the package as released by the developer of that service. This may be appropriate if you're testing new features before distributing the package to other users on your network or if you absolutely need the latest security patch.

If you do use a package released by a service developer, you may be able to use the Red Hat Package Manager (RPM) or the Debian Packaging System (DEB) package as built by those developers for your distribution. Sometimes the developers of a service are well-schooled in how to build a package for a distribution, such as Red Hat or Debian. Otherwise, you'll have to download the package in a format such as a tarball and install it yourself.

Unless the developers of a service have specifically built a RPM or DEB package for your distribution, the disadvantages to installing such a service include

Additional work is required to process and install the service.

Files may be installed in directories not consistent with your distribution.

The service is not included in your distribution's database of packages.

To overcome any resulting problems, you can install the service as packaged for your distribution as soon as it is available.

Note

A "tarball" is an archive, usually compressed, typically in tar.gz or tar.bz2 format. For more information on how to decompress a tarball, see the man page for the tar command. When decompressed, installation instructions are normally made available in an included text file, such as README or INSTALL.

Third Parties

There are two kinds of third-party packages available. One is packaged for a different distribution that uses the same package manager. Another is configured by an independent developer or group who may or may not have customized a package for your distribution.

Linux programs are generally processed into one of two major package formats: the RPM and DEB. These formats are used by almost all the major Linux distributions. Some examples include

RPM: Red Hat, Fedora, SUSE, Mandriva, Turbolinux, Yellow Dog

DEB: Debian, Knoppix, Ubuntu, Xandros

For example, if a package has been created in RPM format only for Mandrake, you can use the rpm command to install that package on other RPM-based distributions, such as Red Hat and SUSE. However, the risks are similar to installing an uncustomized package released from a service developer.

Note

Mandriva is the company formed from the merger of Mandrake, Lycoris, and Conectiva. Mandriva 2006 (released in late 2005) comes from their combined efforts. For more information, see www.mandriva.com.

There are several databases of RPM packages, from where you can search and download the package for the distribution and version of your choice. They include www.rpmfind.net and rpm.pbone.net.

As you'll see in Chapter 4, "Making apt Work for You," distributions based on the Debian package system are more closely integrated. If you want to install a package from a different distribution with Debian packages, all you need to do is refer to its respective repositories. RPM-based distributions are moving in the same direction with yum.

There are developers who maintain their own distributions. One of the most useful is maintained in apt and yum RPM repositories at dag.wieers.com.

1.1.3. Patch Testing

Critical to the concept of patch management is testing. When applied indiscriminately, patches can override working drivers, break existing software, and change the links to key files. They may introduce new features that override services on which you rely.

Best practices suggest that you test every patch before installing it on production computers. Ideally, you'll have one or more extra computers available for this purpose.

Test computers may be a luxury unavailable to administrators of smaller networks. If you're in that position, you should at least test each patch on your own administrative system. Every problem you see on your own system is one less problem that you hear about from each of your users.

There is no single method for patch testing. Testing methods are as diverse as available applications and services. I've divided the testing process into three categories: installation, applications, and services. In any case, always follow this maxim: Do not install a patch on a production computer before testing.

Testing Patch Installations

The first step in testing a patch is installation. When you use the techniques described in this book, there should be no unmet dependencies. However, dependencies often require you to install more patches. Therefore, when you test a patch installation, you need to know what you're installing to know the scope of the patch.

So, in summary, when checking the installation of a patch on a test computer, check for the following:

When checking a patch, make sure all package dependencies are satisfied. List the packages required to satisfy those dependencies.

If there are dependent packages, include those in your testing.

These are general guidelines that vary with what you're installing. For example, if you're installing a new Linux kernel, test everything on a production-level system. If you're upgrading a stand-alone application, make sure that the upgrade won't break features on which your users depend. If you're upgrading a service, make sure that your configuration file still works as intended. If you're upgrading dependent libraries, make sure they don't affect the application or services which depend on them.

If you have problems with a patch, check the troubleshooting steps described at the end of this section.

Testing a New Kernel

The kernel is the most important patch to test. As the core of the Linux operating system, an upgraded kernel can affect everything. There are several principles that you should follow when testing a new Linux kernel:

Make sure you have a reason to change kernels. Check the release notes for that kernel. You may not need the patch. For example, if a new kernel incorporates a driver for network cards that you don't use, there is no reason for you to go through all this trouble.

Never upgrade to a new kernelin a way, that overwrites your existing kernel. Do not delete the existing kernel until you're finished testing the new kernel.

Most distributions support side-by-side installations of new and existing kernels. In fact, when you install (and don't upgrade) an existing kernel using a customized binary package, Red Hat/Fedora, SUSE, and Debian all "dual-boot" the new and existing kernels.

Test how the new kernel manages your boot process. Watch as it detects your hardware and starts your services. If there are problems or even unusual delays, the new kernel may have a problem with the associated hardware or service.

Check the log files associated with the boot process. Pay particular attention to services that won't start or hardware that isn't detected.

Test the new kernel on your system, based on the release notes. Make sure the new kernel works as intended on your computers.

Test your services and applications under the new kernel. Pay attention to any changes in behavior. Document these changes.

When you're ready to upgrade the kernel on production computers, tell your users what they can expect.

If you identify a problem with the kernel and absolutely need the upgrade, you may need to recompile the kernel yourself. For more information, see the Kernel Rebuild Guide, available from www.tldp.org. After you rebuild, you can install and start the testing process again.

Testing Application Patches

Many applications are patched frequently. Application patches may incorporate security improvements, additional features, bug fixes, or more. When you see an application patch, don't just blindly upgrade. Consider the following:

Read the release notes. Evaluate the reason for the patch. Does it add any features that you need? Does it address any bugs which annoy you about the application? Does it improve your security in a way that applies to your computers or your network?

If there are configuration files associated with an application, back them up. For example, if a user customizes the look and feel of The GIMP, a patch might overwrite some of the settings in each user's .gimp-* configuration directory.

You may want to use configuration files from users who use the application frequently. Tell them that you're considering an upgrade. They can help identify key features that you will want to preserve.

If you decide that you need the application patch, test it carefully. Download and install it on your test system. Make a list of downloaded packages, especially if there are dependencies.

Test the patch on your system. First, based on the release notes, make sure the patch works as intended.

Identify any changes made to applicable configuration files, and assess their impact on your users. Make notes.

If there are problems with the patch, re-evaluate the changes. Make sure you actually need the patch.

When you're ready to install the application patch in production, make sure your users know what to expect.

If you identify a problem with an application patch and absolutely need it, see the troubleshooting steps identified at the end of this section.

Testing Service Patches

Many services are patched frequently. Service patches may incorporate security improvements, additional features, bug fixes, or more. When you see a service patch, don't just blindly upgrade. Consider the following:

Read the release notes. Evaluate the reason for the patch. Does it add any features you need? Does it address any bugs that annoy you about the application? Does it improve your security in a way that applies to your computers or your network?

If configuration files are associated with the service, back them up, at least to another directory.

If you decide that you need the service patch, test it carefully. Download and install it on your test system. Make a list of downloaded packages, especially if there are dependencies.

Test the patch on your system. First, based on the release notes, make sure the patch works as intended.

Identify any differences from the previous configuration files, and assess their impact on your users. Make notes.

If the service is shared on a network, test the upgrade from another computer. Measure the current network demands on this service. If practical, find some way to simulate these demands on the upgraded service.

If there are problems with the patch, re-evaluate the changes. Make sure you actually need the patch.

When you're ready to install the service patch in production, make sure your users know what to expect.

If you identify a problem with an application patch and absolutely need it, see the troubleshooting steps identified at the end of this section.

Patch Troubleshooting

If you have a problem with a patch, the problem may or may not be with the patch. It could be a symptom of a different problem. Because troubleshooting depends on what you're patching, I can provide only general guidelines here. But as with any troubleshooting, it's best to apply the scientific method:

1. Observe the problem. Identify its symptoms, using log files where applicable.

2. Formulate a hypothesis of the cause. Was it the patch? Is there a configuration setting which conflicts with the patch? Is there something else you might need to install?

3. Test the hypothesis. If correct, you should be able to test the patch in a different way to confirm your hypothesis.

If you're having trouble formulating a hypothesis, get support. If you've paid for support, such as with a subscription to Red Hat Enterprise Linux (RHEL), use it. Otherwise, make use of the documentation. Many Linux administrators have documented their experiences in HOWTOs, FAQs, and more. They may have experienced the problems you're seeing now.

If you can't find the answer in existing documentation, you can try documenting your problem on an appropriate mailing list. They're frequently available for distributions, applications, and services. Identify the problem, the symptoms you've seen, the documentation you've read, and what you've tried to do to address the problem. Many Linux gurus are motivated to "show off" by solving your problem.