Upgrading and Repairing Servers

When people think in terms of servers, they often tend to think that a server is a monolithic entity. Nothing could be further from the truth. Usually servers are deployed so that they provide one or perhaps two main network services, at most. This is the case because you can get better performance from a server when it is tuned for the specific function it is providing. In any large enterprise application, you adjust the storage system, size and type of memory and processors, and other functions to suit the application at hand.

The sections that follow describe the different types of servers that are available, from a network services point of view.

The Purpose of a Server

In the past, computers providing network services were often categorized as minicomputers, mainframes, or supercomputers. With the introduction of general-purpose microprocessors that are used both in PCs and for shared services, the term server has come to be used for any system that is microprocessor based and provides network services.

The purpose of a server is to provide reliable and predictable network services to clients. Either the OS or some software application runs a process that provides the service to software that is usually running on another computer, called the client. For example, Microsoft Exchange running on a server that uses the Windows Server 2003 OS collects messages for Outlook clients of that system. However, you can just as well open a copy of Outlook on the server itself, and then your client is running on the same system. It's good to keep in mind that a client represents just a piece of software and not necessarily another system.

Note

Not all systems use the terms clients servers and the same way. In X Window, as discussed shortly, the server is the software receiving the information from the client. It's a little confusing, but it's important to keep in mind for each particular OS.

Server Categories

Servers are generally described as belonging to one of the following categories:

• General-purpose server A general-purpose server is an enhanced PC that is capable of being configured to be able to run a variety of applications.

• Application server An application server's primary purpose is to run a specific application.

• File and print server File and print services are most often OS functions, so a file and print server is a general-purpose server that is tuned specifically to serve files or interface to printers or virtual printers.

• Mail or messaging server A messaging server uses the POP or IMAP protocol to store and send mail messages.

• Collaboration server A real time communication server such as one managing Voice over IP (VoIP) traffic would be included in this category of servers.

• Database server Most enterprise servers use some implementation of the SQL standard, and most are relational. However, there are a broad range of database server types in use, including object databases.

• Domain or logon servers All NOSs establish their credentials by using a domain or logon server. However, it is often possible to replace or supplement authentication services by using other servers' products.

• Backup servers Backup servers copy data from clients or other servers to another location. Backup servers range in type from a general-purpose server to specialized appliances fitted with a backup program.

• Proxy server or firewall When set up as a proxy server or firewall, a server is configured as a router. Two or more network interfaces maintain physical and logical isolation of one network from another. Often these types of servers are set up and sold as routers.

• Storage server A storage server can be as simple as an appliance with an embedded OS, to a specialized network file server, to a large block-oriented intelligent disk array.

• Appliances An appliance is a server with a very specialized purpose and with limited management functions. The term appliance is best applied to a system that can be essentially plugged in to a network and turned on with little or no effort.

From this list you can see that servers fulfill a huge range of roles in a network environment, and many serve multiple roles. A domain server with a shared network resource, for example, provides the NOS server function of authentication as well as the server function of file services. It's best to take a holistic view when considering what is meant by the term server.

Servers Versus Clients

Consider the situation of a terminal service running on a server. Terminal services essentially run instances of the server OS on the server, and they do so by sharing open resources. Terminal servers are typically put into place to run applications that can be viewed through a low-bandwidth connection on a client system or on a computer terminal. Clearly, the terminal server is a server in its own right. The application running inside the server uses the terminal server as a client but would be considered part of the terminal server service to the window that you open on a workstation or terminal. Most people ignore the complexity here and consider the view into the terminal server running the application as a single server service.

It gets more interesting when you consider how a server based on the X Window System is named. The applications running on an X Window server are called clients. The command session(s) running on a terminal or workstation are called the server. X Window systems are, by design, a client/server architecture. It's possible to run both a client and a server on the same computer, but from the standpoint of networking, the X protocol can use the command set from any terminal or workstation (the server) to issue commands to the client system that actually does the processing.

Because X Window is designed as a network architecture, you might wonder how it's possible to run both server and client on the same computer. When a single computer runs both components, either the UNIX local socket or a loopback interface is used to simulate the network component that is part of the transport process. That is, X Window considers these elements to be a transparent network connection for the service. As far as the X Window System is concerned, the server's command is transmitted to these elements, and then the reply is returned by them.

Just keep in mind that for all other NOSs described in this chapter, the X Window System uses exactly the opposite naming convention. It's simply a matter of definition, and of course, point of view.

Web Services

The term server has recently acquired an even more general usage, as many vendors are rushing to create web-based services. The draw to providing web services is that vendors can provide sophisticated services from servers located anywhere in the world for low cost. Because they can sell these services to a larger audience by using a per-usage model, the margins behind this kind of business model can be enormous.

To implement a successful web service, a vendor or standards organization must publish and have accepted a set of standards for programming languages and program construction, protocols, and API against which applications can be written to run on a variety of platforms. The most successful web services to date have been based on Sun's Java platform, but with everything from Microsoft's .NET servers, Novell's exteNd strategy, Google's web services API, Amazon Web Servicesto name but a fewexploding onto the scene, the server marketplace is changing once again.

Essentially what web services do is let a small server (perhaps your web server) behave as if it were a much bigger and more capable server. You don't need a local authentication server because that service runs remotely. You don't need an accounting package because that service runs remotely; you don't need an inventory module, a database engine, a messaging server, an Unreal Tournament serveryou get the ideabecause these services all run on a server remotely. You only need to have data pass from client to server securely and reliably to have this work. When done correctly, the service is transparent, and the user doesn't know or need to know the location of the server that is doing the work.

Note

For a nice description of web services and Novell's positioning, see www.novell.com/collateral/4621359/4621359.html. For information about Microsoft's efforts, you can go to http://msdn.microsoft.com/webservices/. You can find a jump page with whitepapers and tutorials at http://msdn.microsoft.com/webservices/understanding/webservicebasics/default.aspx.

When a client makes a request of a web service, a program such as PHP, JSP, ASP, or a COM object makes a service request on the web service server. That request gets processed, and data is returned. When you log on to Hotmail or log in to Microsoft's Passport service to authenticate yourself, you are running a web service.

A successful web service relies on a set of standards being uniformly accepted so that it can be implemented by developers in a consistent way. You only need to consider the history of Java or the way browser technologies have developed to know that even when standards organizations are involved, it is rarely the case that all vendors implement the same standard in a consistent way. Standards committees rely on industry vendors to create products that implement their standards, and in most cases to extend the standards to create new capabilities. If fonts in browsers are step one, one vendor might think stylesheets are the next thing and implement them prior to the process of clearing an RFC. By the time that issue gets worked out, perhaps the next extension is moving stylesheets to XML.

Acceptance of networking standards is clearly a moving target. This process has affected how successful web services have been so far. To combat these problems, standards committees such as W3C, WSI, OASIS, and others have chosen to isolate them by codifying individual protocols and standards and then working on trying to get them all to communicate successfully with one another. You are already probably familiar with the elements of the web service architecture. Figure 18.2 shows how these elements interact.

Web service component standards include the following:

• Protocols Protocols include the common transport protocols, such as HTTP, SMTP, FTP, and XMPP, as well as a web service protocol "stack" that is used to implement specific web services.

• Interchange formats The exchange format with the most juice is XML. Data in XML can be transported using protocols such as Microsoft SOAP, JAX-RPC, and even XML-RPC. SOAP is at the heart of Microsoft's .NET initiative.

• Interface The Web Services Description Language (WSDL) is the best-known interface. It is an XML communications service.

• Security protocols The OASIS standard for security is the WS-Security protocol, which is used to authenticate requests and maintain data security for communications.

• Publication protocol The most commonly used protocol to transmit the information provided by servers running web services is UDDI.

The key advantage of building, deploying, or using servers running web services is that doing so allows applications to become portable and much more interoperable. Because web services are based on an open standard with common transport mechanisms, you can utilize existing infrastructure without having to create or deploy any special hardware, software, or security to get it to work. All that said, the main problem behind web services is that they are still not standardized, they are slow, and web transport is not as secure as internal network protocols. Negatives aside, web services are still the future, and you may find yourself implementing such server applications frequently.

Server Hardware and Drivers

Depending on the NOS you use, your server solution will either come as a software-only package that gets installed on industry-standard equipment or as a hardware/software bundle that the OS vendor provides. There are advantages and disadvantages to both approaches.

Software-Only NOSs and the Microsoft Experience

Microsoft's server OSs, Windows 2003 and, prior to that, Windows 2000 and NT, are software-based approaches to building a NOS. Microsoft relies on its industry associations with companies such as Intel, but others such as Advanced Micro Devices (AMD) instead create what is best called an industry-standard PC and server-based architecture. The size of Microsoft's market actually helps create a diverse hardware standard to support different versions of the OS. This approach is quite helpful when it comes to seeding the desktop and laptop markets with a number of different compatible designs, but it can have unintended consequences when it comes to server hardware.

Windows NT grew in popularity in the late 1990s but was burdened with a reputation in the industry of being unreliable. As Microsoft set about designing and debugging its code for Windows 2000, the company developed and acquired a set of testing tools designed to test memory leaks, check variables at the time of a crash, and perform other analyses that could help understand why people were having to reboot their servers so often. The information gleaned from this experience has led to some interesting conclusions and a number of new directions for Microsoft. The lessons learned apply to all server OSs.

As it turned out, the number-one cause of server failure was termed "human error." Often, server operators restarted their servers for the following reasons:

• They didn't know how to end a process correctly.

• They made a mistake in a command at the command line.

• They installed software incorrectly.

• They set a system setting incorrectly.

• They attempted to "refresh" their server (that is, return to a known state) by restarting the server instead of fixing an underlying problem that they didn't understand.

• They attempted to clear out a log that was full by restarting their system.

Something on the order of a little more than 50% of all system restarts fall into one of these categories.

The second most significant cause of failures was device driver errorsthat is, either the wrong device driver was installed, the device driver was poorly written in some way, or there was an interaction between the device driver and the OS that led to a crash. In many cases, the error involved the device driver trying to write to protected memory in the system. In a very small number of cases, problems in the OS were traced back to memory leaks.

A memory leak occurs when a driver or other software program writes to memory that is assigned to a system process or to another application. When the system tries to use the "corrupted" memory contents, it may hang or cause some other unpredictable result. This can happen when the software is written in a nonstandard way and becomes unsupported in future versions of the OS or simply due to poorly written code.

With this experience, Microsoft changed its tactics somewhat, moving from a "PC Standard Architecture" to a "Certified for Windows XXX" program, where peripherals, software, and device drivers go through quality assurance testing in order to attain this status.

If you have installed a video card, tuner, USB hub, storage device, or almost any other type of peripheral, you may have encountered a dialog box that says "This software is not certified for use with Windows XP...." In most instances, the vendor's documentation tells you to ignore the warning and install the software anyway. The Windows certification program costs a vendor thousands of dollars in fees. For a low-priced item such as a USB hub, most vendors don't bother to go to the trouble. Higher-priced items such as RAID controllers almost always seek this type of certification.

The Certified for Windows program is not all that valuable for a Windows workstation, but it carries added significance for servers. When you want a server to be more reliable, careful selection of the drivers, software, and peripherals that you use is part of "hardening" the server. Hardening is the process by which the server is made more reliable or more secure.

Microsoft committed to a Datacenter Server program, where large SMP computers of 16, 32, and 64 processors are running in the enterprise, so the Datacenter certification is an essential element of any deployment. The Datacenter program actually went further than that. You can only obtain a Windows Datacenter system from a certified vendor, and the vendor is required to certify the systems that they ship as well as to strictly use items that are approved on the Datacenter hardware compatibility list. Essentially, this means that for large Datacenter systems, Microsoft moved to an OS/hardware platform combination, even if Microsoft itself isn't the one delivering the system.

A large number of other OSs are software-only solutions that rely on industry-standard PC or server hardware platforms. In this category are Linux and its many flavors, Sun Solaris for Intel, Novell NetWare, DOS, and a whole host of real-time and other specialized OSs.

Selecting a NOS that is software based is a compromise: You trade a certain level of supposed reliability for access to a much larger universe of hardware. More often than not, it is less expensive to create a NOS server than it is to buy a complete proprietary solution. However, it is certainly possible and often straightforward to build servers with software NOS and even with open source applications such as the webserver Apache or the file server Samba and have them run very reliably for very long periods of time.

Hardware/Software Packaged NOSs

Many computer companies prefer to deliver their OSs as part of a hardware-packaged solution. There are two reasons for this. First, a NOS vendor can ensure a certain level of reliability if it controls not only the OS software but the hardware as well. This rationale has worked well in the server sector, with vendors' products as diverse as IBM AS/400, Sun Solaris, Apple Macintosh, and others. The second reason a bundled server solution makes sense from the vendor point of view is that it allows the vendor to achieve acceptable profit margins on hardware that isn't possible for commodity hardware sold on the open market.

Thus hardware/NOS bundles usually offer better reliability than software-only solutions, provided that you stay within the limits of the software the vendor recommends. You pay for this reliability by having to pay a premium on the hardware you buy as well as by having fewer hardware choices available should you try to upgrade your server. Does this compromise mean that choosing a hardware/NOS bundle is a more reliable than choosing a software-only NOS? Not necessarily.

Right out of the box, most hardware/NOS bundles are rigorously tested and reliable, for the most part. As you start to add additional applications to a server, add peripherals, and change the software on the server, chances are that any hardware/NOS bundle's reliability won't be any better than a carefully chosen software-based NOS solution. Anyone who has loaded large number of extensions on a Macintosh desktop knows that third-party software lowers the reliability of the desktop, often in ways that aren't predictable.

Server Installation/Administration Requirements

As your network grows, it becomes impossible to manage each system individually. Therefore, management tools play a central role in how well one NOS can serve a network's needs. These tools have a direct impact on your bottom line because most organizations find that their number-one IT cost is the salary and overhead of the IT staff. If you can't measure the savings in terms of the salary saved, consider that the time your IT staff spends performing what is essentially maintenance tasks limits the attention that they can pay to implementing new systems. That cost is really a measure of lost opportunity, and the impact is often that your business doesn't grow or you don't achieve economies that a NOS should allow.

Let's briefly look at some of the management tools that should affect your thinking when considering an NOS.

Server Upgrade and Patch Management

All NOS vendors are constantly revising their software to take advantage of advances in hardware design, add new features, and provide services that are becoming standards in the industry. The large NOS vendors typically release major OS upgrades every two to three years, with point releases every six months or so.

Not all NOS versions are equal. Although the latest release may give you the most features, it may not be the most stable version of the NOS for your server. There are sweet spots in the technology, and wise IT staff know, often through word of mouth or through personal experience, which versions those are. Most analysts following the computer industry generally advise their clients to be very wary of deploying major version releasesand particularly so when there are significant changes in the technologies.

An example of a major release where the NOS changed substantially was Novell NetWare 5 and Microsoft Windows 2000. What they both have in common is that they were the versions of those two NOSs where the vendors introduced directory services: NDS in March 1997 and AD in February 2000, respectively.

Typically, the first versions of these breaks in technology have a low adoption rate, often no more than 15% in the first year or 18 months because extra effort is needed to migrate software and settings into the new version. As a general rule, the second release of a NOS is significantly better than the first release. Thus, many people tend to wait and are advised to wait for a Service Pack 2 or a .1 release before moving to a new NOS platform. Many enterprise applications running on stable and established NOSs never get upgraded at all because it is cheaper and less troublesome to simply replace the equipment and the NOS at the same time, migrating data as needed.

Whatever policy you decide to adopt regarding system upgrades, it's important to have a clear policy regarding system software patches, given that you are much more likely to be enticed to apply a patch to solve any current system issues than to replace the NOS. There are three approaches:

• Install all system patches and upgrades as soon as they become available in order to stay current. Most NOS vendors want their customers to be current, and the rationale that is used is that doing so lets the company close any security holes, improve performance, and remove any error conditions they find.

• Install only selected patches or upgrades after waiting an appropriate amount of time to determine stability. Many IT administrators feel that waiting an appropriate amount of time for a service pack or an upgrade to prove itself is a cautious approach that is less likely to cause them trouble.

  It is certainly true that some patches cause more problems than they actually fix. Many NOS vendors simply don't apply the same amount of resources to certifying their patches' compatibility as they do to the version upgrades of their releases.

• Stay with a version of the NOS that is stable and well known. If you have a production server that is operating correctly and reliably, an argument can be made for leaving the server alone and not changing anything. In situations where a server is operating well and is well protected behind up-to-date corporate firewalls, it is reasonable to believe that you already have enough protection in place to keep out intruders and for your server to do its job.

What upgrade rules should you employ? There isn't a correct answer to this question. If you have an intimate connection to both your NOS and enterprise application vendor, then it may make sense to stay current and rely on your VAR or vendor to help you through any difficulties.

However, and this is a big drawback, many upgrades are expensive to apply. If an upgrade falls outside your service contract, working that upgrade into your budget may cause you to forgo additional equipment, software, or services that are needed elsewhere.

Note

There is one inviolable rule for any organization that adopts a strategy of staying current with upgrades and patches: to always make sure you can revert to a previous version of the OS. For example, when Windows offers you the option of saving your previous OS files in order to uninstall an upgrade, you should accept that option. Or when you install a patch into Sun Solaris, that patch should appear in the Patch Manager and give you an uninstall option. Failing that, it is absolutely critical that you create a fully functional image backup of your server prior to applying any system software upgrade or patch. Many IT administrators would also argue that the same is true anytime you upgrade or patch your main server application(s).

The middle course of sticking to known good versions of a NOS is a conservative choice but one that acknowledges that the system software tends to get better over time. If you choose this course of action, then a rule of thumb might be to wait until the second patch of a major upgrade before upgrading or to wait three or four months from the time a significant patch is released before you apply that patch. Those three or four months will give you time to consult with colleagues, watch the online forums or blogs, and consult with system vendors to determine whether any significant problems have occurred. The only downside to this approach is that you won't get the full lifetime of the upgrade or patch that you're paying for. Given that an enterprise application has about a three-year life span at most, a four-month delay will cost you about 10% of the working life of the service.

The last course of action falls into the category of "If it's not broken, don't fix it." Its advantage is that you minimize both your additional costs for upgrades and the potential labor costs necessary to support an upgrade. Although vendors want to sell you the next great thing, you really have to ask yourself if this is money and effort well spent. If you are thorough and put together a costbenefit analysis, in most cases upgrading a stable and reliable server or server application isn't worth doing. Many companies opt to leave their systems in place and intact until they are forced to or required to do a complete system upgrade. There's some chance that you will have a system that will no longer be supported by the original vendors, but there's always someone who will service a legacy system come what may.

A NOS's ability to manage upgrades and patches intelligently is a significant value-added proposition. Of all the NOSs, Microsoft provides probably the best-developed update site (www.microsoft.com/isaserver/default.mspx; see Figure 18.3). It's automated and has both notification and a push capability. Sun's upgrade site is also well implemented, although it's not quite as automated as Microsoft's.

Software Security Limitations

You need to consider several important aspects of software security when implementing both an application and a NOS. First, there is the issue of how authentication is done. The OS contains its own native authentication mechanism. However, depending on the application, any one of the following may be true:

• The application implements its own security scheme and logon.

• The application relies on the security scheme and logon services of the NOS.

• The application passes its logon and security scheme through to the NOSthe so-called pass-through identification scheme.

Each of these options carries unique benefits and risks. A single-logon scheme is less complex and easier to manage, but when it is compromised, it allows universal access to any security-dependent application. A multiple-logon scheme is more complex and requires more overhead to support. One part of your system can be compromised without the entire system being compromised.

A number of companies, including some of the NOS vendors, offer single-logon or single-signon servers. Most of these servers use industry-standard protocols such as Kerberos to create session tickets that can be used to validate a user and his or her session. Anyone who has used Kerberos knows that not all implementations of the standard are the same, and the small differences can lead to problems when you are trying to make different platforms interoperate. Still, single-logon servers offer you the potential of being able to work with many of these differences, and most of these products are very good at servicing both Windows and UNIX requests.

Single-signon or single-logon is not a feature that NOS vendors really want third-party tools to perform. Authentication is at the heart of what NOSs are meant to provide. That's why over time each NOS incorporates more and more heterogeneous logon support for clients. As much as is possible, NOS vendors also lead their application developers to rely on their particular directory service for their authentication needs.

No software can be 100% secure. The more applications that exist, the more users who use an NOS, and the more systems that are deployed, the greater the risk is for a NOS to be attacked. It's a fact that of the NOSs discussed in this chapter, the greater proportion of viruses, worms, denial of service, spyware, and other attacks are aimed at the Windows server platform. That this is largely a function of greater opportunity, but some people make an argument that it is also due in part to the way Microsoft enables its software to be manipulated by users. Be that as it may, the following general advice on software security is worth considering:

• Keep your servers simple; don't run any services that you don't need.

• Keep your servers up-to-date with patches and updates.

• Keep your servers physically secure. Don't allow physical access to a server to anyone who doesn't need to have it.

• Protect your server's software by using good password management practices.

• Employ appropriate antivirus, spyware, and intrusion-detection software.

• Set up and actively maintain a strong firewall.

• Be vigilant about securing your systems from internal users as well as external ones.

• Keep active logs that can be analyzed for problems.

• Have your security audited by reputable outside consultants and companies on a regular basis.

Server Documentation and Support

Supporting a server is considerably different from supporting a desktop system. Whereas a desktop is used by one user or a limited number of users and perhaps supported by one administrator, servers may be touched or modified by a large number of staff or outside support personnel. This fact argues for a different handling of a server systemhandling that provides in-place tools for documentation of the server.

It's a good idea to document a server's contents so that if you or another staff member must access or maintain the server, you will know what the server contains. Nearly all motherboards ship with a decal showing the layout of the motherboard, and the board manufacturers expect you to paste that decal someplace where you will see it when needed. A great location for a motherboard diagram is on the removable side panel of a server housing, on the bottom of the server, or on the case somewhere. If none of those locations are convenient, you should consider attaching a plastic mailing window like the ones used to enclose invoices on packages to the outside of your server. If your motherboard does not come with a schematic or if that schematic is lost, you can print the layout from the server's user's guide. These guides are often posted on the vendors' sites as PDF files.

Many people also create a label that contains pertinent information about the system that an administrator would need to know, such as name, IP address (if a static address is assigned), NIC MAC addresses, and so on. That information is very useful when you are looking for a specific system in a room full of servers or in a server rack. At the very minimum, the name of the server and its logical network location should appear on the front of the server, in plain view. You might want to create a list similar to the one shown in Table 18.4.

Your list might contain additional or fewer pieces of information. If you set up a database or spreadsheet for all your servers, you can put this information in a very convenient and searchable format. It's time-consuming to set up this system, but it's much more time-consuming to figure out these details after the fact.