Linux and the Unix Philosophy

5.2 Tenet 7: Use shell scripts to increase leverage and portability

If you want to take full advantage of software leverage, you need learn how to use shell scripts effectively. We're not going to show you how to use them here. There are already plenty of books on the subject. Most will show you how to use them. Instead, we're going to focus on why you should use them.

Before entering into this discussion, I must caution you that many Unix and Linux kernel programmers disdain shell script programs. They believe that writing shell scripts is not the macho thing to do. Some even equate "experienced shell programmer" with "lightweight Linux programmer." My guess is that they're simply jealous because writing shell scripts doesn't involve much cerebral pain. Or perhaps it's because you cannot use shell scripts in the kernel. Maybe someone will write a Linux kernel that will allow the use of shell scripts in the kernel itself someday. Then they, too, can more fully appreciate the benefits of software leverage.

In examining the case for shell scripts, you may get the impression that the author is anti-C; that is, programs should never be written in a portable language like C anymore. If so, you're missing the point. There are times when writing a program in C instead of the shell makes perfect sense. But those times occur much less often than one might suspect.

Similarly, because of all of the hype surrounding object-oriented languages and tools these days, it's easy to fall into the trap of bypassing the shell and writing all software in a popular language such as Java. While in many ways Java is an excellent language, especially with respect to code reuse, it is still a compiled language (i.e., you must compile a Java program from the source code before the Java run-time engine can interpret it). This section will make you aware of the shell alternative to Java, if it doesn't change your mind altogether.

5.2.1 Shell scripts give you awesome leverage

Shell scripts consist of one or more statements that specify native or interpreted programs and other shell scripts to execute. They run these programs indirectly by loading each command into memory and executing it. Depending on the kind of statement, the top-level shell program may or may not choose to wait for individual commands to complete. The executed commands have been compiled from as many as a hundred, a thousand, or even a hundred thousand lines of C source code, most of which you did not write. Someone else took the time to code and debug those programs. Your shell script is merely the beneficiary, and it uses those lines of code to good advantage. Although you have expended comparatively little effort on your part, you gain the benefit of as many as a million or more lines of code. Now that's leverage.

In the multilevel marketing of plastic housewares, the trick is to get other people to do much of the work for you. You're trying to create a situation where someone else sows and you reap part of the reward. Shell scripts provide that opportunity. They give you the chance to incorporate the efforts of others to meet your goals. You don't write most of the code used in a shell script because someone else has already done it for you.

Let's look at an example. Suppose you wanted a command to list the names of a system's users on a single line. To make it more interesting, let's separate each user's name by commas and display each name only once, no matter how many sessions a user may have opened on the system. This is how our command might look as a shell script written in the bash shell, a popular Linux command interpreter:

echo 'who | awk '{print $1}' | sort | uniq' | sed 's/ /, /g

Although this shell script consists of a single line, it invokes six different executables: echo, who, awk, sort, uniq, and sed. These commands are run simultaneously in a kind of series-parallel progression. Except for the who command, which starts the sequence, each command receives its data from the previous command in the series and sends its output to the next command in the series. Several pipes, denoted by'|' characters, manage the data transfer. The final command in the sequence, sed, sends its output to the user's terminal.

Each command works with the others synergistically to produce the final output. The who command produces a columnar list of the users on the system. It feeds this to awk via the pipe mechanism. The first column in the output from who contains the names of the users. The awk command saves this column and throws away the rest of the data generated by who. The list of users is then sent to sort, which places them in alphabetical order. The uniq command discards any duplicate names caused by users who may have logged in to several sessions at once.

Now we have a sorted list of names, separated by "newlines," the Linux end-of-line or line-feed character. This list is sent to the echo command via a "back-quoting" mechanism that places the output of the previous commands on the echo command line. The bash shell's semantics here dictate that single spaces replace all newlines. Finally, our string of user names separated by spaces is sent to the sed command, and the spaces are converted to commas.

While this might seem quite amazing if you have never seen a Linux system before, it is a typical Linux-style command execution. It is not unusual to invoke multiple commands from a single command line in a shell script.

How much code was executed here? The shell script writer took less than a minute to write the script. Others wrote the commands invoked by the script. Under one version of Linux available today, the six commands used contain the following numbers of source code lines:

One line in this shell script executes the equivalent of 9,535 lines of source code! Although this is not an extraordinary number, this many lines are enough to prove our point. We have increased our power by a factor of 9,353 to 1. Again, we have leverage.

This was a simple example of a shell script. Some shell scripts today span several dozen pages containing hundreds of command lines. When you account for the C code behind each executable, the numbers really start to add up. The resulting leverage boggles the mind. As we shall see later, this phenomenon even impressed Albert Einstein.

5.2.2 Shell scripts leverage your time, too

Shell scripts have an intrinsic advantage in that they are interpreted rather than compiled. In a standard C-language development environment, the sequence of events goes like this:

THINK-EDIT-COMPILE-TEST

The shell script developer's environment is one step shorter:

THINK-EDIT-TEST

The shell script developer bypasses the COMPILE step. This may not seem like a big win given today's highly optimized compilers. Used with speedy RISC processors, such compilers can turn source code into binaries in the blink of an eye. But today's applications are rarely single files that exist by themselves. They're usually part of large build environments that lean toward complexity. What used to take a few seconds to compile on a fast machine may now take a minute or more because of increasing levels of integration. Larger programs can take several minutes or more. Complete operating systems and their related commands can require hours to build.

In skipping the compilation step, the script writers remain focused on the development effort. They don't need to go for coffee or read mail while waiting for the command build to complete. Proceeding immediately from EDIT to TEST, they don't have time to lose their train of thought while waiting for the compiler to finish. This greatly accelerates the software development process.

One key point to consider is execution time versus compilation time. Many smaller applications (remember that we're talking about Unix/Linux here) can accomplish their tasks in a few seconds or so. If the compilation takes significantly longer than that, then using a script may be desirable. On the other hand, if a script may take several hours to run where a compiled program would execute the same in several minutes, then by all means write a C program. Just be sure that you have carefully considered whether there might be a way to save time by scripting various components.

Alas, there is an advantage the C programmer has over the shell script author-namely, an enhanced set of tools for debugging. While developers have created a respectable amount of diagnostic software for the C language, the choices of debugging tools for script writers are severely limited. To date no full-featured debugger for shell scripts has emerged. Shell script writers must still rely on primitive mechanisms such as sh -x to display the names of the commands as they execute. Convenient breakpoint facilities are nonexistent. Examining variables is a tedious process. One could argue that, given the ease of shell programming, more comprehensive debugging facilities are unnecessary. I suspect that most shell script writers would disagree.

What about large IDEs such as Microsoft's Visual Studio or Borland's JBuilder? Don't they provide superior debugging and editing facilities? Yes, they do. They are very good at hiding the complexities of the underlying technology from the developer. Sometimes that's a good thing. But frequently the complaint is that when things go wrong, one still has to look underneath the covers to see what the IDE has done. The pretty outer shell that had made the language easier to use now becomes the developer's nemesis as he or she tries to figure out what went wrong behind the scenes.

Shell scripts, on the other hand, keep things very visible. Everything they do is right there in front of you. Nothing is hidden behind dropdown menus in a slick GUI. The irony is that shell scripts are more visual than the so-called visual products.

5.2.3 Shell scripts are more portable than C

A sure way to leverage your software is to make it portable. Earlier we learned that it is important that you share your software with others. Any program moved easily from one platform to another is likely to be used by many people. The more people using your software, the greater the leverage.

In the Linux environment, shell scripts generally represent the highest level of portability. Most scripts that work on one Linux system are likely to work on another with little or no modification. Since they are interpreted, it is not necessary to compile them or otherwise transform them for use. It is possible to make a shell script nonportable by design, but such instances are rare and are usually not encouraged.

Shell scripts also tend to lack the stigma of "ownership" commonly associated with C source code. People rarely become protective of them. Since they are plainly visible to everyone, no one considers it his or her corporate duty to protect their contents. Still, a measure of caution is in order here. Copyright laws in the United States and other countries provide protection for shell scripts. Whether one would want to copyright a shell script is a matter for the lawyers.

5.2.4 Resist the desire to rewrite shell scripts in C

The chapter on portability urged you not to rewrite shell scripts in C because next year's machine will make them run faster. Since shell scripts are usually highly portable, moving them to Next Year's Machine generally involves virtually no effort. You copy them to the new machine and they run. Period. No mess, no fuss.

Unfortunately, the ability to leave well enough alone is not a virtue typical of programmers. If programmers can find a spare moment to tinker with a shell script, you can bet that they will (1) add some new features to it, (2) attempt to make it run faster by refining the script itself, or (3) try to improve its performance by rewriting part or most of it in C. Can you guess which is most tempting?

The desire to rewrite shell scripts in C stems from the belief that C programs run faster than shell scripts. This eats away at the programmer's desire for a neat, orderly world where everything is well tuned and highly efficient. It's an ego thing. He knows he could have written it in C, and it would have run faster from the beginning. For whatever reason, he didn't. Guilt sets in. If you ask him why he chose to write a program as a shell script, he'll mumble something like, "It was all I had time for." He'll follow this excuse with a promise that, when he gets more time, he will rewrite in C.

It's time to get over it. It's doubtful that he will ever get the chance. Any programmer worth his salary will be much too busy to go back and rewrite a shell script that already works well enough to meet the needs of its users. Life is much too short for that.

Furthermore, the belief that C programs run faster than shell scripts bears some scrutiny. In the first place, a shell script invokes C programs to accomplish its task. Once the C programs are loaded for execution, "pure" C programs enjoy no performance advantage over those called from within a script. Most Linux systems today have tuned the command execution routines so well that the time required to load a program for execution is relatively small compared with the time to perform the complete task.

If you really want your shell scripts to run quickly, then you must look into different ways of solving a problem. Too often, users and programmers fall into a rut, saying that's the way I've always done it, and that's the way I'll always do it. Instead, you need to train yourself to overlook the obvious approaches and find techniques that use the available resources in novel ways.

By way of example, let's look at a situation I ran into a few years ago. I was working in an environment where I received between 50 and 100 pieces of e-mail each day, or about 300 per week. Although I could read and delete some messages, I had to save others for future reference. It wasn't long before I had over 2,000 mail messages spread across 100 directories on my Unix system. Accessing the right one quickly was becoming difficult.

The obvious solution would be to use the Unix grep command to locate each message based on searching its contents for a particular text string. The problem with this approach was that it was very time consuming, even for a fast program like grep. I needed something better.

After a few false starts, I came up with the idea of indexing all of the mail messages. I wrote a shell script that did a grep on every file in the directories using all possible text strings. In effect, the index I created had "pre-grep'ed" the files. When I wanted to find a mail message, I looked in the index for a text string contained in the message, such as the author or the subject. The index would return a pointer to the file or files containing the string. This approach turned out to be far more efficient than running grep every time to find a message, even though it uses a shell script.

This approach worked out so well that I passed the idea on to a co-worker who implemented it on a much larger scale. Within a couple of months, he was using the same technique to index huge numbers of files on our systems. He refined the shell scripts until they could locate a string in several hundred megabytes of text in a few seconds-on our slowest machine. The application worked so well that most people did not believe that it was written as a shell script.

While grep may be much faster than many commands run from a shell script, by using a different approach it is possible to produce a shell script with remarkably higher performance than grep alone. It's just a matter of looking at the problem from a new angle.

In this chapter we have explored the value of leverage. We have seen that leverage can be an especially powerful idea when applied to software. Like any form of compounding, software leverage produces extensive effects for small amounts of effort. Each small program is a seed that becomes a mighty oak when sown.

Shell scripts remain an optimum choice for enhancing software leverage. They allow you to take the work of others and use it to your advantage. Even if you have never written a sort routine, you can have a sort routine written by an expert at your disposal. This kind of approach makes everyone a winner, even those who cannot program their way out of a paper bag.

One of the strengths of the Unix philosophy is its emphasis on a plethora of tiny commands. The shell script is a way to unify them into a powerful whole, thereby giving even inexperienced programmers the ability to perform difficult tasks with ease. Using shell scripts, you can stand on the shoulders of giants, who themselves are standing upon the shoulders of giants, ad infinitum. Now that's leverage!

Albert Einstein once said, "I have only seen two miracles in my life, nuclear fusion and compound interest" (italics added). For all of his wonderful theories, these two ideas evidently impressed him most. He understood that a small amount of something, multiplied repeatedly, can grow to miraculous proportions. It took a keen mind like his to recognize the power in this simple idea.

On the other hand, maybe his wife used to sell Tupperware.