Unix for Mac OS X 10.4 Tiger: Visual QuickPro Guide (2nd Edition)

It is extremely common when using Unix to want to search for specific words or strings of characters inside text files or to search the long output of some command. The main Unix command for this is grep .

Using grep

To search for a string in a text file:

• grep string file

  For example,

  grep memory /etc/rc

  finds all the lines in the file /etc/rc that contain the string memory (in this case there was only one line, so only one line was returned) ( Figure 4.22 ).

  Figure 4.22. Using grep to find the string memory in a file.

  localhost:~ vanilla$ grep memory /etc/rc echo "Starting virtual memory" localhost:~ vanilla$

To make the search case insensitive:

• Use the -i option. For example,

  grep -i apple /etc/services

  Figure 4.23 shows part of the result. Note how both apple and Apple are found.

  Figure 4.23. Performing a case-insensitive search with grep . Your output depends on your machine's configuration. (Partial output shown.)

  localhost:~ vanilla$ grep -i apple /etc/services . . . asip-webadmin 311/udp # AppleShare IP WebAdmin asip-webadmin 311/tcp # AppleShare IP WebAdmin # Ann Huang <annhuang@apple.com> aurp 387/udp # Appletalk Update-Based Routing Pro. Aurp 387/tcp # Appletalk Update-Based Routing Pro. Appleqtc 458/udp # apple quick time Appleqtc 458/tcp # apple quick time # <murali_ranganathan@quickmail.apple.com> appleqtcsrvr 545/udp # appleqtcsrvr appleqtcsrvr 545/tcp # appleqtcsrvr . . . localhost:~ vanilla$

Tips

• To see the output one screenful at a time, pipe the output of grep through the less pager:

  grep -i apple /etc/services less

• You can use to interrupt the command and get back to the prompt.

To search for a string in multiple files:

• Simply add more files to the argument list, perhaps by using wildcards. For example,

  grep -i network *

  As we discussed in Chapter 2, the * (asterisk) is a special character expanded by the shell to be a list of many files.

  This is probably the most common use of grep to find all the occurrences of a string throughout multiple files. When searching multiple files, grep adds the filename at the beginning of each line of output so that you know which file each line came from:

  grep " boot" /etc/rc*

  gives output as shown in Figure 4.24 .

  Notice the use of quotes to make the search string include a space character (searches for " boot" instead of just boot ); TRy removing the quotes to see the difference.

  Figure 4.24. Using grep to search multiple files for a string that includes a space.

  localhost:~ vanilla$ grep " boot" /etc/rc* /etc/rc: echo "CD-ROM boot procedure complete" /etc/rc: echo "Configuring kernel extensions for safe boot" /etc/rc.netboot:# Prevent inadvertent problems caused by interrupting the shell during boot. localhost:~ vanilla$

Tip

• If you give an argument to grep that is a directory (instead of a regular file), grep gives you an error message saying that the file "is a directory." You can tell grep to skip directories by adding the -d skip optionfor example,

  grep -d skip NETWORK */*

To recursively search all the files in a directory:

• Use the - r option. For example,

  grep -ri network /System/Library/StartupItems

  performs a case-insensitive search (note the -i option) of all the files inside /System/Library/StartupItems and all its subdirectories.

Tip

• Using the -r option to grep can cause the command to take a long time to complete, so you might want to redirect the output to a file and put the job in the background (review "Redirecting stdout " and "Running a Command in the Background" in Chapter 2):

  grep -ri network /System/Library/StartupItems > found.txt &

To find the lines that do not match:

• Use the -v option.

  grep -v tcp /etc/services

  finds all the lines in the directory /etc/services that do not contain the string tcp .

To search the output of another command:

• Pipe the output of the other command through grep .

  last grep reboot

  shows all the reboots this month.

  (The last command shows a history of logins to your machine, as well as crashes and reboots. Type man last .)

Tip

• Use multiple grep s in the pipeline to narrow your request.

  last grep reboot grep "Feb 16"

  finds all the reboots on February 16. See how the first grep filters the output of the last command, and the second grep filters the output further, narrowing down the result.

The grep and egrep ( e for extended ) programs have a huge number of options. Table 4.1 lists some of the more common ones. See the Unix manual for more (type man grep ).

Table 4.1. Options for grep and egrep

^[*] These options are not as common as the others, but Mac OS X does have them.

Using patterns in your search

You will often want to search for something more complicated than a literal string of characters. You might want to search only for lines that begin with a certain string, or for lines that contain a range of dates, such as Feb 15 or Feb 16 .

The egrep command supports an extremely powerful (and complex) pattern-matching system called regular expressions. The re in egrep stands for regular expression . (The grep command also supports a small number of regular expressions. To avoid switching back and forth, we will stick with egrep here.)

Regular expressions are used in a large number of situations in Unix, not only with the grep and egrep commands. For example, the Unix programs sed , awk , and vi all use regular expressions, as do the C, Perl, Tcl, Python, and Java programming languages. The basic syntax of regular expressions is the same or very similar in a variety of situations, so once you learn how to use them in one area, you have a head start on using them in another.

Regular expressions are built up like mathematical formulas (see the sidebar "Learning More About Regular Expressions"). You can do a lot with the few rules we'll show you here.

An important concept to grasp in using regular expressions is that when you search for "hello," you are really searching for a pattern consisting of six atoms ( h, e, l, l, and o ). In regular expressions, an atom is a part of the overall expression that matches one character. The most common kind of atom is simply a literal character, so the atom h matches the letter h . But atoms don't stop there. For example, the atom [a-d] matches one letter from the range a, b, c, or d . (The [and] are used to define a set of characters.) So when you see the word atom used in the examples below, keep in mind that an atom can be as simple as one character, or it can be a more complex notation that matches one character from a list of possibilities.

Regular expressions have a few major rules, which are demonstrated in the examples below.

Also, note that you always enclose the pattern inside single quotes; this is to prevent the shell from misinterpreting any of the characters used in regexes (as they're traditionally called) that have special meaning to the shell: [] {} . * .

To find lines starting with a specific string of characters:

• Put a ^ at the beginning of the string;

  grep ^# ~/bin/reverse

  finds only lines beginning with # , and grep -v ^# script.pl would find all lines not beginning with # . Figure 4.25 shows the output from both command lines.

  Figure 4.25. Using a pattern that matches the character at the start of a line.

  localhost:~ vanilla$ cat ~/bin/reverse #!/usr/bin/perl # reverse cat script # @file = <>; while ( @file ) { print pop(@file); } localhost:~ vanilla$ grep ^# ~/bin/reverse #!/usr/bin/perl # reverse cat script # localhost:~ vanilla$ grep -v ^# ~/bin/reverse @file = <>; while ( @file ) { print pop(@file); } localhost:~ vanilla$

  The ^ character in a regex is called an anchor because it anchors the search string to the start of the line.

To find lines ending with a string:

• Putting a $ at the end of the string

  grep today$ filename

  finds only lines ending with today .

  The $ anchors the search string to the end of the line.

Tip

• If you anchor a pattern to both the start and the end of linefor example, ^word$ then only lines that exactly match will be found. That is, only lines that consist solely of the pattern, with nothing before or after.

Testing regular expressions

Regular expressions can get quite complex, and learning how to use them takes practice. Luckily there is an easy way to test them to see if they match what you think they will match.

If you use one of the grep commands ( grep or egrep ) with a pattern but without giving it a filename or input from a pipe, then it waits for you to type input and repeats back to you any lines that match.

In most of our examples we use the egrep command because it supports extended regular expression. In cases where we don't need extended regexes, we will use plain old grep .

To test a regular expression:

Tips

• In each of the tasks below, try testing the regular expression with several different lines of input to see how each one behaves. Try to predict what will and will not match.

• If you make a mistake while typing a test expression, you may notice that the key doesn't work the way you expect. Instead, press , which will erase the whole line. This is a very useful trick that works in many command-line situations.

To find lines containing a string in which one character can vary:

• Use square brackets to create an atom from a list of characters. For example:

  egrep '[FNW]orm A-100'

  The atom [FNW] means "one character that matches any of the three atoms F , N , or W ." Figure 4.27 shows examples of testing this pattern with three different matching lines.

  Figure 4.27. Using an atom that matches any character in a list.

  localhost:~ vanilla$ egrep '[FNW]orm A-100' Worm A-100 is a very virulent worm. Worm A-100 is a very virulent worm. Norm A-100 isn't really normal. Norm A-100 isn't really normal. Form A-100 must be filled out in pink ink. Form A-100 must be filled out in pink ink. ^C localhost:~ vanilla$

To create an atom that is anything not in a list:

• Use the ^ character as the first character in the list. For example:

  egrep '[^FNW]orm A-100'

  Figure 4.28 shows examples of testing this pattern (again, you type the text that's in bold). Matching lines will not contain any of the following:

  Form A-100 Norm A-100 Worm A-100

  Figure 4.28. Using an atom that matches any character not in the list.

  localhost:~ vanilla$ egrep '[^FNW]orm A-100' Worm A-100 no longer matches. Either does Norm A-100. Nor even Form A-100. But form A-100 does because of the lowercase f. But form A-100 does because of the lowercase f. See you in Dorm A-100 See you in Dorm A-100 how about this? ^C localhost:~ vanilla$ /Applications (Mac OS 9)/FileMaker Pro 5 Folder/FileMaker Pro

  This line will match:

  Dorm A-100

Tips

• Notice how the last example is different from the -v option (described earlier). The -v option finds all lines that do not match the whole pattern. The example here finds lines that contain 'orm A-100' but only if the first letter before 'orm' is not F , N or W .

• Notice that the use of ^ here is different from using ^ as the start-of-line anchor. The ^ behaves differently when it is the first character in a square- bracket list.

To create an atom from a range of numbers:

• Use egrep and put square brackets around the range:

  egrep 'Feb 1[5-9]' mail.log

  The [5-9] means " 5 , 6 , 7 , 8 , or 9 " in regular-expression language.

Tip

• You can use multiple lists, such as [2-3][0-5] (that means " 20-35 ").

To create an atom from a range of letters:

• Use egrep and the square brackets, and a-z for lowercase, A-Z for uppercase. For example:

  egrep 'Appendix [B-D]' book.txt

Tip

• You can make the range case insensitive by using lowercase and uppercase ranges in the atom:

  egrep 'Appendix [B-Db-d]' book.txt

To use a wildcard character:

• Use the . character to mean "any single character." For example:

  egrep '.oy'

  behaves as shown in Figure 4.29 . Notice that the line that begins with oy doesn't match, because there is no character before the oy .

  Figure 4.29. Using a . (period) to match any single character.

  localhost:~ vanilla$ egrep '.oy' toy should match toy should match so should boy so should boy even coy even coy but not this line oy this one doesn't match either! but oy, this one does. but oy, this one does. ^C localhost:~ vanilla$

To find lines in which an atom is repeated zero or more times:

• Use the * quantifier. (That's an asterisk on your keyboard, but it's generally referred to as a star in Unix speak.) For example,

  grep 'Form A-10*'

  behaves as shown in Figure 4.30 .

  Figure 4.30. Using * to match zero or more of an atom.

  localhost:~ vanilla$ grep 'Form A-10*' Form A-100 Form A-100 Form A-10 Form A-10 Form A-1 Form A-1 ^C localhost:~ vanilla$

  In this case the atom is the 0. The * quantifier means "zero or more of the preceding atom." Notice how it found the line in which did not appear at all.

Tip

• Be careful when using the * character. If an argument contains a * and you don't want the shell to expand it to a list of filenames (see "Wildcards" in Chapter 2), then you must make sure that any use of * on the command line is enclosed in quotes or that you escape the * by preceding it with a backslash; for example:

  grep fo\*bar *.txt

  In that case the shell would expand the second * to match all the filenames that end in .txt, but the shell would pass the string fo*bar to grep as an argument without expanding the * .

More rules and tools for building regex atoms

The regex examples shown above allow you to perform some fairly sophisticated matching, but there are a lot more ways to create atoms and patterns. Table 4.2 describes several additional tools you will find useful in constructing more complex patterns. All the tools and rules in Table 4.2 require egrep . The real key is to experiment using the testing approach described above.

Table 4.2. Rules and Tools for Regex Atoms