Overview

Not all text data is suitable for storage in a relational database table. For example, letters and memos do not fit the model of rows and columns . A file that is processed as a sequential list of bytes, such as a letter or memo, is known as a stream file. Stream files are widely used in business.

Stream files are not organized into rows (records) and columns (fields). A stream-file record is a series of characters that ends with a special code, such as a carriage -return character followed by a line-feed character. Stream-file records may vary in length. One record of a file may be 20 bytes long, the next 150, and a third 12. Stream files may contain binary data (such as sounds or pictures) or text data.

Qshell is well suited to working with stream files. Most Qshell commands read a stream of characters and write a stream of characters.

Standard Files

Qshell defines three standard stream files by default. You do not have to declare, open , or close these files. The default input file is known as standard input and is abbreviated to stdin . It is assigned to the keyboard, but you may use redirection to assign it to another file, or a pipeline to read the output of another Qshell script or utility.

There are two predeclared output files: standard output ( stdout ) and standard error ( stderr ). The first is for the normal output of commands, while the second one is for error messages. Both files are directed to the display by default when Qshell runs interactively, but may be redirected to disk files or piped to other Qshell utilities. The fact that there are two output files allows you to separate good output from error output.

The Qshell interpreter is a good example of the use of stream files. In an interactive Qshell session, the input to Qshell is the keyboard, and the output device is the display. The session is a conversation. Each time you press the Enter key, Qshell reads the stream of characters you typed, attempts to carry out your instructions, and gives you a stream of characters as output, even if that output is only a prompt.

A terminal session is shown in Figure 10.1. It shows a user typing the List Directory Contents command ( ls ) to determine what .csv files were in the current directory. Qshell reads the command from stdin and sends the list of file names to stdout. The user then types the Remove Directory Entries command ( rm ) to delete a file. Qshell reads the command from stdin and responds by sending a confirmation message to stderr. Qshell reads the user's response from stdin. And so the dialog continues.

/home/JSMITH $ ls *.csv cust.csv two.csv uuu.csv /home/JSMITH $ rm -i uuu.csv rm: 001-2145 Do you want to remove the file or directory uuu.csv (Y or N)? y /home/JSMITH $ ls *.csv cust.csv two.csv /home/JSMITH $

Figure 10.1: By default, Qshell utilities use the keyboard and display devices for input.

Redirection

Stdin, stdout , and stderr may be assigned to devices through a process known as redirection, which is somewhat like an override in OS/400. Qshell supports nine redirection operators, shown in Table 10.1. (The "here" document and noclobber option are explained later in this chapter.)

The most commonly used redirection operators are <, >, and >>. They are illustrated in this section. The others will be explained when file descriptors are discussed.

The redirection operators may be separated from the file names that follow them, but it is not necessary. In other words, the following two commands are equivalent:

ls *.qsh >temp.txt ls *.qsh > temp.txt

To illustrate the use of redirection operators, the input data in Figure 10.2 is used as the basis of the examples that follow.

cat goodoleboys.txt Name Born Phone Dog Wife Shotgun Paid ========= ======== ======== ======== ========= ======= ===== Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Bubba Oct 13 444-1111 Buck Mary Jean 12 Billy BobJune 11 444-4340 Leotis Lisa Sue 12 Amos Jan 4 333-1119 Amos Abigail 20 Otis Sept 17 444-8000 Ol' Sal Sally 12 Claude May 31 333-4340 Blue Etheline 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 Junior April 30 BR-549 Percival Lilly Faye 12 Bill Feb 29 333-4444 Daisy Daisy 20 Ernest T.?? none none none none

Figure 10.2: The goodoleboys.txt file is used in this chapter's examples.

Figures 10.3 through 10.6 use the Translate Characters utility ( tr ), which can translate characters in one character set to corresponding characters in another character set. Tr works well for these examples because it reads from stdin and writes to stdout. In Figure 10.3, the redirection operator causes tr to read the goodoleboys.txt file, not the keyboard.

tr [:lower:] [:upper:] NAME BORN PHONE DOG WIFE SHOTGUN PAID ========= ======== ======== ======== ========= ======= ===== CHUCK DEC 25 444-2345 BLUE MARY SUE 12 .50 BUBBA OCT 13 444-1111 BUCK MARY JEAN 12 BILLY BOB JUNE 11 444-4340 LEOTIS LISA SUE 12 AMOS JAN 4 333-1119 AMOS ABIGAIL 20 OTIS SEPT 17 444-8000 OL' SAL SALLY 12 CLAUDE MAY 31 333-4340 BLUE ETHELINE 12 ROSCOE FEB 2 444-2234 ROVER ALICE JEAN 410 ARLIS JUNE 19 444-1314 REDEYE SUZY BETH 12 .75 JUNIOR APRIL 30 BR-549 PERCIVAL LILLY FAYE 12 BILL FEB 29 333-4444 DAISY DAISY 20 ERNEST T. ?? NONE NONE NONE NONE

Figure 10.3: Lowercase letters are translated to their uppercase equivalents. Output goes to the display because stdout is not redirected.

Figure 10.4 also read the goodoleboys.txt file and translates lowercase letters to their uppercase equivalents. In this case, though, output goes to the file upper.txt, erasing any existing data in it. If the file does not exist, it is created.

tr [:lower:] [:upper:] upper.txt /home/JSMITH $ cat upper.txt NAME BORN PHONE DOG WIFE SHOTGUN PAID ========= ======== ======== ======= ======== ======= ===== CHUCK DEC 25 444-2345 BLUE MARY SUE 12 .50 BUBBA OCT 13 444-1111 BUCK MARY JEAN 12 BILLY BOB JUNE 11 444-4340 LEOTIS LISA SUE 12 AMOS JAN 4 333-1119 AMOS ABIGAIL 20 OTIS SEPT 17 444-8000 OL' SAL SALLY 12 CLAUDE MAY 31 333-4340 BLUE ETHELINE 12 ROSCOE FEB 2 444-2234 ROVER ALICE JEAN 410 ARLIS JUNE 19 444-1314 REDEYE SUZY BETH 12 .75 JUNIOR APRIL 30 BR-549 PERCIVAL LILLY FAYE 12 BILL FEB 29 333-4444 DAISY DAISY 20 ERNEST T. ?? NONE NONE NONE NONE /home/JSMITH $

Figure 10.4: The > operator replaces data in file upper.txt.

In Figure 10.5, the translated data from the goodoleboys.txt file is appended to the upper.txt file, so its existing two records are not overwritten. Again, if the file did not exist, it would be created.

cat upper.txt I LIKE CHEESE! + + + + + /home/JSMITH $ tr [:lower:] [:upper:] >upper.txt /home/JSMITH $ cat upper.txt I LIKE CHEESE! + + + + + NAME BORN PHONE DOG WIFE SHOTGUN PAID ========= ======== ======== ======== ========= ======= ===== CHUCK DEC 25 444-2345 BLUE MARY SUE 12 .50 BUBBA OCT 13 444-1111 BUCK MARY JEAN 12 BILLY BOB JUNE 11 444-4340 LEOTIS LISA SUE 12 AMOS JAN 4 333-1119 AMOS ABIGAIL 20 OTIS SEPT 17 444-8000 OL' SAL SALLY 12 CLAUDE MAY 31 333-4340 BLUE ETHELINE 12 ROSCOE FEB 2 444-2234 ROVER ALICE JEAN 410 ARLIS JUNE 19 444-1314 REDEYE SUZY BETH 12 .75 JUNIOR APRIL 30 BR-549 PERCIVAL LILLY FAYE 12 BILL FEB 29 333-4444 DAISY DAISY 20 ERNEST T. ?? NONE NONE NONE NONE /home/JSMITH $

Figure 10.5: Because of the >> operator, new data is appended to the output file.

The tr command in Figure 10.6 translates the uppercase letters in source member MYPGM of file MYLIB/QRPGLESRC to their lowercase equivalents. Other characters are unaffected. The output is placed in source member MYPGM2 in the same source physical file.

cat /qsys.lib/mylib.lib/qrpglesrc.file/mypgm.mbr D CD S 35 DIM(3) CTDATA PERRCD(1) C *ENTRY PLIST C PARM INDEX 1 0 C PARM DESCR 35 * C INDEX IFLT 1 C INDEX ORGT 35 C MOVE *ALL'X' DESCR C ELSE C MOVE CD(INDEX) DESCR C ENDIF * C RETURN /home/smith $ tr [:upper:] [:lower:] >/qsys.lib/mylib.lib/qrpglesrc.file/mypgm2.mbr /home/smith $ cat /qsys.lib/mylib.lib/qrpglesrc.file/mypgm2.mbr d cd s 35 dim(3) ctdata perrcd(1) c *entry plist c parm index 1 0 c parm descr 35 * c index iflt 1 c index orgt 35 c move *all'x' descr c else c move cd(index) descr c endif * c return

Figure 10.6: File redirection can be used with database file members .

If the > operator is used alone (that is, it does not follow a command), the effect is to clear the file. Consider this example:

>upper.txt /home/JSMITH $ cat upper.txt /home/JSMITH $

The redirection operator is used without a command, so it clears upper.txt. This is the Qshell version of CL's Clear Physical File Member command (CLRPFM).

A word of warning is in order:

Do not use the same file in both input and redirection operators.

Here is an example of what you should not do:

# WARNING: Do NOT do the following !!!! tr [:upper:] [:lower:] < myfile >myfile # Do NOT do this !!!! # WARNING: Do NOT do the preceding !!!!

The lowercase version of myfile will not replace the uppercase version. Redirection is a function of Qshell, not of tr . Therefore, the output redirection operator will clear myfile before tr begins to read the data.

The Noclobber Option

By default, the > redirection overwrites the contents of an existing file. In Figure 10.7, output.txt is loaded with a list of the Qshell script files (the files with an extension of .qsh). Then, the file is replaced with a list of the names of .csv files.

ls *.qsh >output.txt /home/JSMITH $ ls *.csv >output.txt /home/JSMITH $ cat output.txt cust.csv two.csv uuu.csv /home/JSMITH $

Figure 10.7: The output of the second ls command replaces , or "clobbers," the output of the first ls command.

To prevent the > redirection operator from overwriting an existing file, use the "noclobber" option, which is turned on using the set -C command. Figure 10.8 illustrates the use of this command to prevent Qshell from overwriting file output.txt. When the script attempts to replace the contents of output.txt with a list of the names of .csv files, Qshell responds with error message 001-0054. The cat command shows that the second list did not change the contents of output.txt.

set -C /home/JSMITH $ ls *.qsh >output.txt /home/JSMITH $ ls output.txt output.txt /home/JSMITH $ ls *.csv >output.txt qsh: 001-0054 The noclobber option is set and file output.txt already exists. /home/JSMITH $ cat output.txt args01.qsh args02.qsh args03.qsh /home/JSMITH $

Figure 10.8: Use the noclobber option to prevent accidental erasure of data.

There are two ways to turn on the noclobber option:

set -C set -o noclobber

To turn off the noclobber option, use a plus sign instead of a minus sign, like this:

set +C set +o noclobber

You can override the noclobber option on a single operation by using the > redirection operator, as shown in Figure 10.9. The set utility turns on the noclobber option. The first ls fails because output.txt already exists and the > redirection operator is used. The second ls command succeeds because the > redirection operator overrides the noclobber option. Notice that the > operator (in the first ls command) generates an error message, but the > (in the second ls command) does not. The cat command shows that the .csv files are listed in file output.txt.

set -C /home/JSMITH $ ls *.csv >output.txt qsh: 001-0054 The noclobber option is set and file output.txt already exists. /home/JSMITH $ ls *.csv >output.txt /home/JSMITH $ cat output.txt cust.csv two.csv uuu.csv /home/JSMITH $

Figure 10.9: The > redirection operator overrides the noclobber option.

Pipes and Pipelines

A pipeline is a series of commands connected with the pipe symbol, the vertical bar (). A pipe channels stdout of one command to stdin of another. Figures 10.10 through 10.12 show pipelines in action. Input comes from the goodoleboys.txt file, which was shown in Figure 10.2.

Figure 10.10 shows the use of grep , a search utility, to look in the goodoleboys.txt file for records that include the string 444-. Instead of writing the selected records to stdout, it sends them into the pipe. The sort routine reads from stdin, so it gets the selected data from the pipe, sorts it, and sends it along to stdout. Since stdout is not piped or redirected from sort , the sorted data appears on the display.

grep '444-' Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Bubba Oct 13 444-1111 Buck Mary Jean 12 Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410

Figure 10.10: Grep is a search utility. It looks for records that match a pattern.

Figure 10.11 uses the grep utility with two pipes. Grep selects the records that contain 444- and sends them to tr , which converts the lowercase letters to uppercase, leaving all other characters as they are, and passes the data along to the sort utility. After sorting, the data is sent to stdout, which is the display in this case because there is no output-redirection operator or pipe following the sort.

grep '444-' ARLIS JUNE 19 444-1314 REDEYE SUZY BETH 12 .75 BILLY BOB JUNE 11 444-4340 LEOTIS LISA SUE 12 BUBBA OCT 13 444-1111 BUCK MARY JEAN 12 CHUCK DEC 25 444-2345 BLUE MARY SUE 12 .50 OTIS SEPT 17 444-8000 OL' SAL SALLY 12 ROSCOE FEB 2 444-2234 ROVER ALICE JEAN 410

Figure 10.11: There are two pipes in this pipeline.

Pipes can also be combined with redirection operators, as Figure 10.12 shows. Grep selects all goodoleboys.txt records that contain 444- and sends them to tr , which converts lowercase letters to uppercase and passes the data along to sort . Sort sends the sorted data to the file sorted.txt. In this case, the sorted records containing 444- are redirected to the file sorted.txt.

> grep '444-' sorted.txt /home/JSMITH $ > cat sorted.txt ARLIS JUNE 19 444-1314 REDEYE SUZY BETH 12 .75 BILLY BOB JUNE 11 444-4340 LEOTIS LISA SUE 12 BUBBA OCT 13 444-1111 BUCK MARY JEAN 12 CHUCK DEC 25 444-2345 BLUE MARY SUE 12 .50 OTIS SEPT 17 444-8000 OL' SAL SALLY 12 ROSCOE FEB 2 444-2234 ROVER ALICE JEAN 410 /home/JSMITH $

Figure 10.12: Pipes can be combined with redirection operators.

Redirection Operators and Pipes

It is common for beginning shell users to confuse redirection operators and pipes. There is, however, a great difference between the two concepts. Consider the following two commands:

myscript.qsh > somefile myscript.qsh somefile

In the first line, the output of myscript.qsh will be stored in somefile . In the second command, the output of myscript.qsh is to be read by a program or script called somefile . Since files in Unix systems may contain any type of data, including programs, using an output redirection operator instead of a pipe before a program or script destroys or corrupts the program or script.

That warning is worth repeating:

Using an output redirection operator instead of a pipe before a program or script destroys or corrupts the program or script.

It will help you to use these properly if you always look at the filename to which stdout is being redirected. If the file is a program or script, use a pipe. If it is a data file, use a redirection operator. It will also help if you think of a pipe operator as a symbol for a temporary file. Imagine that the command preceding the pipe symbol writes to a temporary file, and the command following the pipe symbol reads the temporary file.

Tee

You might decide that you would like to see the data that travels through a pipe, perhaps for debugging purposes. If so, use the tee utility, also known as duplicate standard input . As you might guess, the tee utility gets its name from the "T" shape a plumber might install in a real pipe to branch its flow between two locations. Tee reads stdin and writes the unchanged data to both stdout and one or more other files. The syntax of tee utility is shown here:

tee [ -ai ] [ file ]

By default, tee replaces the data in a file that already exists. To append the data instead, use the a option. Use of tee is shown in Figure 10.13. It passes the output of grep along to tr , and at the same time makes a copy of the output in the file tee.out. Notice that the data is in mixed case in the tee.out file, because it had not been fed to the tr command at that point. Notice also that the data in tee.out is not sorted, because the sort process had not yet run.

grep '444-' > tr [:lower:] [:upper:] sort >sorted.txt /home/JSMITH $ cat tee.out Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Bubba Oct 13 444-1111 Buck Mary Jean 12 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 /home/JSMITH $

Figure 10.13: After the pipeline finishes, the cat command shows the input that grep was given.

In Figure 10.14, grep again selects records that contain the string 444- from file goodoleboys.txt. Then, tee writes the records to two files, tee.out and two.out, and at the same time passes the records to the sort utility.

grep '444-' Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Bubba Oct 13 444-1111 Buck Mary Jean 12 Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 /home/JSMITH $ cat tee.out Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Bubba Oct 13 444-1111 Buck Mary Jean 12 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 /home/JSMITH $ cat two.out Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Bubba Oct 13 444-1111 Buck Mary Jean 12 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 /home/JSMITH $

Figure 10.14: The tee utility writes the records from grep to tee.out and two.out, and passes the records to sort.

Overriding Redirection

You can temporarily override stdout on a line-by-line basis. That is, even though stdout is assigned to a certain file, you can direct the output of individual commands to other files. Consider this example of redirected output:

print "Something" if [ -z "" ] then date >>errorlog.txt print "Script:

print "Something" if [ -z "$1" ] then date >>errorlog.txt print "Script: $0" >> errorlog.txt print "Parm 1 missing, default used." >> errorlog.txt fi print "Something else"

" >> errorlog.txt print "Parm 1 missing, default used." >> errorlog.txt fi print "Something else"

The first and last print commands write to stdout. The output from those commands will go to the display, to a file, or into a pipe, depending on how stdout is directed. Look at the if block, however. Three commands ” date , and two echo commands ”do not write to stdout, but append to errorlog.txt. Figure 10.15 shows what happens when this code is executed. When the user types the name of the script, error01.qsh, on the command line, Qshell runs the script. Since stdout is not redirected, the first and last print commands write the string "Something" and "Something else" to the terminal. The cat utility shows what has been written to file errorlog.txt.

error01.qsh Something Something else /home/JSMITH $ cat errorlog.txt Wed Dec 25 08:27:35 2002 Script: /home/JSMITH/error01.qsh Parm 1 missing, default used. /home/JSMITH $

Figure 10.15: The previous script redirected both the stdout and stderr files.

Here Documents

A "here" document is a special way of capturing input lines and directing them as a group to another process or file. The "here" document operator, <<, is followed by an end-of-data delimiter string of the programmer's choosing.

Figure 10.16 shows how to create a file from the keyboard using a "here" document. The user chooses the string EOD to indicate the end of the input data. The cat utility displays the lines to stdout , which is overridden to file mydata.txt.

cat <mydata.txt > Jack,12 > Bill,20 > Leroy,15 > EOD /home/JSMITH $ cat mydata.txt Jack,12 Bill,20 Leroy,15 /home/JSMITH $

Figure 10.16: This example shows how to use a "here" document to enter data into an IFS file from the keyboard.

You may follow the << operator with a hyphen if you wish. This allows you to enter a tab character before the end-of-data delimiter. Since you can't type a tab character from a 5250 terminal session, you are not likely to use this option unless you edit scripts on an ASCII-based machine.

Filters

A filter is a program that reads from stdin and writes to stdout . Filters are designed with a single purpose in mind, and are written to be flexible. The sort utility, for example, is a filter that resequences, merges, and sequence-checks text files. Because sort is so powerful, it is unlikely that you will need to write another program to sort.

Generally, you should write scripts and programs that embrace the filter philosophy. As a rule, scripts should not converse with the user. For instance, suppose you want the user to enter his or her name when a script begins to run. You could use print or echo to display a prompt, and use read to accept the user's input, like this:

echo "Please enter your name:" read name

However, this degree of interactivity would make it difficult or impossible to use the script in a pipeline. It is better to provide a way to specify a command line option. In the following example, the user name is entered through a parameter:

myscript.qsh -n 'Joe Smith'

As you can see, the use of switches can replace interactive I/O. In this case, the programmer writes the script so that it interprets the parameter following the n option as a user's name. This is demonstrated in the following example:

if [ "" ] && [ "" ] && [ = "-n" ] then name= else name=$LOGNAME fi

If the first and second positional parameters are defined and the first parameter is -n , the second parameter provides the value for the variable name; otherwise , name gets its value from the Qshell predefined variable LOGNAME, which is the ID of the user who is running the shell. There is no need to use echo and read statements to include the user's name.

I O Utilities

To write effective Qshell scripts, you must learn to make Qshell scripts read data from and write data to files. The Qshell I/O utilities, listed here, are streamoriented and easy to use:

• Dspmsg
• Echo
• Print
• Printf
• Read

Print

While there is only one input utility, there are several Qshell output utilities. The print utility, which writes to output files, is probably the one you'll need most often. Here is its syntax:

print [ -nrR ] [ -u [ n ] ] [ argument ]

Print writes zero or more arguments to stdout . Each argument is separated by a space. Unless you include the -n option, print ends with a newline character, which forces the next output into another record. Print lets you embed control sequences (listed in Table 10.2) in output.

If you don't want print to interpret the control sequences, use option -r or -R . Figure 10.17 shows a simple example of the print utility.

print "Tom 14 Bobby 21 Jack 15" Tom 14 Bobby 21 Jack 15

Figure 10.17: The and control characters insert tabs and newline characters into the output stream.

To include a backslash in a string, you normally code two backslashes, although one backslash is often sufficient. Single backslashes are acceptable in print commands if they are used before characters that are not control characters. Here, single backslashes are used without problem:

print "Look in directory homemystuffzipfiles." Look in directory homemystuffzipfiles.

In the commands in Figure 10.18, backslashes must be doubled so control sequences will not be interpreted. With only one backslash after the colon , Qshell interprets the sequence as a tab. Doubling the backslash doesn't solve the problem, although it might seem that it should. It is only by coding three backslashes that a single backslash is printed to stdout.

print "Look in directory c: empmystuffzipfiles." Look in directory c: empmystuffzipfiles. /home/JSMITH $ print "Look in directory c: empmystuffzipfiles." Look in directory c: empmystuffzipfiles. /home/JSMITH $ print "Look in directory c:\tempmystuffzipfiles." Look in directory c: empmystuffzipfiles. /home/JSMITH $

Figure 10.18: If the character following the backslash is a control character, you will have to code three backslashes to print one.

Printf

The printf utility is based on C's printf function. It differs from the print utility in several ways:

• Printf provides a way to format the way variables are printed.
• Printf does not automatically generate an end-of-line sequence when it writes. If you want to advance to a new line, you must include a control sequence in the format string.
• Printf does not support the print options. For instance, you cannot use the -u option to direct print to a certain file; you have to use a file descriptor with redirection operators.

The syntax of printf is as follows :

printf format [ argument ]

You must provide at least one argument ”a format string ”to printf . The format string may contain plain text, control characters, and special formatting sequences.

Printf supports the control sequences that print supports. That is, you can use to tab,  to backspace , and so on. Again, the printf utility does not issue a linefeed after printing unless there is a sequence in the format string.

Arguments are matched with formatting sequences from left to right. If there are fewer arguments than formatting sequences, printf reuses formatting sequences as often as necessary. In Figure 10.19, for example, there are two conversion characters in the format string, but four arguments following the conversion string. Printf uses the format string twice.

a=12 /home/JSMITH $ b=28 /home/JSMITH $ c=25 /home/JSMITH $ d=7 /home/JSMITH $ printf "%d %e " $a $b $c $d 12 2.800000e+01 25 7.000000e+00 /home/JSMITH $

Figure 10.19: Printf reuses the format string as necessary.

The format of a conversion-formatting sequence defines how a value appears in output. Its syntax is as follows:

%[flags][width][.precision]conversion

Since the percent sign indicates the beginning of a format sequence, you must code two percent signs in order to print one. In addition to the percent sign, you must include a conversion character to tell printf how to format an argument. The acceptable conversion characters are listed in Table 10.3. For example, %d means that a number is to be printed in decimal format, while %o means a number is to be printed in octal format.

You may use the optional portions of the formatting sequence ”flags, width, and precision ”to further modify the appearance of the output. The five flags are used only with numeric values. You may use more than one of them in a format sequence. The flags and their meanings are listed in Table 10.4.

The width is the minimum number of characters to be sent to output. If you code an asterisk (*), the width is taken from the next argument. The function of the precision depends on the conversion character, as shown in Table 10.5.

The next several pages provide examples of the printf utility. In the following lines, the variable price has the value 19250.2:

print $price 19250.2 /home/JSMITH $ printf "$%-.2f " $price 250.20

The print utility displays this value with the minimum number of needed digits. The printf utility shows two decimal digits because of the precision entry. Both utilities left-adjust the value. Print does so by default. Printf left- adjusts because of the minus sign.

In the following example, printf right-aligns the value of the variable within a field width of 14 digits:

print $price 19250.2 printf "$%14.2f " $price $ 19250.20

Because of the precision, printf displays two digits after the decimal point.

In the case of integer values, the precision value specifies the minimum number of digits to be displayed, as shown here:

print $count 4 /home/JSMITH $ printf "%.4d " $count 0004

In this case, the minimum number of digits to display is four.

The asterisk character (*) tells printf that the field width is specified in the next argument. In the following example, the field width is four and the value to be printed is seven:

printf "%0*d " 4 7 0007

Here, printf right-aligns the value of variable age within a width of 15 characters:

print $age 45 /home/JSMITH $ printf "%15i " $age 45

When combined with command substitution, printf can place edited values into Qshell variables. Command substitution is a technique whereby the output of a command is substituted for the command itself. To tell Qshell to use command substitution, enclose the command within parentheses and precede the open parenthesis with a dollar sign:

amount=$(printf "$%-12.4f" $price) /home/JSMITH $ echo "/$amount/" /250.2000 /

The f in the format string means that the value is to be printed as a floatingpoint (real) number. The 12 refers to the overall length of the number, including sign and decimal positions . The 4 means that four decimal positions are to be shown. The hyphen means that the number is left-adjusted within the 12 positions. When the dollar sign, which is not part of the formatting sequence, is placed in front of the number, the total length is 13.

Echo

The easiest to use (and least powerful) of the output utilities is echo , which sends one or more arguments, followed by an end-of-line sequence, to stdout. The word echo can be followed by one or more arguments separated by white space. An argument can be a quoted string, an unquoted string, or a variable. In Figure 10.20, the lang variable is interpreted unless surrounded by single quotes.

lang=RPG /home/JSMITH $ echo $lang rulz! RPG rulz! /home/JSMITH $ echo "$lang rulz!" RPG rulz! /home/JSMITH $ echo '$lang rulz!' $lang rulz! /home/JSMITH $

Figure 10.20: Echo is easy to use, but not as powerful as print.

Print is superior to echo in two ways:

• Print allows options.
• Print can interpret control commands within the arguments it sends to output files.

Dspmsg

The Display Message ( dspmsg ) utility is a sort of soft-coded printf command. This feature was designed to make it easier to use scripts with different national languages. Instead of hard-coding messages in your scripts, you store messages in a message catalog and use dspmsg to retrieve them. If you decide to run your scripts in an environment where people use another language, you can translate the messages without having to modify the scripts.

You'll need a message catalog to hold the messages. Creating a message file is not unlike traditional iSeries programming. Enter the source-code directives into a member of a source physical file member and compile it. Table 10.6 explains the directives.

The source member follows these rules:

• Blank lines are ignored.
• Comments begin with a dollar sign and space.
• Directives begin with a dollar sign that is not followed by a blank.
• Message text begins with a number.

The example source member in Figure 10.21 is shown in a browse session of SEU. The first line is a comment because it begins with a dollar sign and a space. The second line says that the double quote is used as a quotation character to delimit the message text. The third line indicates that the following messages are in set 1. Lines 4.00 through 13.00 define the messages of set 1. The remainder of the source member defines message set 2.

Columns . . . : 1 71 Browse JSMITHS/SRC SEU==> MYMSGCAT *************** Beginning of data ********************** 0001.00 $ My message catalog 0002.00 $quote " 0003.00 $set 1 0004.00 1 "File %s not found. " 0005.00 2 "Directory %s not found. " 0006.00 3 "File %s is a directory. " 0007.00 4 "File %s is not a directory. " 0008.00 8 "File name was not specified. " 0009.00 11 "File name %1$s cannot be created in directory %2$s 0010.00 because %2$s does not exist. " 0011.00 999 "Unexpected error in script %s 0012.00 User: %s 0013.00 Time: %s " 0014.00 $set 2 0015.00 1 "Directory name was not specified. Enter it now." 0016.00 2 "File %s was not found. " 0017.00 21 "The number of files cannot be more than %ld. " ****************** End of data *************************

Figure 10.21: Use SEU or another editor to enter message text.

You may continue a message by ending all but the last line with a backslash. Messages 11 and 999 of set 1 in Figure 10.21 are continued messages. Message 11 spans two lines of source code; message 999 spans three.

Notice the %s and %ld sequences. These are placeholders for arguments. Qshell will replace these strings with the arguments you specify when you display a message. If you want a specific argument, follow the percent sign with the ordinal number of the argument and a dollar sign. For instance, %3$s on line 9.00 means that the third argument is to be displayed as a string. If you omit the ordinal number and dollar sign, Qshell fills in the arguments in the order you specify them. Table 10.7 provides a quick reference of the placeholders.

You may also include certain control sequences like those used with print and printf . The control sequences are listed in Table 10.8. You should end all messages with a sequence. If you do not, Qshell will not advance to a new line of output after displaying the message.

To create or update the message catalog from a source physical file member, use either CL's Generate Message Catalog command (GENCAT) or the Merge Message Catalog command (MRGMSGCLG). The two commands are identical in function. The following command creates a message catalog called mymsgcat in directory /home/jsmith from the source in member MYMSGCAT in MYLIB/MYSRC:

GENCAT + CLGFILE('/home/jsmith/mymsgcat') + SRCFILE('/qsys.lib/mylib.lib/mysrc.file/mymsgcat.mbr')

Once the message catalog has been created, you can use Qshell's dspmsg utility to write to stdout. Here is the syntax of the dspmsg utility:

dspmsg [ -n ] [ -s set ] catalog msgid [ defaultMsg [ arguments ] ]

The - n option prevents Qshell from interpreting the placeholder sequences. The - s option tells which message set contains the message to be displayed. The default value is one. Neither option is required. The catalog and msgid parameters, however, are required. The catalog name is the one you specified in the CLGFILE parameter of GENCAT or MRGMSGCLG. The message ID comes from the message catalog source code.

The default message is the string to be displayed if the message does not exist. If you do not wish to specify a default message, but do want to specify arguments, use a dummy default message. The arguments to be passed to the message, if there are any, are specified last.

Here is a basic example of dspmsg :

dspmsg mymsgcat 8 File name was not specified.

This command displays message 8 of set 1. Since the -s option is not used, set 1 is assumed.

The following example displays message 8 of set 1, if it exists. If it does not exist, the string Message not found is displayed:

dspmsg mymsgcat 8 'Message not found' File name was not specified. /home/JSMITH $

The following example displays message 45 of set 1, if it exists. If it does not exist, it displays Message not found and advances to a new line because of the option:

dspmsg mymsgcat 45 'Message not found ' Message not found /home/JSMITH $

The next several examples involve placeholders. This command replaces the %s placeholder with the string mydata.txt :

dspmsg -s 1 mymsgcat 1 'Message not found ' mydata.txt File mydata.txt not found.

This command replaces the %s placeholders with the name of the script, the name of the user, and the current date:

dspmsg -s 1 mymsgcat 999 'Message not found '

dspmsg -s 1 mymsgcat 999 'Message not found ' $0 $LOGNAME "$(date)" Unexpected error in script /home/jsmith/myscript.qsh User: JSMITH Time: Wed Dec 25 11:32:23 2002

$LOGNAME "$(date)" Unexpected error in script /home/jsmith/myscript.qsh User: JSMITH Time: Wed Dec 25 11:32:23 2002

To substitute a particular value for the placeholder, use a command like this:

dspmsg -s 2 mymsgcat 21 'Message not found ' 12 The number of files cannot be more than 12.

Message 21 of set 2 is displayed, with the %ld placeholder replaced by the value 12.

Finally, this example displays message 11 of set 1, replacing the %1$s placeholder with the value myfile.text and both instances of the %2$s placeholder with the value mydir:

dspmsg -s 1 mymsgcat 11 'Message not found ' myfile.txt mydir File name myfile.txt cannot be created in directory mydir because mydir does not exist.

Read

The read utility reads a line of data from a file and assigns the data to one or more variables. By default, read accepts data from the keyboard.

The read utility is Qshell's only input utility. The syntax of read is as follows:

read [ -r ] [ -p prompt ] [ -u [ n ] ] [ name ]

The three options are listed and briefly explained in Table 10.9.

Shell scripts most often read text data, not binary data. That is, Qshell reads everything up to an end-of-line character (usually a linefeed character, but sometimes a carriage return and linefeed sequence) as a record. If a line ends with a backslash () followed immediately by the end-of-line sequence, Qshell assumes that the input data is continued on the next record of input. However, if you use the -r option, the backslash is not treated as a continuation character.

For example, the following file contains four records, but Qshell will read them as three records if the -r option is not specified:

This is one record. This is the second record. This is the third record.

The read utility places the retrieved data into one or more variables. Do not precede the variable names with a dollar sign! If you do not list at least one variable, read places the data into variable REPLY.

The read utility separates the contents of the record into fields. By default, fields are separated by white space ”one or more blanks, tabs, or newline characters. Read fills in the variables with the field values until it runs out of data or variables. If there are more data values than variables, all remaining data values are placed in the last variable.

The script fragment in Figure 10.22 reads text records, breaking the fields on white space.

while read name birthmonth birthday phone dog wife rest do print "Name: " $name print "Dog: " $dog print "Born: " print "Month: " $birthmonth print "Day: " $birthday print "Phone: " $phone print "Wife: " $wife print "Rest: " $rest done

Figure 10.22: By default, the read utility assumes fields are separated by white space.

Figure 10.23 lists a line of input that could be used for this script, and what the output would look like. Notice that the birth month and day values must be read into two separate values, because they are separated with white space. Notice also that the wife's second name, Sue , is placed in the rest variable, not included in the wife variable as it should be. The rest variable serves as a catch-all for the last three values.

Input: Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Output: Name: Chuck Dog: Blue Born: Month: Dec Day: 25 Phone: 444-2345 Wife: Mary Rest: Sue 12 .50

Figure 10.23: The read utility places remaining fields into the last input variable.

Since values often include spaces, Qshell provides a way to use some other character as a field separator. The field separator is a predefined Qshell variable called IFS , for Internal Field Separators. Do not confuse this with the Integrated File System, which is also often called the IFS. When you see the term IFS in this book or other Qshell literature, you should be able to tell from context which one the author is referring to. To try to minimize the confusion, this book refers to the Integrated File System as the IFS and to the Internal Field Separators variable as the IFS variable .

By default, the IFS variable has a value of a single binary zero. This special value tells Qshell that fields are to be separated by white-space characters. To change the value of the IFS variable to another character, such as a comma, use the Qshell assignment operator, as with any other variable:

IFS=','

Since the input data is now separated by commas rather than white space, the previous script must be modified, as shown in Figure 10.24.

while read name birthday phone dog wife rest do print "Name: " $name print "Dog: " $dog print "Born: " $birthday print "Phone: " $phone print "Wife: " $wife print "Rest: " $rest done

Figure 10.24: The revised script is written with the assumption that input is comma-delimited.

Notice that the birth date is now one field, not two. The revised input and corresponding output are shown in Figure 10.25. Now, the birth date and wife's name are correctly interpreted during the read.

Input: Chuck,Dec 25,444-2345,Blue,Mary Sue,12,.50 Output: Name: Chuck Dog: Blue Born: Dec 25 Phone: 444-2345 Wife: Mary Sue Rest: 12 .50

Figure 10.25: Here is the comma-delimited input and the corresponding output.

Finally, the read utility's -p option allows a prompt string to be displayed on standard error:

read -p "Please enter your name:" name

You will probably not find much use for this option, because the prompt only displays when the script is run with the source (dot) utility, and the source utility is not the way you will usually want to run Qshell scripts.

File Descriptors

A file descriptor is a single-digit integer that is assigned to an open file. Input and output operations always reference the file descriptor, not the actual file name . Qshell uses file descriptors 0, 1, and 2 for stdin, stdout , and stderr respectively. All three of these file descriptors are assigned to the terminal unless redirected to other files.

Taking explicit control of the file descriptors in a shell script is a rather infrequently used and somewhat complex feature. You might wonder why you would use this feature.

You might need to maintain access to many open files for input and output at the same time. You might be trying to retrofit existing programs or tools that use file or socket descriptors. You might be trying to swap devices so that an application's input and output descriptors match some reasonably complex requirements.

In some cases, the simpler solution of common redirection will serve you well. In other cases, you might need to take direct control of the file descriptors that Qshell maintains and uses for the utilities it invokes.

Redirection Operators

File descriptors 3 through 9 are available for you to use as you wish. To use these file descriptors, you must use the redirection operators listed in Table 10.10. The letters m and n in the table represent file-descriptor numbers .

At least some of these should look familiar to you because Table 10.10 is a more complete version of Table 10.1. The file descriptor preceding the redirection operator is optional. If you do not code a file descriptor, Qshell assumes file-descriptor 0 for input operations and file-descriptor 1 for output operations.

Therefore, the following two tr commands are equivalent:

tr [:lower:] [:upper:] upper.txt tr [:lower:] [:upper:] 0upper.txt

The Ulimit Utility

One of the features added to V5R2 Qshell was the ulimit utility, which displays or sets system resources. The n option allows you to display or set the maximum number of file descriptors that a process can open, as in the following example:

ulimit -n 200 /home/jsmith $

Opening and Closing File Descriptors

To open and close files under file descriptors for the current Qshell session or running script, use the exec utility with the open and close redirection operators. Here are some examples:

• Open file goodoleboys.txt for input as file descriptor 3:

  exec 3

• Open a file as input under file descriptor 9 that is named in positional parameter 1:

  exec 9<

• Open stdin (file descriptor 0) to file temp.txt:

  exec < temp.txt

• Close stdin (file descriptor 0):

  exec <&-

• Close stdout (file descriptor 1):

  exec >&-

• Close the file opened under file descriptor 3:

  exec 3<&-

• Open file descriptor 7, assigning it to stdout:

  exec 7>&1

  Notice that an ampersand, not a dollar sign, precedes the numeral 1. Both file descriptors 1 and 7 point to stdout. The two output streams are merged.

• Assign stderr to wherever stdout is assigned. The two output streams are merged:

  exec 2>&1

You can use the test utility to determine whether or not a file descriptor is open and associated with a terminal, as the following example shows:

if [ -t 0 ] then print "Enter name,age,phone." fi

If stdin is assigned to the keyboard, Qshell displays the message "Enter name, age, phone." However, if stdin is redirected to another file or if input is coming from a pipe, Qshell does not output the prompting message.

Input through File Descriptors

To read from a file, use the read utility with the -u option. Follow the -u with the number of the file descriptor from which to read. You can separate the option and the file descriptor number, but you don't have to, so the following two commands are equivalent:

read -u 5 line read -u 5 line

Output through File Descriptors

There are two ways to write to a file descriptor: an easy way and an easier way. You need to know both methods because the easy way works in all cases, but the easier way only works with the print utility.

The easier way is to specify the file descriptor number in the -u (unit) option of the print utility:

print -u 3 $name

The easy method is to use a duplication redirection operator (>) with any Qshell command that writes to stdout. Follow the > operator with an ampersand and a file descriptor number. Think of the ampersand as meaning "whatever is assigned to." So, you might read the string >&3 as "write to whatever is assigned to file descriptor 3." The > operator usually clears the file, but for file descriptors 3 and above, Qshell appends the data instead.

Since both methods work for print , the following two statements do the same thing:

print -u 7 $name print $name >&7

Each command writes the value in the variable name to the file opened under file descriptor 7.

In Figure 10.26, the first exec opens member TEMP in source physical file JSMITHS /QRPGSRC for output with file descriptor 3. The two echo commands write to the source member. The second exec closes the file. The cat command shows that the two lines are in the source physical file member, as they should be.

exec 3>/qsys.lib/jsmith.lib/qrpgsrc.file/temp.mbr echo "line 1" >&3 ecbo "line 2" >&3 exec 3>&- cat /qsys.lib/jsmith.lib/qrpgsrc.file/temp.mbr line 1 line 2 /home/JSMITH $

Figure 10.26: Qshell output is directed to a source physical file member via a file descriptor.

In the following example, if the first parameter is missing, a message is sent to stderr in order to demonstrate proper usage for the script:

if [ -z ] then print -u2 "Usage: $(basename

if [ -z $1 ] then print -u2 "Usage: $(basename $0) file ..." exit 1 fi

) file ..." exit 1 fi

Without the redirection operator, the print statement would send the output to stdout.

Figure 10.27 builds an FTP script that will be used for logging into an FTP server and exchanging files. First, it reads file tkmjzzypokrtmmoos to get the user ID and file jnnoiiesokjlxrros to get the password. It uses file descriptor 3 for both files, as well as the ftpmodel file, which contains the miscellaneous FTP commands. When it encounters the special commands *get and *put in the model file, the script reads the names of files to get and put from two other files, ftpgetlist.txt and ftpputlist.txt. All output is sent to stdout.

#!/bin/qsh # get user id from text file tkmjzzypokrtmmoos exec 3

Figure 10.27: This script builds an FTP script from five IFS files.

Figure 10.28 shows the input files and the output produced by the script in Figure 10.27.

Input: tkmjzzypokrtmmoos myid Input: jnnoiiesokjlxrros mypass Input: ftpmodel.txt namefmt 1 lcd /home/mydir cd /home/yourdir *put *get quit Input: ftpgetlist.txt inputa inputb Input: ftpputlist.txt data1 data2 data3 data4 data5 Output: myid mypass namefmt 1 lcd /home/mydir cd /home/yourdir put data1 put data2 put data3 put data4 put data5 get inputa get inputb quit

Figure 10.28: Here is the input and output of the ftpbuild.qsh script.

Redirection and Compound Statements

Qshell permits redirection with scripts, compound statements, and sub- shells , as well as with individual statements. Using redirection a single time can improve the performance of multiple statement operations by avoiding additional Qshell file operations.

The following two examples of using while loops with redirection operators are functionally equivalent.

while read line do echo $line >> temp.txt done > temp.txt

The content of file goodoleboys.txt is appended to the output file temp.txt one line at a time. The first example redirects the output of the echo command, while the second redirects the output of the entire loop. The example that does the redirection of the entire loop is much faster because it only opens the output file once. The other example causes Qshell to process ten file-open requests .

You may direct input and/or output by placing the redirection after the done end-of-loop delimiter or at the end of any other multi-statement block.

Figure 10.29 shows the use of redirection operators with for loops. The first loop numbers and lists the positional parameters. The second loop is identical to the first, except that the output of the loop is piped into the sort utility.

pos=0 for i do let pos=pos+1 echo $i ($pos) done mydata.txt (1) yourdata.txt (2) hisdata.txt (3) herdata.txt (4) ourdata.txt (5) theirdata.txt (6) pos=0 for i do let pos=pos+1 echo $i ($pos) done sort herdata.txt (4) hisdata.txt (3) mydata.txt (1) ourdata.txt (5) theirdata.txt (6) yourdata.txt (2)

Figure 10.29: Qshell permits redirection of the output produced within any loop.

The example in Figure 10.30 is identical to Figure 10.27, except that file descriptors are not used. Instead, stdin and stdout are redirected with each command, including the loops.

#!/bin/qsh # get user id from text file tkmjzzypokrtmmoos read id

Figure 10.30: Like the script in Figure 10.27, this script produces an FTP script. However, this script uses redirection of loops rather than file descriptors.

Summary

Qshell is well suited to processing stream files. The primary stream files Qshell uses are standard input, standard output, and standard error. These three files may be redirected to disk files, or they may be piped to other commands. Piping and redirection are the result of programs called filters , which make it possible to develop complex applications by joining specialized programs and scripts in sequence.

Qshell includes five utilities that are designed especially for input and output operations. The use of file descriptors enables Qshell to work with more than three files at one time. Qshell also provides an easy way to redirect the output of a loop.

Chapter 11 Command Line Arguments