The Perl Utility

Overview

As mentioned in the previous chapter, the perl utility provided by the iSeries Tools for Developers product is a PASE utility. Qshell provides direct support to run PASE utilities without starting a PASE environment or leaving the Qshell environment.

Perl is a truly popular piece of scripting/programming software (covered by the GNU General Public License) that can be used to do just about anything. Larry Wall originally released perl in 1987 as a Unix system-administration tool. It has now reached version 5.8, and runs on dozens of platforms.

Some of perl's many strengths include the following:

• Broad standard library and third-party libraries
• Powerful string-processing operators
• Automatic memory management
• An eclectic syntax, which provides perl programmers many ways to accomplish the same task

Many books, tutorials, and Web resources are available for learning perl. Start at the perl home page, www.perl.com, and the Comprehensive Perl Archive Network, www.cpan.org, for downloads of perl distributions and modules to do just about anything you could think of.

Perl has also been ported to iSeries as an ILE application (i.e., outside of PASE), but the PASE application is much newer , and the ILE version appears to be less frequently used. The perl utility shipped with iSeries Tools for Developers is version 5.005_03 for AIX. The current version available on CPAN in the OS/400 section is "5.8.0@18380."

Running Perl

If you examine the series of directories that were added to the PATH variable in Figure 21.3 of the previous chapter, you will see the following statement:

# Add the PASE binary locations to the path export PATH="$PATH:/QIBM/ProdData/DeveloperTools/pase/bin"

This statement sets the path to enable access to Tools for Developers PASE utilities, including perl, in a Qshell resource file. After you have run it to update your path, you can run the perl utility.

Alternatively, you can allow all users on the system to access perl from a standard location by adding a link to perl in the /usr/bin directory. Figure 22.1 shows how to create the link and start perl. The v option shows the version of perl.

ln -s /QIBM/ProdData/DeveloperTools/pase/bin/perl /usr/bin/perl perl -v This is perl, version 5.005_03 built for aix Copyright 1987-1999, Larry Wall Perl may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the Perl 5.0 source kit. Complete documentation for Perl, including FAQ lists, should be found on this system using 'man perl' or 'perldoc perl', If you have access to the Internet, point your browser at http://www.perl.com/, the Perl Home Page.

Figure 22.1: Create a symbolic link to allow all users on the system access to perl from the standard location at /usr/bin/perl.

Perl is an ASCII application. Use ez to edit perl scripts in ASCII and to edit perl data files. See chapter 21 for other strategies for dealing with ASCII files in Qshell.

The Perl Command

The syntax of the perl command has two forms:

perl [options] [--] ProgramFile arguments ... perl [options] [-e commands ] [-] arguments ...

You will use one or the other of these forms, but not both, in one command. You can write perl script (program code) either from the command line or in a text program file. Alternatively, write the perl program as a series of script files using the magic number (#!) syntax. Examples in this chapter use perl scripts.

Perl Options

Table 22.1 shows some of the more commonly used command-line options for perl.

Perl Syntax

Perl is a large and complex language, so this chapter only presents highlights of it. The tables describing the language in this chapter are not complete. Instead, they focus on the most frequently used perl constructs.

Variables

All variables in perl are prefixed with a special character, depending on the treatment of the variable. Variables prefixed with a dollar sign are scalar variables (numbers or strings). Variables prefixed with an "at" sign are array values. In general, arrays are indexed by numbers starting with zero, but variables prefixed with a percent sign are associative arrays, which are indexed by strings. When indexing into an array, indicate you want the result to be a scalar value by using the dollar-sign prefix.

Like Qshell, perl uses predefined special variables to represent common information. Table 22.2 shows some of the more commonly used perl special variables and arrays.

Figure 22.2 shows usage patterns of perl variables, demonstrated by a simple lookup facility. Comments in the perl script explain details about individual steps.

cat lookupExample.pl #!/usr/bin/perl -w # These are simple variable assignments and substitutions. $firstName = "Billy"; $lastName = "Bob";x $number = "444-4340"; $fullNameAndNumber = "$firstName $lastName" . "at" . $number; print "Name and number: $fullNameAndNumber "; # Simple array assignment. Indexes 0 to 4 @names = ("Billy Bob", "Chuck", "Bubba", "Amos", "Claude"); for ($i=0; $i<$#names; $i++) { print "Name #$i: $names[$i] "; } # Simple associative array of phone numbers Indexed by name. %phoneNumbersByName = ("Billy Bob", "444-4340", "Chuck", "444-2345", "Bubba", "444-1111", "Amos", "333-1119", "Claude", "333-4340"); # List all of the keys (indexes) of the associative array. @keys = keys(%phoneNumbersByName); # About to output an array as a string, set the output field # separator to a comma plus a space. $" = ","; print "Indexes of phone list: @keys "; # Retrieve a phone number $target = "Billy Bob"; if (defined $phoneNumbersByName{$target}) { $lookup = $phoneNumbersByName{'Billy Bob'}; } else { $lookup = "Not found"; } print "Lookup of phone number for $target is $lookup "; lookupExample.pl Name and number: Billy Bob at 444-4340 Name #0: Billy Bob Name #1: Chuck Name #2: Bubba Name #3: Amos Indexes of phone list: Amos, Bubba, Claude, Chuck, Billy Bob Lookup of phone number for Billy Bob is 444-4340

Figure 22.2: This example perl code (plus comments) declares and uses perl variables to demonstrate a simple lookup capability using an associative array.

Operators

In addition to the traditional mathematical operators found in many languages, perl has operators for the manipulation, comparison, and construction of strings. Table 22.3 shows the more commonly used operators.

Functions

Perl provides a standard set of built-in functions for string and array manipulation. Perl also provides a large set of functions that provide interaction with and access to operating-system services. Table 22.4 shows some commonly used functions.

Use flags to modify the match and substitution functions:

• The m flag enables a multiline match. In a multiline match, the newline character is not a special character. Match it with the character.
• The g flag matches as many times as possible (globally).
• The i flag matches in a case-insensitive fashion.
• The o flag expands variables only once.

In addition to the flags supported by the match function, the substitution function accepts an additional flag, e , to interpret the replacement string as an expression.

You can also use flags to modify the translation function:

• The c flag complements the search list.
• The d flag deletes all characters not found in the search list.
• The s flag "squeezes" all sequences of characters in the search list, replacing them with the single target character.

Regular Expressions

The perl language supports an extensive regular-expression language for specifying patterns. All characters match themselves except for the special characters explained in Table 22.5. Group a series of expressions into a single expression.

Perl Examples

The first examples in this section are constructed simply, while subsequent examples use more complex constructs. Examples for phone-number lookup, PTF status, and database access are repeated in C, Java, and perl, for comparison purposes. The data for all of the exercises is the goodoleboys.txt file, which was shown in the previous chapter, in Figure 21.1.

Figure 22.3 demonstrates how to use a command in the perl language on the command line, with the e option. The match operator implicitly matches the current input line. The n option implicitly loops the script around all standard input.

perl -wne 'if (m/444/) { print "$_"; } ' < goodoleboys.txt Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 Bubba Oct 13 444-1111 Buck Mary Jean 12 Billy Bob June 11 444-4340 Leotis Lisa Sue 12 Otis Sept 17 444-8000 Ol' Sal Sally 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 Arlis June 19 444-1314 Redeye Suzy Beth 12 .75

Figure 22.3: This example searches for the string 444 in the input. Use the -e option to write perl program code on the command line. The -n option loops the program over all lines in the input.

Where Figure 22.3 uses the -n operator, Figure 22.4 uses the <> operator. This example explicitly iterates through all lines of standard input. The <> operator sets the $_ variable to the input line, and returns true if there are still lines remaining in the input. The chomp function removes the newline from the input stream, requiring you to add it again if you print the $_ variable.

cat showLocalBoys.pl #!/usr/bin/perl -w while (<>) { chomp; if (m/444-/) { print "line $.: $_ "; } } showLocalBoys.pl < goodoleboys.txt line 3: Chuck Dec 25 444-2345 Blue Mary Sue 12 .50 line 4: Bubba Oct 13 444-1111 Buck Mary Jean 12 line 5: Billy Bob June 11 444-4340 Leotis Lisa Sue 12 line 7: Otis Sept 17 444-8000 Ol' Sal Sally 12 line 9: Roscoe Feb 2 444-2234 Rover Alice Jean 410 line 10: Arlis June 19 444-1314 Redeye Suzy Beth 12 .75

Figure 22.4: Use Qshell to run this perl script, which shows all records in the input file that match a local dialing prefix.

In Figure 22.5, the match function tells perl to search for strings that have the format of a telephone number. For a more explicit match, the match expression includes word boundaries ( b ) around the telephone number. The match specifically matches three occurrences of the 4 character, a dash, and four occurrences of any numeric character ( d ).

cat showLocalBoyDialList.pl #!/usr/bin/perl -nw chomp; if (m/(4{3}-d{4})/) { $name = substr($_, 0, 10); $phone = ; print "$name $phone "; } showLocalBoyDialList.pl goodoleboys.txt Chuck 444-2345 Bubba 444-1111 Billy Bob 444-4340 Otis 444-8000 Roscoe 444-2234 Arlis 444-1314

Figure 22.5: This perl example uses a complex regular expression to match a phone-number format. It uses the $1 variable to retrieve the previously matched sub-expression group from the match function.

Enclosing the entire phone number inside parentheses creates a sub-expression. The special variable $1 holds the value of the first sub-expression after a match. If the match function returns true, the substr() function removes the first 10 characters and assigns them to the name variable. The script then prints out the name of the phone number's owner and uses the special $1 variable to print out the number.

In Figure 22.6, telephone numbers that end in 2300 through 9999 in local exchange 444 are updated to be part of a new local exchange, 445.

cat newLocalPrefix.pl #!/usr/bin/perl -nwei.backup chomp; if (m/(4{3}-(d{4}))/) { $last4phone = ; if ( >= 2300) { s/(4{3}-(d{4}))/445-$last4phone/; print STDERR "Updating 444-$last4phone "; } } print "$_ "; newLocalPrefix.pl goodoleboys.txt Updating 444-2345 Updating 444-4340 Updating 444-8000 ls good* goodoleboys.txt goodoleboys.txt.backup cat goodoleboys.txt Name Born Phone Dog Wife Shotgun Paid ========= ======== ======== ======== ========= ======= ===== Chuck Dec 25 445-2345 Blue Mary Sue 12 .50 Bubba Oct 13 444-1111 Buck Mary Jean 12 Billy Bob June 11 445-4340 Leotis Lisa Sue 12 Amos Jan 4 333-1119 Amos Abigail 20 Otis Sept 17 445-8000 Ol' Sal Sally 12 Claude May 31 333-4340 Blue Etheline 12 Roscoe Feb 2 444-2234 Rover Alice Jean 410 Arlis June 19 444-1314 Redeye Suzy Beth 12 .75 Junior April 30 BR-549 Percival Lilly Faye 12 Bill Feb 29 333-4444 Daisy Daisy 20 Ernest T. ?? none none none none

Figure 22.6: This perl script updates a phone list using the -i option to perform an in-place edit with a backup.

Always use a backup file extension with the i parameter. It is very easy to make mistakes, and if you tell it to, perl with gladly throw away your original data and leave you with the corrupted data remnants resulting from a buggy script. The i parameter is used to update the input files in place. Backup files are created with a .backup extension. Perl uses all text output to the stdout file handle as replacement for the lines in the original file. Text output to the stderr file handle is not assumed to be part of the edited file.

This example demonstrates the ability to interchange strings and numeric values. The special variable $2 receives the four final digits of the telephone number. That value is used in a numeric comparison with 2300. If the value is greater than or equal to 2300, the substitution function changes the exchange to 445. The script prints a status message to stderr.

Building on the previous examples, Figure 22.7 uses the data file from Figure 22.6 to demonstrate a functioning, interactive phone-number lookup application. The application uses two associative arrays to provide lookup services based on the data file. Compare and contrast this perl application to the similar Java example in Figure 23.6 and C example in Figure 24.5. From a perl perspective, the associative array provides exactly the right functionality for a lookup program of this type. The robust and flexible regular expressions that perl provides are only used slightly in this example because the data file is composed of fixed-length fields. If the fields were delimited or needed some additional processing, the regular expression capabilities would make perl a much more attractive solution for this program.

cat phoneList.pl #!/usr/bin/perl -w if ($#ARGV < 0 $ARGV[0] eq '-h') { print "Usage: phoneList [-h] ... "; exit(1); } # An associative array that allows lookup of the phone number # by the good ole boy's name %numberByOleBoy = (); # An associative array that allows lookup of the good ole boy # by his wife's name. %oleBoyByWife = (); # Read input from all the data files passed on the command line while () { chomp; if ($. == 1 $. == 2) { # The first 2 lines of each data file represent # header lines. next; # Go to the next input line. } # For each field: # - Take the characters from the correct column for the field # into a variable. # - Remove any trailing spaces at the end of the variable $oleboy = substr($_, 0, 10); $oleboy =~ s/s+$//g; $oleboyswife = substr($_, 37, 10); $oleboyswife =~ s/s+//g; $phonenumber = substr($_, 19, 8); $phonenumber =~ s/s+//g; # Build an associative array so that we can lookup phone # number based on the ole boy's name. $numberByOleBoy{$oleboy} = $phonenumber; # Build an associative array so that we can lookup the # ole boy's name based on the wife's name. if ($oleboyswife ne "none") { $oleBoyByWife{$oleboyswife} = $oleboy; } } # Set the output to be unbuffered. $ = 1; # Get the user's input. print "Enter 'exit' or a name to lookup: "; while (<>) { chomp; if ($_ eq "exit" $_ eq "quit") { last; } # Is the name a good ole boy? if (defined $numberByOleBoy{$_}) { # Yes, print out the number. print "Found good ole boy's number at $numberByOleBoy{$_} "; } # Otherwise, is the name a good ole boy's wife? elsif (defined $oleBoyByWife{$_}) { # Yes, print out the number. print "Found good ole boy's number using his wife at $numberByOleBoy{$oleBoyByWife{$_}} "; } else { print "Didn't find number (the lookup is case sensitive) "; } print "Enter 'exit' or a name to lookup: "; } phoneList.pl goodoleboys.txt Enter 'exit' or a name to lookup: Bill Found good ole boy's number at 333-4444 Enter 'exit' or a name to lookup: Daisy Found good ole boy's number using his wife at 333-4444 Enter 'exit' or a name to lookup: Claude Found good ole boy's number at 333-4340 Enter 'exit' or a name to lookup: Etheline Found good ole boy's number using his wife at 333-4340 Enter 'exit' or a name to lookup: Fred Didn't find number (the lookup is case sensitive) Enter 'exit' or a name to lookup: exit

Figure 22.7: This example shows a functioning, interactive phone-number lookup interface.

When developing perl programs that interact with Qshell, you can use Qshell utilities to perform tasks while processing input or output for the utilities with perl. Interacting with Qshell commands in this fashion lets you reuse the broad set of utilities for their functions.

Figure 22.8 shows a simple version of running Qshell commands and processing their output from perl. Since perl is a PASE utility, the PASE qsh utility is used to execute a Qshell command.

cat qshellFromPerl.pl #!/usr/bin/perl -w if ($#ARGV != 0 $ARGV[0] eq "-h") { print "Usage: qshellFromPerl [-h] "; print "Run a qshell command using PASE perl utility "; exit(1); } # Create a file handle connected to the output # of a Qshell process # A first character of "" would be used to create # a file handle that could be written to, in order # to send input to the created process. open(QSH, "qsh -c "$ARGV[0]" ") die "Couldn't open the Qshell command"; while () { chomp; print "QSHOUT: $_ "; } exit(0); qshellFromPerl.pl "ls -S" QSHOUT: 819 lookupExample.pl QSHOUT: 819 newLocalPrefix.pl QSHOUT: 819 phoneList.pl QSHOUT: 819 phoneListDB.pl QSHOUT: 819 ptfStatus.pl QSHOUT: 819 qshellFromPerl.pl QSHOUT: 819 showLocalBoyDialList.pl QSHOUT: 819 showLocalBoys.pl

Figure 22.8: Use the perl utility to process the output from any Qshell utility. Process the output in whichever way is appropriate.

Building on the ability to invoke a Qshell command, Figure 22.9 uses the Qshell system utility to interact directly with a CL command. Invoking the system utility like this is a shortcut to processing that may use a temporary file to hold the output from a Qshell or CL command or utility.

cat ptfStatus.pl #!/usr/bin/perl -w if ($#ARGV < 0 $ARGV[0] eq "-h") { print "Usage: ptfStatus [-h] "; print "Query the status of a list of PTFs "; exit(1); } $debug = 0; # Create a file handle connected to the output # of a process running the system command to process # the CL command DSPPTF. open(DSPPTF, 'system "DSPPTF OUTPUT(*PRINT)" ') die "Couldn't open DSPPTF command"; while () { chomp; # PTF list in the output is a 7 character name, # 2 alpha, 5 digits or # 3 alpha, 4 digits # prefixed by one space on each line. print "<$_> " if $debug; if (m/^s((D{2}d{5})(D{3}d{4}))s/) { # Use the PTF that we just matched in the parenthesis $ptf = ""; # Grab the PTF status from the line. $status = substr($_, 13, 19); # Remove trailing whitespace from status. $status =~ s/s*$//g; $ptfList{$ptf} = $status; print "OUT:$ptf $status " if $debug; } } print "------- -------------------- "; print "PTF Status "; print "------- -------------------- "; for ($i=0; $i<=$#ARGV; $i++) { $ptf = $ARGV[$i]; if (defined $ptfList{$ptf}) { print "$ptf $ptfList{$ptf} "; } else { print "$ptf Not Loaded "; } } exit(0); ptfStatus.pl SI06971 SI06972 SI06973 SI06974 SI06975 SI06976 SI06977 ------- -------------------- PTF Status ------- -------------------- SI06971 Temporarily applied SI06972 Superseded SI06973 Not Loaded SI06974 Not Loaded SI06975 Temporarily applied SI06976 Temporarily applied SI06977 Superseded

Figure 22.9: This perl example uses the system utility to run a CL command, and then processes the output.

When running CL commands, the system utility processes temporary spooled files and job-log data created by the command.

In the case of a native Qshell command, no temporary file is created. The APIs read or write directly to the running application via an interprocess communication mechanism called a pipe . As the program runs, it runs at the same time as the started application, reading data from or writing it to the pipe. The pipe buffers the data as appropriate, but there is never a single complete copy of all of the data.

Compare and contrast the perl code in Figure 22.9 to the Java example in Figure 23.8 and the C example in Figure 24.7. The advanced functions available with perl regular expressions start to become more important in this example, enabling us to easily pick out the PTF number from the data stream that is produced by the Qshell command. Very little programming is needed to start and process the output from a Qshell utility; it almost happens naturally.

Since perl is a PASE application, the PASE system utility is executed in this example, not the Qshell system utility. The PASE and Qshell system utilities both work similarly. Use the PASE utility qsh to execute a Qshell utility from within perl (or other PASE applications). The output from the CL command is processed into an associative array. The associative array serves as the lookup mechanism for a list of randomly selected PTFs (Program Temporary Fixes).

The perl language provides a standard way to access a database using the perl DBI module. Provided with the perl implementation on iSeries is the DB2 implementation of perl database (DBI) access. Figure 22.10 demonstrates a more realistic phone-number lookup application than the example shown in Figure 22.7. The names and phone numbers reside in the relational database on the iSeries. Compare and contrast this perl database example to the Java example in Figure 23.8 and the C example in Figure 24.8.

cat phoneListDB.pl #!/usr/bin/perl -w use DBI; use DBD::DB2; use DBD::DB2::Constants; if (defined $ARGV[0]) { print "Usage: phoneList [-h] "; exit(1); } # Create the database connection and the statement to use. $hdbc = DBI->connect("dbi:DB2:*LOCAL") die "Couldn't connect"; $ps = $hdbc->prepare("SELECT PHONENUMBER FROM QSHELLDATA.CUTOMERS " . "WHERE NAME = ? OR WIFE = ?") die "Couldn't prepare, err=$hdbc->err"; # Set the output to be unbuffered. $ = 1; # Get the user's input. print "Enter 'exit' or a name to lookup: "; while (<>) { chomp; if ($_ eq "exit" $_ eq "quit") { last; } # Set the input parameters $ps->bind_param(1, $_); $ps->bind_param(2, $_); $ps->execute() warn "Couldn't execute, err=$hdbc->err"; # retrieve the first row. @row = $ps->fetchrow; if (!defined $row[0]) { print STDERR "Didn't find customer: $_ "; } else { print STDOUT "Found $_ at $row[0] "; } $ps->finish(); print "Enter 'exit' or a name to lookup: "; } $hdbc->disconnect(); exit(0); phoneListDB.pl Enter 'exit' or a name to lookup: Bill Found Bill at 333-4444 Enter 'exit' or a name to lookup: Daisy Found Daisy at 333-4444 Enter 'exit' or a name to lookup: Claude Found Claude at 333-4340 Enter 'exit' or a name to lookup: Etheline Found Etheline at 333-4340 Enter 'exit' or a name to lookup: Fred Didn't find customer: Fred Enter 'exit' or a name to lookup: exit

Figure 22.10: Use perl to access the iSeries database. The additional modules for database access are provided with the implementation of perl. This example does the phone-number lookup demonstrated in Figure 22.7, but uses the database for input, instead of a text file.

Using the perl operator along with the die or warn function lets a concise error-handling model take shape in this program. The database functions used by this program are designed to be similar to the Java and C models, which helps developers who program in multiple languages. The strength of automatic memory-management and automatic conversion of data types shows here. The program doesn't need to concern itself about the size or type of data parameters, or with allocating storage for them.

Summary

The perl language provides a powerful text-processing environment. You can use perl to access Qshell and iSeries services, creating simple yet powerful scripts or applications.