IBM Rational ClearCase, Ant, and CruiseControl: The Java Developers Guide to Accelerating and Automating the Build Process

Build auditing refers to the capability whereby during the build process, the versions of all the source files used, the compilation settings, and the environment are automatically recorded. With ClearCase you can achieve this only if you are using dynamic views. The ClearCase multiversion file system (MVFS) that is used with dynamic views audits all the low-level system calls (opens, reads, writes) that are performed on ClearCase data. This capability was primarily designed for C or C++ builds carried out using the clearmake tool; however, it can still be used for Java builds. This section describes build auditing in more detail and shows how it can be used to audit Ant-initiated Java builds. First, however, I will explain a couple of the concepts behind build auditing: derived objects and configuration records.

Note

ClearCase build auditing is a new feature in IBM Rational ClearCase version 7, so some of the information in this section might have changed slightly before the final release.

What Is a Derived Object?

A derived object is any file that is created by a build script under the control of the ClearCase clearmake, omake, or clearaudit command. Derived objects can be created only in dynamic views, and they can be shareable or nonshareable. During a clearmake or omake audited build, information about any newly built derived objects is written to the ClearCase VOB. This information can then be used by future builds to determine whether the derived object can be reused, thus avoiding rebuilding and potentially decreasing the build time. The process of pulling in a derived object built in another view is called "winking in." Derived objects created in this way are thereby shareable.

Because shareable derived objects can be created and reused only when a build is invoked via the clearmake or omake command, you have to write makefiles to take advantage of this capability. This is obviously not something today's Java developers are usually prepared to undertake. However, an audited build invoked by the clearaudit command or the ClearCase Ant "build audit listener" can also create derived objects. These derived objects are called "nonshareable" because they cannot be automatically reused by builds initiated in other views. However, the information about the derived object, such as each of the file versions that went into producing it, can still be reported and used.

The cleartool lsdo command is used to find and query derived objects. This command displays the fully qualified names of all derived objects created in the current VOB directory, regardless of the view in which the derived object was built. For example, for a traditional C++ project, cleartool lsdo might produce the following output:

M:\build_view\CPPBank\>cleartool lsdo 14-Sep.20:28 "Account.o@@15-Jan.20:28.110" 14-Sep.19:56 "Account.o@@14-Jan.19:56.105" 14-Sep.19:56 "Bank.o@@14-Jan.19:56.104" 14-Sep.20:28 "BankTest.exe@@15-Jan.20:28.111" 14-Sep.19:56 "BankTest.exe@@14-Jan.19:56.108" 14-Sep.19:56 "BankTest.o@@14-Jan.19:56.103" 14-Sep.19:56 "Customer.o@@14-Jan.19:56.106" 14-Sep.19:56 "TransRecord.o@@14-Jan.19:56.107" ...

In this example, you can see a number of derived objects, either Windows executables or object files. As in this case, an environment may consist of many different derived objects that are essentially variants of the same file. An example is multiple instances of an executable that has been built with different versions of code. To distinguish between them, each derived object has an extended name in the form filename@@unique-identifier:

BankTest.exe@@14-Jan.19:56.108

Each derived object also has an associated configuration record that describes how it was constructed. Note that you might see a different variant of identifier (such as BankTest.exe@@2006--01-14T19:56.108) depending on what ClearCase release or environment you are using. Configuration records are discussed next.

What Is a Configuration Record?

A configuration record is a compiled object and cannot be viewed using standard operating system commands. Instead, you must use the cleartool catcr derived object identifier command:

>cleartool catcr Account.o@@15-Jan.20:28.110

Executing this command produces output similar to the following:

Derived object: \CPPBank\RationalBank\Account.o@@15-Jan.20:28.110 Target Account.o built by alex.ClearCase Users Host "gorilla" running NT 5.1 (i586) Reference Time 15-Jan.20:28:31, this audit started 15-Jan.20:28:33 View was gorilla:c:\ClearCase_Storage\views\GORILLA\alex\build_view.vws Initial working directory was M:\default\CPPBank ---------------------------- MVFS objects: ---------------------------- \CPPBank\Account.cpp@@\main\2 <14-Jan-06.19:45:05> \CPPBank\Account.h@@\main\1 <14-Jan-06.09:38:53> \CPPBank\Account.o@@14-Jan.19:56.105 ---------------------------- Build Script: ---------------------------- @echo 'Building file: Account.cpp' @echo g++ -O0 -g3 -Wall -c -fmessage-length=0 -oAccount.o Account.cpp @g++ -O0 -g3 -Wall -c -fmessage-length=0 -oAccount.o Account.cpp @echo 'Finished building: Account.cpp' ----------------------------

Note that you generate a configuration record only if you use clearmake, omake, or clearaudit to initiate the build. However, clearaudit has some auditing limitations compared to the others, as indicated in the following list. In general, a configuration record consists of the following sections.

• Header section Shows who and what view initiated the build, as well as where and when.

• MVFS section Shows all the MVFS files and directories read during the build. This includes versioned objects and view-private files (including checked-out versions), as well as all the derived objects produced by the target rebuild.

• Non-MVFS section Shows any declared dependencies outside a VOB, identified by a date and time stamp.

• Variables and Options section (clearmake/omake only) Shows any macros defined during the build.

• Build Script section (clearmake/omake only) Shows the actual script that was read from the makefile and executed by clearmake or omake.

You can use various options to customize the display of the configuration record, as well as an additional and sometimes very useful command cleartool diffcr to compare two derived objects. Essentially, this command can show you the difference (in ClearCase element versions) between derived objects.

Now that you have a basic understanding of derived objects and configuration records, I will walk through an example of carrying out an audited Java build using ClearCase and Ant.

Auditing Ant Builds with ClearCase

To audit your Ant-based Java builds, you can use the ClearCase Ant build audit listener. This is an Ant library that audits aspects of your build process, allowing you to create derived objects and configuration records for Java builds. However, this capability does not give you automatic build avoidance capabilities. The listener comes bundled with ClearCase as the file CCAudits.jar and is held in the ClearCase bin directory. For Ant to execute this listener successfully, you need to add the path to the CCAudits.jar file to your Java CLASSPATH. For example, on Windows, your CLASSPATH should be set similar to the following:

CLASSPATH=%CLASSPATH%;C:\Program Files\Rational\ClearCase\bin\CCAudits.jar

On Linux or UNIX, it should be set similar to the following:

CLASSPATH=$CLASSPATH:"/opt/rational/clearcase/java/lib/CCAudits.jar"

On Windows you can set this configuration permanently by setting the environment variables using the System applet in Control Panel. On Linux or UNIX you can add the CLASSPATH setting to your login script.

Once you have specified a suitable CLASSPATH, you must then decide which Ant tasks you want to audit. You can specify the set of tasks by adding the following line to your default.properties or build.properties file:

CCAudits.auditable_task_list=javac:jar:war

This example audits the javac, jar, and war tasks. (If you do not specify any values for this property, the default audited task list is the javac, javah, jar, war, and ear tasks.) To execute an audited build, you simply execute your normal Ant target but specify the build audit listener:

>ant -listener CCAudits dist

This executes an audited build of the Ant dist target.

When a build is executed in a dynamic view and using the build audit listener, every build output (such as a Java .class file) is recorded as a derived object. A number of additional derived objects are also created in the CCAudits directory where the build was executed from. This directory contains a set of derived objects named after your project and audited targets. For each build audit, ClearCase creates a single configuration record, recording all the source versions used. All MVFS files read during execution of the audited shell are listed as inputs to the build, and all MVFS files created become derived objects, associated with the single configuration record. What this means is that after the build has finished, if you carried out a listing of the derived objects using the cleartool lsdo command, you would see output similar to the following:

M:\RatlBankModel_rel\RatlBankSources\model>cleartool lsdo - recurse --12-29T15:04 "CCAudits\clean@@--12-29T15:04.2147483695" --01-03T13:20 "CCAudits\compile@@--01-03T13:20.2147483716" --01-03T13:20 "CCAudits\dist@@--01-03T13:20.2147483729" --01-03T13:20 "CCAudits\init@@--01-03T13:20.2147483684" --12-29T15:04 "CCAudits\javadoc@@--12-29T15:04.2147483734" --12-29T15:04 "CCAudits\junit-all@@--12-29T15:04.2147483675" --01-03T13:20 "CCAudits\make-manifest@@--01-03T13:20.2147483727" --01-03T13:20 "CCAudits\RatlBankModel@@--01-03T13:20.2147483694" --01-03T13:20 "build\test\com\ratlbank\model\TestAccount.class@@--01-03T13:20.2147483772" --01-03T13:20 "build\test\com\ratlbank\model\TestBank.class@@--01-03T13:20.2147483744" --01-03T13:20 "build\test\com\ratlbank\model\TestCustomer.class@@--01-03T13:20.2147483743" --01-03T13:20 "build\test\com\ratlbank\model\TestTransRecord.class@@--01-03T13:20.2147483741" ... --01-03T13:20 "dist\RatlBankModel.jar@@--01-03T13:20.2147483681"

This is essentially a listing of all the .class and .jar files that the build produced. If you executed cleartool catcr on any of these derived objects, you would see the build's configuration record:

[View full width]

You will notice that with this invocation you do not get to see the Variables and Options section. You see only the MVFS objects in terms of the Java .class files that were bundled into the RatlBankModel.jar file. To see the ClearCase element versions for the .java source files that make up these classes, you can use some additional options to the cleartool catcr command. For example, the -union option consolidates all recursive configuration records into a single output:

[View full width]

This example shows you the versions of the source files (such as Account.java) that have gone into the build. You can specify the output format for the cleartool catcr command in a number of ways; see the manual page for further examples. This type of build auditing information has many uses. For example, you could use it to construct release notes or include it as part of your release distribution.

Another feature of ClearCase Ant build auditing is that you can display the configuration record for the Ant project or its targets, which are written as special derived objects to the CCAudits directory, as in the following:

[View full width]

In this example, as well as the list of derived objects, you can also see the Variables and Options section recording how the build was constructed in terms of the JDK and Ant version. The only caveat to this process is that the configuration information is available only as long as the derived objects exist. If you subsequently execute an Ant clean, the derived objects are removed. However, you can check in (or stage) the derived object to preserve this information. Chapter 10, "The Art of Releasing," discusses a mechanism to achieve this.

Earlier I discussed being able to compare configuration records. Suppose you used the process just described, and now you have two derived objects called X and Y. For the second build you knew that a number of elements had changed, but you were not sure which ones. To find out this information, you could use the cleartool diffcr command:

>cleartool diffcr -flat -ele -s X Y

The arguments to the diffcr command (-flat -ele) produce a nicely formatted output of just the ClearCase elements that have changed. The output from this command would be similar to the following:

< 1 \src\com\ratlbank\model\Account.java@@\main\2 <2006-01-08T18:30:11-02> > 1 \src\com\ratlbank\model\Account.java@@\main\4 <2006-01-08T19:36:48-05> ---------------------------- < 1 \src\com\ratlbank\model\Customer.java@@\main\2 <2006-01-01T14:36:12-03> > 1 \src\com\ratlbank\model\Customer.java@@\main\3 <2006-01-08T18:06:48-05>

As you can see, the difference between these two builds is with the Java files Account.java and Customer.java, which have had new versions implemented in the later build.

Executing an Audited Build Using Eclipse

To execute an audited build, you need to configure the Eclipse-invoked version of Ant to use the CCAudits.jar file and the -listener argument. To do this, change the configuration for running Ant builds by selecting Run, External Tools, External Tools. This brings up the Ant Build configuration, as shown in Figure 9.8.

In the Configurations list, select the build file configuration you want to change. (In this case, it is for invoking the RationalBankModel build.xml file.) Then, on the Main tab, in the Arguments box enter this value

-listener CCAudits

so that the CCAudits listener is invoked by the build. Next, on the Classpath tab, shown in Figure 9.9, highlight User Entries. Click the Add External JARs button and browse to the location of the CCAudits.jar file, such as C:\Program Files\Rational\ClearCase\bin.

Click Apply and then close the External Tools window. If you run this configuration now, an audited build is executed, and the CCAudits directory is created as just described. To check that this has worked, you can enter cleartool lsdo -r from the command line to see that some derived objects have been created.

Viewing a Configuration Record Using Eclipse

You also can view the configuration records for the derived objects that were created directly from inside Eclipse. To do this you need to add a new External Tool configuration. Select Run, External Tools, External Tools. You see the window shown in Figure 9.10. In the Configurations list, highlight Program, and then click the New button. Enter the following values:

• Name cleartool catcr

• Location The location of the cleartool executable, such as C:\Program Files\Rational\ClearCase\bin\cleartool.exe or an equivalent location for Linux or UNIX

• Arguments catcr ${resource_loc}

This creates a new configuration for executing the cleartool catcr command.

Highlight any of the derived objects that were created during the build, such as the files in the CCAudits directory, and then select Run, External Tools, cleartool catcr. The cleartool catcr command is run, and the output is displayed in the Eclipse console, as shown in Figure 9.11.

You can change the exact invocation of the cleartool catcr command depending on how you want the output to be displayed. See the cleartool catcr manual page for more information on the available options.

Executing an Audited Build Using CruiseControl

To execute an automated audited build, you need to configure CruiseControl to build in a dynamic view. Furthermore, you need to ensure that when CruiseControl initiates Ant, an appropriate CLASSPATH is set up and the ClearCase build audit listener is called correctly. The best way to achieve this is to create an Ant wrapper script that sets up an environment for an audited build, as shown in Listing 9.4.

Listing 9.4. ClearCase Audited Build Wrapperaudited-ant.bat

You need to change the #Path to ClearCase# field to the location where ClearCase is installed in your environment, such as C:\Program Files\Rational\ClearCase. This script should be version-controlled alongside your existing Ant scripts:

JavaTools/ant/bin

You then can configure CruiseControl to build in a dynamic view and call this new script:

<project name="RatlBankModel_int"> <property name="dir.ratlbankmod"value="M:\RatlBankModel_int\RatlBankSources\model"/> ... <clearcaseviewstrapper viewpath="${dir.ratlbankmod}" voblist="\RatlBankSources,\RatlBankReleases"/> <modificationset> <clearcase branch="RatlBankModel_Int" viewpath="${dir.ratlbankmod}" recursive="true"/> </modificationset> <schedule interval="600"> <ant antWorkingDir="${dir.ratlbankmod}" antscript="${dir.javatools}\ant\bin\audited-ant.bat" buildfile="${dir.ratlbankmod}/build.xml" target="integration"/> </schedule> ... </project>

In this example, the RatlBankModel project is accessed using a dynamic view. On Windows, dynamic views can always be accessed through the special M: drive, as in this example. On UNIX, this access occurs through the special /view mount point:

<property name="dir.ratlbankmod" value="/view/RatlBankModel_int/vobs/RatlBankSources/model"/>

Note that for builds to work in dynamic views, you should make sure that the build view has been started and that the VOBs that are to be accessed in the view are mounted as well. This is typically a problem only when a machine is rebooted. However, to prevent this problem, you can make use of the <clearcaseviewstrapper> plug-in as shown above.