IBM Rational ClearCase, Ant, and CruiseControl: The Java Developers Guide to Accelerating and Automating the Build Process

You can execute your build either manually or automatically. Normally a manual process involves several steps, either by typing in commands or by navigating through a series of windows or menus. In this section, manual execution means deciding to invoke a build script from the command line. This could be a single script or several. Automatic execution means that the decision is made for you with the same scripts executed by a scheduling toolin this case, CruiseControl.

The Manual Approach

Chapter 4, "Defining Your Build and Release Scripts," described how any Ant build script can be executed manually. It is simply a matter of running the ant command from the command line against the build.xml file and using the specific target you want to execute as a parameter. For example:

>ant compile

With Ant's popularity you also can execute Ant build scripts directly from inside many IDEs, as I discussed with Eclipse in Chapter 4. This combination of capabilities is certainly sufficient to meet the needs of developers conducting Private Builds within their own workspaces. However, there are also other instances where you might want to invoke the build manually, such as when you need to debug your build scripts or when you want more control when executing Release Builds.

The Automatic Approach

Executing the build scripts manually is fine in some circumstances, but for day-to-day development, having someone around to manually invoke a build at regular intervals can create unnecessary delays in the development process. This is particularly relevant if a continuous integration approach is required, but it also applies if builds are carried out on a scheduled basis at least once a day. For example, if a nightly build is to be carried out, you wouldn't want someone waiting around till the middle of night and then manually invoking the build!

The solution to this problem is to use a build scheduling tool. You can implement this scheduling process yourself via cron, at, or even the ClearCase scheduler. However, you can use a number of tools that offer significant additional functionality in automating the build process. This chapter describes how to use one of these tools, CruiseControl, to automate this process.

A typical network setup for CruiseControl is illustrated in Figure 6.1.

Figure 6.1 includes a dedicated build server to execute the builds on and upon which CruiseControl is installed. The figure also includes our SCM repository (ClearCase) and a Web/application server (Apache/Tomcat or WebSphere Application Server).

The basic way that CruiseControl works is to monitor a project integration stream or branch. If changes have been committed to this branch, the build is executed on a predefined schedule. This build is basically an invocation of the Ant build scripts that have already been defined.

In Figure 6.1, since the application being developed is a Web application, CruiseControl can (via our Ant build scripts) also deploy the application to the Web or application server for us. Finally, CruiseControl is also responsible for informing us of the build's status, such as its success or failure. The next section describes how to configure a CruiseControl environment to implement this process.

Unlike with Ant, you probably do not have a copy of CruiseControl. Therefore, you must download it and install a specific version into the JavaTools directory structure that was created in Chapter 3, "Configuring Your SCM Environment." CruiseControl usually has two downloadable filesa binary compiled version and a source code version. Because I will make changes to the base distribution in the next chapter, here I describe how to download the source code version and install it into the JavaTools directory structure.

Download and Install CruiseControl

First, download and extract the CruiseControl distribution from the CruiseControl Web site: http://cruisecontrol.sourceforge.net/download.html. There should be a link for downloading the source code distribution; in this case download the zip file cruisecontrol-2.4.x.zip (or the latest version of CruiseControl that is available). Extract this file to a directory. I recommend creating an src directory in the JavaTools cruisecontrol component and extracting the file here first. Note that you can subsequently add this directory to source control too if you make any changes to the distribution. Unlike Ant, CruiseControl has no automated installation script, so you should create a template directory structure similar to that shown in Figure 6.2. This will hold all the built CruiseControl programs and libraries.

CruiseControl comes with prebuilt binaries, and since the code is written in Java it is platform-independent and can be used as is. However, for completeness (and because I will be customizing CruiseControl in subsequent chapters), I recommend learning how to build CruiseControl from its source. To do this, go to the command line, navigate to the main directory under cruisecontrol/src/cruisecontrol-2.4.x, and execute the following command on Windows:

>build.bat

or the following command on Linux/UNIX:

>build.sh

The CruiseControl build process has no install target, so you have to copy its files into the JavaTools directory manually. I recommend the following process:

You might also want to copy the CruiseControl docs directory into the JavaTools cruisecontrol directory. Each of these files and directories should then be added to source control and a new baseline or label applied so that any user can pick it up. Now that CruiseControl has been installed, the next step is to set up a workspace in which to execute it.

Create a Workspace

To execute CruiseControl, you need to create some configuration files; these are best kept in a specific directory workspace. As usual, these files should be version-controlled! I recommend creating (or reusing) a ClearCase view onto the JavaTools cruisecontrol component and then creating a directory structure for the CruiseControl configuration files under an etc directory. This structure is illustrated in Figure 6.3.

You need to create a logs directory to hold the project logs for each CruiseControl-invoked build and an artifacts directory to hold the results of any builds you want to save (such as distribution .jar or .war files). Finally, I recommend copying the CruiseControl stylesheets from the main/reporting/jsp directory (all the .xsl and .css files) to this location because you might want to change them to fit your company's look and feel. These stylesheets are used to format the HTML e-mails that are sent to indicate build success or failure. I will discuss them in more detail in the next chapter.

You can now use this workspace to configure a CruiseControl project. Next, we will walk through an example of setting up an environment to build our RatlBankModel project.

Create a Project Build Environment

For each project that CruiseControl will build, you need to create a project build environment containing the following:

• An Ant script for CruiseControl to execute

• A view for CruiseControl to execute the build in

You can create the Ant script in two ways. You can create a new Ant "build control file" in the CruiseControl workspace (for example, etc/config/RatlBankModel_V1_build.xml), or you can add a new target to a project's existing Ant build file. My preference is the latter because it keeps everything in a single place, and there is no danger that the two sets of files could get out of step.

To configure an existing project, you would add a target (called integration, for example) to the project's Ant build.xml file. This target will be used to execute a project's Integration Build. Even if this target is normally used by the CruiseControl build, there is no reason why it couldn't be executed manually as well. Listing 6.1 shows a typical updated build.xml file.

Listing 6.1. CruiseControl Build Target

You can see here that the integration target is carrying out Integration Build-type operations. In other words, not only is it building the Java .jar distribution files, but it also is deploying the application and producing the documentation. You will notice that this example also has a call to the target update-view to update a snapshot view. This particular target sets the failonerror attribute to false so that if you were executing the build in a dynamic view (and the command cleartool update is not required or permitted) the build would not fail.

This leads us to the second pointcreating a view for CruiseControl to execute the build in. For every project to be executed by CruiseControl, you should create an Integration Build view (a convention discussed in Chapter 3). For example, for our RatlBankModel project, create a view called RatlBankModel_bld. This view should be configured to look at the latest versions of files on the project integration or release branch. It should not be used for any purpose other than executing your CruiseControl Integration Builds. I prefer to create snapshot views from which to carry out Integration Builds, so in this case in Windows I would have a snapshot view rooted at C:\Views\RatlBankModel_bld. I will use this location in the following section when I define the CruiseControl project configuration file.

Define the Project

All CruiseControl configuration parameters are defined in an XML file (by convention called config.xml or cc-config.xml). In this case I will create a config.xml file in the JavaTools/cruisecontrol/etc directory. The general format of a CruiseControl config.xml file is as follows:

<cruisecontrol> <project name="project1"> <listeners> ... </listeners> <bootstrappers> ... </bootstrappers> <modificationset> ... </modificationset> <schedule> ... </schedule> <log> ... </log> <publishers> ... </publishers> </project> <project name="project2"> ... </project> ... </cruisecontrol>

As you can see, the config.xml file can reference multiple projects that CruiseControl will schedule and run. As an example, I will walk through the process of configuring a single CruiseControl project called RatlBankModel. This example will be complete enough to execute a simple CruiseControl build; however, I will subsequently expand on it in Chapter 7 to illustrate more complex examples.

Define Properties

As of CruiseControl 2.2.2, support for user-defined properties was made available. This allows you to define properties in a way similar to Ant (as explained in Chapter 4). For example, to define a property value for the JavaTools and RatlBankModel home directories, you could use the following:

<property name="dir.javatools"value="C:\Views\RatlBankModel_bld\JavaTools"/> <property name="dir.ratlbankmodel"value="C:\Views\RatlBankModel_bld\RatlBankSources\model"/>

You can then refer to properties in exactly the same way as in Ant. For example, to refer to the JavaTools property, you would use ${dir.javatools}. Unlike in Ant, properties in CruiseControl are not completely immutable. Whoever sets a property last freezes its value within the scope in which the property was set. For example, you may set a property at the global level and then redefine it at the project level. If you try to set a property more than once within the same scope, only the last assignment is used. Finally, you can also use the property ${project.name} to refer to the name of the current project. CruiseControl sets this for you automatically.

Define Listeners

CruiseControl Listeners can be notified on every project event, but most are designed to handle a subset of events. The Listener that you will typically use is called <currentbuildstatuslistener>. It is used to specify a file that records whether CruiseControl is currently building a project. This information is used by the CruiseControl Build Results web application (described in Chapter 7). The <listeners> enTRy that you would create in this case in your config.xml file would be as follows:

<listeners> <currentbuildstatuslistener file="logs/${project.name}/buildstatus.txt"/> </listeners>

The file that this listener produces contains an HTML snippet that records the build's current status and the last time the status changed.

Define Bootstrappers

CruiseControl Bootstrappers are run before a build takes place, regardless of whether the build is necessary. Typically, one of the things you want to do before a build is make sure that all your build scripts and environment are up to date. This is particularly relevant if you will build in a ClearCase snapshot view. You can use a specific <bootstrapper> for ClearCase called <clearcasebootstrapper> to specify the name and location of a file to update. The files you typically want to update are build.xml and default.properties. The <bootstrappers> entry you would create in your config.xml would be as follows:

<bootstrappers> <clearcasebootstrapper file="build.xml" viewPath="${dir.ratlbankmodel}"/> <clearcasebootstrapper file="default.properties" viewPath="${dir.ratlbankmodel}"/> </bootstrappers>

If you were using a dynamic view to build in, the build files would be automatically up to date, and you could leave out the <clearcasebootstrapper> plug-ins. Instead, you might want to make use of the <clearcaseviewstrapper> plug-ins to ensure that the build view is started and the relevant VOBs are mounted prior to the Ant build, as follows:

<property name="dir.ratlbankmodel" value="M:\RatlBankModel_bld\RatlBankSources\model"/> <bootstrappers> <clearcaseviewstrapper viewpath="${dir.ratlbankmodel}" voblist="\RatlBankSources,\RatlBankReleases"/> </bootstrappers>

This example would ensure that the dynamic view RatlBankModel_bld was started (as referenced via the property ${dir.ratlbankmodel} and that the VOBs \RatlBankSources and \RatlBankReleases were also mounted.

Define SourceControls: Check for Modifications

CruiseControl was designed with continuous integration in mind. Therefore, the default mode of working is to initiate a build only when files or directories are changed on a branch or stream. (The next chapter looks at how to schedule builds not on changes but at specific times.) This set of files and directories is called a <modificationset>. For ClearCase, CruiseControl needs to know which ClearCase branch (or UCM stream) to look at and which view to look into to find the changes. It does this via the <clearcase> source control plug-in. To use this plug-in, the <modificationset> entry you would create in your config.xml would be as follows:

<modificationset quietperiod="90"> <clearcase branch="RatlBankModel_Int" viewpath="${dir.ratlbankmodel}"/> </modificationset>

By default, CruiseControl recurses down the complete directory structure specified by the viewpath attribute (in this case, the RatlBankModel top-level directory). You can, however, specify multiple <modificationset> elements to check only parts of a large project rather than checking all files every time. I discuss different approaches for this in Chapter 7.

In the <modificationset> entry you will notice that I have also specified an additional parameter called quietperiod. This refers to how many seconds must pass before a build can commence. It is included to prevent a build from being carried out when a developer is in the process of checking in multiple files. If an additional modification (a check-in) is detected in the quiet period, CruiseControl waits until the quiet period has finished and then rechecks for modifications. The default value for quietperiod is 60 seconds.

If you are using UCM, you can make use of the <ucm> source control plug-in. This works in a way similar to the <clearcase> plug-in but reports on UCM activities rather than element versions. To use this plug-in, the <modificationset> enTRy you would create in your config.xml would be as follows:

<modificationset quietperiod="90"> <ucm stream="RatlBankModel_Int" viewpath="${dir.ratlbankmodel}" contributors="true"/> </modificationset>

As you can see, this example is very similar; however, the stream name is specified rather than the branch name. There is also an additional option to report on contributor activities. These are the activities that developers have worked on and delivered into the build rather than just the activities on the integration stream itself. There are a number of capabilities and best practice ways of working with UCM and CruiseControl; for more information, see www.buildmeister.com/cruisecontrol+ucm.php.

Define Builders: Schedule the Build

Probably the most important aspect of CruiseControl configuration is defining the criteria for when a build should occur. CruiseControl allows builds to be scheduled either on an interval basis (such as every 30 minutes) or at a specific date and time (such as a nightly build). By default, builds are executed only if modifications are found. It is usually not a good idea to mix interval-based builds and time-based builds in the same project, because the interval-based builds will consume all the changes before they can be detected by the time-based builds. I will walk through some more complex examples of build schedules in Chapter 7; however, for now I will describe how to carry out a simple interval-based build schedule to occur every 20 minutes.

To implement a schedule, you would create a schedule entry in your config.xml file and define the interval in seconds. For 20 minutes this would be 1200 seconds. Inside the <schedule> entry you define what Builder you want to use to carry out the build. In this case you want to run Ant on RatlBankModel's build.xml script. The <schedule> entry that you would create in your config.xml would be as follows:

<schedule interval="1200"> <ant antscript="${dir.javatools}/ant/bin/ant.bat" antWorkingDir="${dir.ratlbankmodel}" buildfile="${dir.ratlbankmodel}/build.xml" target="integration"/> </schedule>

In this example, you can see that I call the ant Builder using our JavaTools version of Ant and the build.xml file that is in the RatlBankModel build view. The target in this build.xml file is the integration target I created earlier.

Merge Test Results

As part of the Integration Build process, you should run some JUnit unit tests to assess whether the build is of sufficient quality to be used for further development or testing. As part of a continuous integration implementation, the notification of the failure of unit tests gives the developers immediate feedback on their work and consequently a chance to fix problems before their code base has significantly evolved. The output of a JUnit test run is usually an XML log file with the test results; normally it is stored in the RatlBankModel project's build directory. For this log file to be presented back via the CruiseControl reporting application, you need to merge this log file into CruiseControl's own log file. The <log> entry you would create in config.xml would be as follows:

<log> <merge file="${dir.ratlbankmodel}/build/TESTS- TestSuites.xml"/> </log>

The file TESTS-TestSuites.xml contains the results of a complete JUnit test runall the test cases.

Define Publishers

The final configuration entry you need to carry out before you have a working CruiseControl build configuration is to define how and when you want to inform users of the build's results. You do this by creating a <publishers> enTRy. Publishers are run after a build has completed. They are typically run regardless of whether the build was successful. As part of this process you can perform various activities, such as copying the results of the build to a central location, sending an instant message, or executing an additional command. The most common activity is to send an e-mail with the build's results. You can do this via either the <email> or <htmlemail> element. The difference between the two is that specifying the <email> element just sends an e-mail with a link to the CruiseControl Build Results web application. Specifying <htmlemail> formats the e-mail as HTML and includes information such as the list of modifications and information on any errors found during the build (see Figure 6.4). Since most e-mail readers can display HTML e-mails, I will create an <htmlemail> enTRy. The <publishers> entry that you would create in config.xml would be as follows:

<htmlemail mailhost="smtp.mailhost.com" returnaddress="bldadmin" xsldir="${dir.javatools}/cruisecontrol/etc/webcontent/xsl" css="${dir.javatools}/cruisecontrol/etc/webcontent/css/cruisecontrol.css" buildresultsurl="http://build-server:8080/cruisecontrol/buildresults/${project.name}" subjectprefix="[CruiseControl]: " <always address="integrators@mailhost.com"/> <failure address="developers@mailhost.com"/> </htmlemail>

This example specifies that an e-mail will be sent in two situations. First, an e-mail will be sent to the integration team every time a build is carried out, irrespective of whether it succeeds or fails. Second, if a build fails, an e-mail will be sent to all the developers. The e-mail will include a link to the CruiseControl Build Results web (as specified via the buildresultsurl entry). (I describe how to install the Build Results web in the next chapter.) You need to change smtp.mailhost.com to the name of your SMTP server and the e-mail addresses accordingly.

The remaining two attributes are xsldir and css. They specify the files to be used to transform the CruiseControl log files into a suitable HTML presentation format. I recommend copying the directories reporting/jsp/webcontent/xsl and css from the CruiseControl distribution directory to the JavaTools cruisecontrol/etc directory structure (as shown in Figure 6.3) so that they are held in a central location where you can version-control them.

The second publisher I will use is <artifactspublisher>. This can be used to publish or copy the results of the build to a specific location. An example of <artifactspublisher> is as follows:

<publishers> <artifactspublisher dir="${dir.ratlbankmodel}/dist" dest="artifacts/${project.name}"/> </publishers>

This example copies all the files that have been built in the dist directory (usually the .jar files) to the artifacts directory that was created earlier. These files actually are placed in a subdirectory called RatlBankModel/timestamp, where timestamp tells when the publisher was executed. Once these files have been published, they are available directly from the CruiseControl Build Results web, which is described in the next chapter.

The Completed config File

The basic CruiseControl configuration file is now completed. The completed config.xml file for our RatlBankModel project is shown in Listing 6.2.

Listing 6.2. RatlBankModel config.xml File

Test the config File

Now that a completed config.xml file exists, you can start CruiseControl. To do so you need to go to the command line and execute the CruiseControl executable against this file. You would execute the following on Windows:

>cd C:\Views\javatools_int\JavaTools\cruisecontrol\etc >..\bin\cruisecontrol.bat -configfile config.xml

or the following on Linux/UNIX:

>cd /Snapshot_views/javatools_int/JavaTools/cruiscontrol/etc >../bin/cruiscontrol.sh -configfile config.xml

CruiseControl then starts. By default, CruiseControl assumes that its configuration file is called config.xml, so there really is no need to specify the -configfile parameter here. However, I have included it in case you want to have a differently named file or multiple configuration files. Either way, you should see output similar to the following:

"c:\j2sdk1.4.2_09\bin\java" -cp c:\j2sdk1.4.2_09\lib\tools.jar; C:\Views\javatools_int\JavaTools\cruisecontrol\bin\\..\dist\ cruisecontrol.jar ... -Djavax.management.builder.initial= mx4j.server.MX4JMBeanServerBuilder CruiseControl -configfile config.xml [cc]Jan-27 18:17:35 Main - CruiseControl Version 2.4.2-dev Compiled on January 27 2006 1333 [cc]Jan-27 18:17:36 LConfigManager- Calculating MD5 [C:\Views\javatools_int\JavaTools\cruisecontrol\etc\config.xml] [cc]Jan-27 18:17:36 LConfigManager- reading settings from config file[613652380a55bb6564ab3ddaf143f421] [cc]Jan-27 18:17:36 trolController- projectName = [RatlBankModel_int] [cc]Jan-27 18:17:36 LConfigManager- using settings from config file [C:\Views\javatools_int\JavaTools\cruisecontrol\etc\config.xml] [cc]Jan-27 18:17:36 trolController- projectName = [RatlBankModel_rel] [cc]Jan-27 18:17:36 LConfigManager- using settings from config file [C:\Views\javatools_int\JavaTools\cruisecontrol\etc\config.xml] [cc]Jan-27 18:17:36 Project - Project RatlBankModel_int starting [cc]Jan-27 18:17:36 Project - Project RatlBankModel_int started [cc]Jan-27 18:17:36 Project - Project RatlBankModel_int: idle [cc]Jan-27 18:17:36 Project - Project RatlBankModel_int: next build in 5 minutes [cc]Jan-27 18:17:36 Project - Project RatlBankModel_int: waiting for next time to build

If CruiseControl stops right away, this usually indicates a problem with the config.xml file. The description of the problem will be located in the output.

To verify that CruiseControl is working correctly, modify a file on the integration branch or stream that you have set up to be monitored. In the console output you should see that a modification has been found and a build initiated. Depending on the results of the build, you should also get an e-mail.

This completes our tour of how to create a simple CruiseControl configuration. The next chapter expands on this configuration to implement some best practices.