Real World Web Services

The code for this application relies on a library called Quartz, a pure Java scheduler. Quartz (http://www.quartzscheduler.org/) provides reliable scheduling for tasks. You can define a task schedule with potentially very complex rules (for example, fire a task every weekday at 2 a.m. or once every two hours). The reliability can be achieved using Quartz with a relational database. Quartz can automatically store and update job information in the database via JDBC.

This sample application is less concerned with the broader feature set of Quartz and instead focuses on using Quartz to provide basic job-scheduling capabilities. Therefore, the core of the application is based on an implementation of the interface org.quartz.Job. This interface defines a single method, execute(JobExecutionContext context), called when the task is to be performed. All the data associated with the task is passed in via the org.quartz.JobExecutionContext object.

When the application starts, it initializes a single org.quartz.Scheduler (watcherScheduler) object. Then, as the user adds watchers, the implementation adds an org.quartz.SimpleTrigger and the appropriate org.quartz.Job implementation. That's it: Quartz takes care of calling application tasks, handling the scheduling as needed.

8.4.1 AbstractWatcher Implementation

The underlying Java code for this application is divided into a few basic classes, as shown in Figure 8-6. There's an abstract base, AbstractWatcher, which includes several static access methods for the JSP pages. The rest of the application code consists of concrete implementations of this base class, one for each type of supported Watcher (AmazonWatcher, EBayAuctionWatcher, EBaySearchWatcher, GoogleWatcher, and RSSWatcher). The Job interface is required by Quartz.

The main class, AbstractWatcher, is shown in Example 8-4. The first thing you'll notice about the AbstractWatcher class is that the application relies heavily on the supporting Quartz classes. Our main data store is the Quartz Scheduler object. In addition to the obvious advantage of code reuse, it means that the application relies on the Quartz data storage facilities. In this example, the application uses the default RAM-based storage mechanism, but it's possible to use the built-in Quartz data storage facilities (as described at http://quartz.sourceforge.net/firstTutorial.html#jobStores) to save the queue and related data to a supported relational database.

Example 8-4. AbstractWatcher.java

package com.cascadetg.ch08; import java.util.Hashtable; import org.quartz.*; public abstract class AbstractWatcher implements Job { static final boolean debug = false; static Scheduler watcherScheduler = null; static final Object[] watchers = { new AmazonWatcher( ), new EBayAuctionWatcher( ), new EBaySearchWatcher( ), new GoogleWatcher( ), new RSSWatcher( )}; /** Constant for use with the frequency. */ static public final String hourly = "Hourly"; /** Constant for use with the frequency. */ static public final String daily = "Daily"; /** Constant for use with the frequency. */ static public final String weekly = "Weekly"; /** * This is the main method that is required to be implemented. If * the update completes successfully, it should return true, * otherwise, return false. */ abstract boolean update( ); abstract public void execute(JobExecutionContext context) throws JobExecutionException; public abstract AbstractWatcher getNew( ); static public void init( ) { try { if (watcherScheduler == null) { SchedulerFactory schedulerFactory = new org.quartz.impl.StdSchedulerFactory( ); watcherScheduler = schedulerFactory.getScheduler( ); watcherScheduler.start( ); } } catch (Exception e) { e.printStackTrace( ); } } /** The (human-readable) description of the type of Watcher */ public abstract String getType( ); public static AbstractWatcher getAbstractWatcher(String input) { for (int i = 0; i < watchers.length; i++) { if (input.equals(((AbstractWatcher)watchers[i]).getType( ))) return ((AbstractWatcher)watchers[i]).getNew( ); } return null; } public static String[] getTypes(boolean spaces) { String[] temp = { new AmazonWatcher( ).getType( ), new EBayAuctionWatcher( ).getType( ), new EBaySearchWatcher( ).getType( ), new GoogleWatcher( ).getType( ), new RSSWatcher( ).getType( )}; if (spaces) for (int i = 0; i < temp.length; i++) temp[i] = temp[i].replace('_', ' '); return temp; } public static AbstractWatcher[] getSortedWatches( ) { init( ); java.util.TreeMap map = new java.util.TreeMap(java.text.Collator.getInstance( )); try { if (watcherScheduler .getJobNames(Scheduler.DEFAULT_GROUP) .length == 0) return null; String[] keys = watcherScheduler.getJobNames(Scheduler.DEFAULT_GROUP); for (int i = 0; i < keys.length; i++) { AbstractWatcher temp = getWatcher(keys[i]); map.put( temp.getLastUpdate( ) + " " + temp.getType( ), temp); } } catch (SchedulerException e) { e.printStackTrace( ); } Object[] myCollection = map.values( ).toArray( ); AbstractWatcher[] result = new AbstractWatcher[myCollection.length]; for (int i = myCollection.length - 1; i >= 0; i--) result[i] = (AbstractWatcher)myCollection[i]; return result; } public static AbstractWatcher getWatcher(String id) { try { JobDetail myJob = watcherScheduler.getJobDetail( id, Scheduler.DEFAULT_GROUP); AbstractWatcher temp = getAbstractWatcher( (String)myJob.getJobDataMap( ).get("type")); temp.setJobDetail(myJob); return temp; } catch (Exception e) { e.printStackTrace( ); } return null; } static final long MINUTE = 60L * 1000L; static final long HOUR = 60L * 60L * 1000L; static final long DAY = 24L * 60L * 60L * 1000L; static final long WEEK = 7L * 24L * 60L * 60L * 1000L; public static void addWatch( String type, String target, String frequency) { AbstractWatcher myWatcher = getAbstractWatcher(type); myWatcher.setTarget(target); myWatcher.setFrequency(frequency); myWatcher.update( ); new_id++; myWatcher.id = new_id; long timing = HOUR; if (frequency.equals(daily)) timing = DAY; if (frequency.equals(weekly)) timing = WEEK; if (debug) timing = MINUTE; SimpleTrigger trigger = new SimpleTrigger( myWatcher.getID( ) + "", Scheduler.DEFAULT_GROUP, new java.util.Date( ), null, SimpleTrigger.REPEAT_INDEFINITELY, timing); init( ); try { watcherScheduler.scheduleJob( myWatcher.getJobDetail( ), trigger); } catch (Exception e) { e.printStackTrace( ); } } public static void removeWatch(String id) { try { watcherScheduler.deleteJob(id, Scheduler.DEFAULT_GROUP); } catch (SchedulerException e) { e.printStackTrace( ); } } private String target = null; /** The meaning of the target depends on the type of Watcher. */ public String getTarget( ) { return target; } /** The meaning of the target depends on the type of Watcher. */ public void setTarget(String target) { this.target = target; } private String frequency = hourly; /** The frequency the Watcher will be updated. Defaults to hourly. */ public void setFrequency(String frequency) { this.frequency = frequency; } /** The frequency the Watcher will be updated */ public String getFrequency( ) { return frequency; } private String url; public String getURL( ) { return url; } public void setURL(String url) { this.url = url; } private java.util.Date last_update = new java.util.Date( ); public java.util.Date getLastUpdate( ) { return last_update; } public void setLastUpdate(java.util.Date new_date) { last_update = new_date; } Hashtable attributes = new Hashtable( ); public java.util.Map getAttributes( ) { java.util.TreeMap map = new java.util.TreeMap(java.text.Collator.getInstance( )); map.putAll(attributes); return map; } public void setAttribute(String attribute, String value) { if (attribute == null) return; if (value == null) return; attributes.put(attribute, value); } public String getAttribute(String attribute) { return (String)attributes.get(attribute); } public String getFormattedAttribute(String attribute) { String temp = (String)attributes.get(attribute); if (temp.startsWith("http://")) { StringBuffer wrapped = new StringBuffer( ); wrapped.append("<a target='_blank' href='"); wrapped.append(temp); wrapped.append("'>"); wrapped.append(temp); wrapped.append("</a>"); return wrapped.toString( ); } return temp; } public String getHumanType( ) { return getType( ).replace('_', ' '); } static long new_id = System.currentTimeMillis( ); long id; public String getID( ) { return id + ""; } JobDetail jobDetail; public JobDetail getJobDetail( ) { if (jobDetail != null) { return jobDetail; } jobDetail = new JobDetail( this.id + "", Scheduler.DEFAULT_GROUP, this.getClass( )); jobDetail.getJobDataMap( ).put("url", url); jobDetail.getJobDataMap( ).put("target", target); jobDetail.getJobDataMap( ).put("type", this.getType( )); jobDetail.getJobDataMap( ).put("frequency", this.getFrequency( )); jobDetail.getJobDataMap( ).put("attributes", attributes); jobDetail.getJobDataMap( ).put("id", id); jobDetail.getJobDataMap( ).put("date", this.getLastUpdate( )); return jobDetail; } public void setJobDetail(JobDetail in) { this.setURL((String)in.getJobDataMap( ).get("url")); this.setTarget((String)in.getJobDataMap( ).get("target")); this.setFrequency((String)in.getJobDataMap( ).get("frequency")); this.attributes = (Hashtable)in.getJobDataMap( ).get("attributes"); this.id = ((Long)in.getJobDataMap( ).get("id")).longValue( ); this.last_update = ((java.util.Date)in.getJobDataMap( ).get("date")); } }

The code for adding new watchers contains a constant not used in the application directly, however, the Minute constant, which is used for time intervals, is also useful for debugging purposes. When using the MINUTE constant on a development system, be careful. Most large-scale web service providers (including Google, Amazon, and eBay) block connections considered abusive.

Quartz keeps track of tasks using serialized data. Each instance of a task being run is called a Job. Therefore, the AbstractWatcher class implements the Quartz Job interface and handles the various bookkeeping details that arise when a Job is created, run, and destroyed. For example, the getJobDetail( ) and setJobDetail( ) methods pass data between the AbstractWatcher classes and the associated Quartz JobDetail object.

Quartz creates and deletes instances of the various AbstractWatcher objects as tasks are run. For example, when an Amazon watcher task is added to the list of tasks to be performed, the data relating to the task is stored as a JobDetail object. Later, when the task is executed, the data needed for the task is returned as a JobDetail object. Quartz can be configured to save the data associated with a JobDetail object different ways, including in memory or in a relational database. It's possible to pass in a serialized instance of a Java object, but serialization is notably finicky about versioning. As it is, the sample application shown here relies on simply serializing a java.util.Hashtable object. This allows a developer to rework the underlying implementations without having to worry about the proper versioning of data in the Quartz persistent store. Regardless of how the task implementations are reworked, the stored data is simply an ordinary java.util.Hashtable.

8.4.2 Amazon Watcher

The code for the first watcher, the AmazonWatcher class, is shown in Example 8-5. It relies on the same SOAP access code as originally described in Chapter 4.

Example 8-5. Amazon watcher implementation

package com.cascadetg.ch08; import org.quartz.JobExecutionContext; import org.quartz.JobExecutionException; import com.amazon.soap.*; import com.cascadetg.ch04.DeveloperTokens; public class AmazonWatcher extends AbstractWatcher { public String getType( ) { return "Amazon"; } public boolean update( ) { this.setLastUpdate(new java.util.Date( )); // Mac OS X for Java Geeks ISBN //String isbn = "0596004001"; try { AmazonSearchService myAmazonSearchService = new AmazonSearchServiceLocator( ); AmazonSearchPort myAmazonSearchPort = myAmazonSearchService.getAmazonSearchPort( ); AsinRequest myAsinRequest = new AsinRequest( ); // Use this to set your Amazon Associates ID // For more info on Amazon Associates, see... // http://www.amazon.com/associates myAsinRequest.setTag(DeveloperTokens.amazon_associates); myAsinRequest.setDevtag(DeveloperTokens.amazon_token); myAsinRequest.setAsin(this.getTarget( )); myAsinRequest.setLocale("us"); myAsinRequest.setType("heavy"); ProductInfo myProductInfo = myAmazonSearchPort.asinSearchRequest(myAsinRequest); Details[] myDetailsArray = myProductInfo.getDetails( ); Details myDetail = null; if (myDetailsArray != null) { myDetail = myDetailsArray[0]; this.setAttribute( "Product Name", myDetail.getProductName( )); this.setAttribute( "Release Date", myDetail.getReleaseDate( )); this.setAttribute( "Actual Prize", myDetail.getOurPrice( )); this.setAttribute( "List Price", myDetail.getListPrice( )); this.setAttribute( "Used Price", myDetail.getUsedPrice( )); this.setAttribute( "Sales Rank", myDetail.getSalesRank( )); this.setAttribute( "Availability", myDetail.getAvailability( )); this.setURL(myDetail.getUrl( )); } } catch (Exception e) { e.printStackTrace( ); } return false; } public AbstractWatcher getNew( ) { return new AmazonWatcher( ); } public void execute(JobExecutionContext context) throws JobExecutionException { context.getJobDetail( ).getJobDataMap( ).put( "date", new java.util.Date( )); this.setJobDetail(context.getJobDetail( )); this.update( ); context.getJobDetail( ).setJobDataMap( this.getJobDetail( ).getJobDataMap( )); System.out.println( this.getLastUpdate( ).toString( ) + " - Running Amazon " + this.getID( )); } }

8.4.3 eBay Auction Watcher

The watcher, shown in Example 8-6, allows a user to monitor a specific auction. The code relies on the eBay access code developed in Chapter 4, in this case, passing in a new eBay verb, GetItem. Setting this to be updated hourly is typical: it's always good to be on top of the auction for a signed collector's edition of Wil Wheaton's Just a Geek.

Example 8-6. Monitoring a specific eBay auction

package com.cascadetg.ch08; import org.quartz.*; import com.cascadetg.ch04.EbayAPISimpleCall; public class EBayAuctionWatcher extends AbstractWatcher { public String getType( ) { return "eBay_Auction"; } public boolean update( ) { this.setLastUpdate(new java.util.Date( )); EbayAPISimpleCall myCall = new EbayAPISimpleCall( ); myCall.setApiVerb("GetItem"); myCall.setArgument("Id", this.getTarget( )); myCall.setArgument("DetailLevel", 8 + ""); org.jdom.Document myResults = myCall.executeCall( ); org.jdom.Element root = myResults.getRootElement( ); org.jdom.Element item = root.getChild("Item"); try { this.setAttribute( "Bid Count", item.getChildText("BidCount")); } catch (Exception e) { } try { this.setAttribute( "Current Price", item.getChildText("CurrentPrice")); } catch (Exception e) { } try { this.setAttribute("End Time", item.getChildText("EndTime")); } catch (Exception e) { } try { this.setAttribute( "Winning Bidder", item.getChild("HighBidder").getChild( "User").getChildText( "UserId")); } catch (Exception e) { } try { this.setAttribute( "Winning Bidder Email", item.getChild("HighBidder").getChild( "User").getChildText( "Email")); } catch (Exception e) { System.out.println("No UserID Email Found"); } try { this.setAttribute( "Winning Bidder Feedback", item .getChild("HighBidder") .getChild("User") .getChild("Feedback") .getChildText("Score")); } catch (Exception e) { } return false; } public AbstractWatcher getNew( ) { return new EBayAuctionWatcher( ); } /* * (non-Javadoc) * * @see com.cascadetg.ch08.AbstractWatcher#execute(org.quartz. JobExecutionContext) */ public void execute(JobExecutionContext context) throws JobExecutionException { context.getJobDetail( ).getJobDataMap( ).put( "date", new java.util.Date( )); this.setJobDetail(context.getJobDetail( )); this.update( ); context.getJobDetail( ).setJobDataMap( this.getJobDetail( ).getJobDataMap( )); System.out.println( this.getLastUpdate( ).toString( ) + " Running eBay Auction " + this.getID( )); } }

8.4.4 eBay Search Watcher

The code in Example 8-7 shows how to monitor a particular search query on eBay (for example, all auctions containing the term "expresso"). The eBay access code in Chapter 4 actually makes the connection.

Example 8-7. eBay search watcher

package com.cascadetg.ch08; import org.quartz.JobExecutionContext; import org.quartz.JobExecutionException; import com.cascadetg.ch04.EbayAPISimpleCall; public class EBaySearchWatcher extends AbstractWatcher { public String getType( ) { return "eBay_Search"; } public boolean update( ) { this.setLastUpdate(new java.util.Date( )); EbayAPISimpleCall myCall = new EbayAPISimpleCall( ); myCall.setApiVerb(EbayAPISimpleCall.GetSearchResults); myCall.setArgument("Query", this.getTarget( )); myCall.setArgument("Order", "MetaHighestPriceSort"); org.jdom.Document myResults = myCall.executeCall( ); org.jdom.Element root = myResults.getRootElement( ); long count = Long.parseLong( root .getChild("Search") .getChild("GrandTotal") .getText( )); this.setAttribute( "eBay Total Matching Listings", Long.toString(count)); if (count > 0) { org.jdom.Element item = root.getChild("Search").getChild("Items").getChild( "Item"); this.setAttribute( "eBay Highest Bid Item Price", item.getChildText("LocalizedCurrentPrice")); this.setAttribute( "eBay Highest Bid Item Bids", item.getChildText("BidCount")); this.setAttribute( "eBay Highest Bid Item Link", item.getChildText("Link")); } return false; } public AbstractWatcher getNew( ) { return new EBaySearchWatcher( ); } public void execute(JobExecutionContext context) throws JobExecutionException { context.getJobDetail( ).getJobDataMap( ).put( "date", new java.util.Date( )); this.setJobDetail(context.getJobDetail( )); this.update( ); context.getJobDetail( ).setJobDataMap( this.getJobDetail( ).getJobDataMap( )); System.out.println(this.getLastUpdate( ).toString( ) + " Running eBay Search " + this.getID( ) ); } }

8.4.5 Google Watcher

The code in Example 8-8 shows a watcher intended to monitor a particular Google search term, relying on the Google SOAP access as defined in Chapter 4. This application is interested in a smaller data set than that shown in Chapter 4 and Chapter 7.

Example 8-8. Google watcher

package com.cascadetg.ch08; import org.quartz.JobExecutionContext; import org.quartz.JobExecutionException; import com.google.soap.search.*; import com.cascadetg.ch04.DeveloperTokens; public class GoogleWatcher extends AbstractWatcher { public String getType( ) { return "Google_Search"; } public boolean update( ) { this.setLastUpdate(new java.util.Date( )); GoogleSearch search = new GoogleSearch( ); // Set mandatory attributes search.setKey(DeveloperTokens.googleKey); search.setQueryString(this.getTarget( )); // Set optional attributes search.setSafeSearch(true); // Invoke the actual search GoogleSearchResult result = null; try { result = search.doSearch( ); } catch (GoogleSearchFault e) { e.printStackTrace( ); } // process the result if (result != null) { this.setAttribute( "Google Number of Hits", Integer.toString( result.getEstimatedTotalResultsCount( ))); GoogleSearchResultElement[] mySearchElements = result.getResultElements( ); for (int i = 0; i < mySearchElements.length; i++) { this.setAttribute( "#" + (i + 1) + " Result ", mySearchElements[i].getTitle( )); this.setAttribute( "#" + (i + 1) + " Snippet ", mySearchElements[i].getSnippet( )); this.setAttribute( "#" + (i + 1) + " URL ", mySearchElements[i].getURL( )); if (i > 3) { i = mySearchElements.length + 1; } } } return false; } public AbstractWatcher getNew( ) { return new GoogleWatcher( ); } public void execute(JobExecutionContext context) throws JobExecutionException { context.getJobDetail( ).getJobDataMap( ).put( "date", new java.util.Date( )); this.setJobDetail(context.getJobDetail( )); this.update( ); context.getJobDetail( ).setJobDataMap( this.getJobDetail( ).getJobDataMap( )); System.out.println( this.getLastUpdate( ).toString( ) + " Running Google " + this.getID( )); } }

8.4.6 RSS Watcher

Finally, the code in Example 8-9 shows the code for reading an RSS feed. It relies on the Informa library (http://informa.sourceforge.net/, originally used in Chapter 7) for parsing the RSS.

Example 8-9. RSS watcher

package com.cascadetg.ch08; import org.quartz.JobExecutionContext; import org.quartz.JobExecutionException; import de.nava.informa.core.ChannelIF; import de.nava.informa.core.ItemIF; import de.nava.informa.impl.basic.ChannelBuilder; import de.nava.informa.parsers.RSSParser; public class RSSWatcher extends AbstractWatcher { public String getType( ) { return "RSS_Feed"; } public boolean update( ) { this.setLastUpdate(new java.util.Date( )); try { java.net.URL inpFile = new java.net.URL(this.getTarget( )); ChannelIF channel = RSSParser.parse(new ChannelBuilder( ), inpFile); this.setAttribute( "Site Description", channel.getDescription( )); this.setAttribute("Site URL", channel.getSite( ).toString( )); Object[] items = channel.getItems( ).toArray( ); for (int i = 0; i < items.length; i++) { ItemIF current = (ItemIF)items[i]; try { if (current.getTitle( ).length( ) > 0) if (!current.getTitle( ).equals("<No Title>")) this.setAttribute( "#" + (i + 1) + " Title", current.getTitle( )); } catch (Exception e) { } try { if (current.getLink( ) != null) this.setAttribute( "#" + (i + 1) + " URL", current.getLink( ).toString( )); } catch (Exception e) { } try { if (current.getSubject( ).length( ) > 0) this.setAttribute( "#" + (i + 1) + " Subject", current.getSubject( )); } catch (Exception e) { } try { String temp = current.getDescription( ); if (temp.length( ) > 255) { temp = temp.substring(0, 253); temp = temp + "..."; } temp = replaceToken(temp, "<", "&lt;"); temp = replaceToken(temp, ">", "&gt;"); if (temp.length( ) > 0) this.setAttribute( "#" + (i + 1) + " Description", temp); } catch (Exception e) { } if (i > 3) { i = items.length + 1; } } } catch (Exception e) { e.printStackTrace( ); } return false; } public AbstractWatcher getNew( ) { return new RSSWatcher( ); } public void execute(JobExecutionContext context) throws JobExecutionException { context.getJobDetail( ).getJobDataMap( ).put( "date", new java.util.Date( )); this.setJobDetail(context.getJobDetail( )); this.update( ); context.getJobDetail( ).setJobDataMap( this.getJobDetail( ).getJobDataMap( )); System.out.println( this.getLastUpdate( ).toString( ) + " Running RSS " + this.getID( )); } /** All parameters must not be null. */ static public String replaceToken( String input, String token, String value) { if (input == null) throw new NullPointerException("replaceToken input is null"); if (token == null) throw new NullPointerException("replaceToken token is null"); if (value == null) throw new NullPointerException("replaceToken value is null"); boolean done = false; int current = 0; int last = 0; StringBuffer results = new StringBuffer(""); while (!done) { last = current; current = input.indexOf(token, current); if (current == -1) done = true; if (!done) { results.append(input.substring(last, current)); results.append(value); current = current + token.length( ); } else { results.append(input.substring(last)); } } if (input.length( ) > 0) if (results.toString( ).length( ) == 0) return input; return results.toString( ); } }

The code shown in Example 8-9 is, interestingly, the longest of the watchers (despite relying on the Informa library) and also the most awkward. The reason for this is a basic problem with RSS: too much of the flow of RSS isn't properly defined (or at least, wildly and creatively interpreted).

Some RSS feeds literally return the entire text of their article as their description, complete with formatting. A few feeds with a few complete articles in each feed can completely overwhelm the formatting of the page. You can't just chop the feed description at a certain point without dealing with the possibility of embedded HTML as well; an opening < character with no closing tag will ruin your page. Here I replace the < and > characters with the equivalent HTML entities, which means that end users will see the tags interspersed in the text and chop the text after a certain point. It might be better to simply strip the contents of the tags, but at times, certain entries will consist of nothing but (for example) an IMG tag pointing to an interesting graphic. Stripping tags in that instance means that the feed entry will appear to be empty. You might look for IMG tags and replace them with text such as [Image here], perhaps looking for the alt text. Soon, however, the logic for this starts to approach a minimal web browser, and your code is much more complex.