Tuning WebLogic Applications

Many aspects of WebLogic's performance were already covered in earlier chapters. We examined the impact of various features on the flexibility of your application setup and discussed the trade-offs involved. We've looked at various session persistence mechanisms, proxy web servers, the benefits of JDBC connection pools and data sources, the collocation and transaction optimizations that occur when using EJBs, the load-balancing and failover features in a WebLogic cluster, the different ways to reduce the overheads of network traffic, the issues involved when JMS connection factories are used in a transacted session, and the various packaging and deployment options that also make a difference in application performance. Many of these performance enhancements are the outcome of adjusting either the configuration settings supported by WebLogic-specific deployment descriptors or those configuration options that can be viewed and modified from the Administration Console.

We won't revisit all of these issues here. We presume that you have read (or browsed through) the previous chapters and now have a good understanding of how to build applications on WebLogic Server, making effective use of the supported J2EE services.

Instead, we discuss specific bottlenecks that can impede the performance of WebLogic applications. After all, WebLogic can run only as well as the applications deployed to it. Before you consider allocating blame for missing performance targets on other aspects of your configuration, it is wise to first scrutinize your application setup and try to resolve any glaring performance hitches. This chapter hopefully adds to your existing arsenal of weapons for improving application performance. Along the way, we provide details on how to monitor the application resources you may use and how to interpret these performance figures.

15.1.1 Managing Sessions

HTTP session management is very important in determining the performance of your web applications. In general, you should consider using HTTP sessions only when the session-state information cannot reliably be held on the client. You shouldn't use the server-side HTTP session to share objects between different parts of the web application. Whenever possible, you should place frequently used values in local variables. The best performance usually can be extracted by using sessions only when necessary, and then holding as little data as necessary.

If your web application is deployed to a cluster and you've enabled session persistence, in-memory replication is almost 10 times faster than JDBC-based persistence. Whenever possible, you should place fewer, coarse-grained objects into the HTTP session, rather than multiple, fine-grained objects. In case of in-memory replication, every "put" operation on the session means that the update needs to be propagated to the secondary server that holds a replica of the HTTP session. In case of JDBC-based session persistence, every "put" operation triggers a database write of the entire object. Fewer coarse-grained objects means you are able to minimize the number of "put" operations on the HTTP session. If the objects in your session are too large say, an object tree a single change still requires you to update the entire tree, and hence replicate the entire tree. Clearly, this is less efficient than if your session data had been split across a number of variables, in which case changes to individual variables would require less replication. Thus, your decisions on the size of the objects in the HTTP session will depend on the volume of changes to the session data and the structure of the session data. Ultimately, you need to strike a balance between the volume of data replicated and the number of replications.

Chapter 2 explains the various session persistence mechanisms WebLogic provides for a web application.

15.1.1.1 Monitoring servlets and sessions

To monitor servlets, select a web application in the Administration Console and select the Monitoring tab. From here you can monitor the web application of which the servlet is a part, the servlet runtimes, and active sessions. When monitoring servlets, you will be able to view deployed servlets and JSPs, together with an indication of the number of times the page or servlet has been accessed, the number of times the servlet has been reloaded, the URL pattern mapped to the servlet, and statistics on how much time was spent in each servlet. This may help determine potential bottlenecks. As the current trend is to use an MVC architecture, in which a controller servlet delegates incoming requests to appropriate action handlers, the statistic is less useful. In this case, it is far more useful to employ a profiler to determine the time spent in each handler.

When monitoring a web application, you will find a list containing the context root for the web application, together with the number of active sessions, the number of total sessions, and the highest number of active sessions that you have had. To optimally tune your application, you must know not only the average number of sessions, but also the average session size information that WebLogic does not provide. One way to do this is to use memory profiling tools and examine the relevant objects and their sizes.

You can make WebLogic gather additional session statistics by selecting the Session Monitoring Enabled option. You then will need to reboot your server for session monitoring to take effect. This provides access to the Time Last Accessed attribute of a servlet, which may be useful to determine the behavior of your clients on the server. For example, you can use this to tune the default session timeout.

15.1.2 Optimizing JDBC Pools

WebLogic's support for JDBC connection pools minimizes the cost of opening and closing connections to the database. The connection pool can grow and shrink between the configured minimum and maximum number of open connections. In addition, you can specify an initial capacity for the pool that defines the initial number of physical database connections that are set up for the pool.

During development stages, you can set the initial capacity to a low number to help WebLogic start up faster. However, in a production environment you should consider setting the pool's initial capacity equal to its maximum capacity. All database connections should preferably be acquired before WebLogic becomes available. If the pool's initial capacity is lower than its maximum capacity, WebLogic needs to create additional connections when there is increased load on the JDBC pool. When the server is under load, it is preferable that all of its resources are used to serve requests as fast as possible, rather than spending time creating database connections.

The pool's maximum capacity defines the maximum number of database connections that the pool holds at any time. You need to set the pool's maximum capacity to a sensible number, taking into account any limitations of the DBMS. The ideal setting for the maximum pool size depends on your application. A good place to start is to set the pool's maximum capacity to the size of your execution thread count (discussed later). This makes sense, as you don't want an execution thread waiting to obtain a connection. However, as usual in performance tuning, the details may vary. If much of the work involved in entering the server will not require any JDBC connections, you could set the maximum pool size to a value less than the number of execute threads. On the other hand, the connection pool size may need to be greater than the number of execute threads if your application requires more than one connection for each thread. This situation arises, for example, when an EJB method that is using a connection calls another EJB method that has the "Requires New transaction" setting, before committing its own transaction. The method marked as Requires New will get its own connection from the connection pool.

Statement caching can offer a performance boost to JDBC access as well. By caching the compiled prepared (or callable) statements used most often, WebLogic can avoid recompiling a prepared statement the next time it is used. By default, WebLogic caches 10 prepared statements per pooled connection. The size of the statement cache may be limited by the number of open cursors allowed by the database. Chapter 5 discusses the advantages and limitations of enabling statement caching.

You should spend some time investigating the interaction between your application and JDBC resources, as often the performance of many applications is constrained by the database. A good indicator that the database is possibly the culprit is when you have low CPU utilization on a server, no matter how much load you apply.

15.1.2.1 Monitoring the connection pool and transactions

Connection pools can be monitored on each server by choosing the pool from the Administration Console and selecting the Monitoring tab. This will give you a good indication of the number of current connections using the pool, the highest number of connections, and the total number of connections. It also provides information on connection requests that have been made but could not be serviced because all of the connections were being used. This list, called the Waiters list, gives a good indication of potential bottlenecks in your database. The Wait Seconds High setting indicates the highest number of seconds that a request had to wait for a free connection, while Waiters High indicates the highest number of requests that had to be kept waiting for a connection. If this value is greater than zero, it indicates a shortage of JDBC resources. If client requests to the connection pool have to wait, consider increasing the size of the pool.

To monitor transactions, select the Monitoring/JTA tab for a server in the Administration Console. The statistics that are displayed here can give you an indication of the transactional behavior of your system. These are described in Table 15-1. Depending on your application, unusually high levels of rolled-back or heuristic completions of transactions could provide further information in tracking down performance problems. Likewise, excessive timeouts may indicate an overutilization of participating resources.

An important figure here is the ratio of rolled-back transactions to the total transactions. Rolled-back transactions, especially those that roll back due to a timeout, are unfortunate not only has the effort taken to roll back the transaction and all participating resources been wasted, but also the work hasn't been completed. Even worse, this same work probably will be retried later. You should always examine the reasons for a high ratio of rolled-back transactions it probably will lie in your code somewhere, usually as a result of poor transaction management or resource utilization.

The Monitoring/JTA tab also provides further links:

Monitor all transactions by name

This option allows you to monitor statistics about named transactions coordinated by the server. For example, if you have an EJB method that has a Requires transaction attribute, transactions due to calls to this method will appear here.

Monitor all transactions by resource

This option lets you monitor statistics for transaction resources accessed by the server. For example, it will include aggregates for your transactional connection pools and JMS file stores.

Monitor inflight transactions

This option shows inflight transactions those that have started, but which have not yet committed or rolled back.

If you are using EJBs in your application, you may want to examine these statistics together with the transaction and locking statistics that are available for EJBs.

15.1.3 Tuning the JMS Server

WebLogic provides a whole host of performance-related settings that can be configured for the JMS service. For each server or JMS destination, you can adjust the message/byte thresholds and quotas. You also can configure an upper limit for the number of JMS sessions that may be retrieved from a session pool. In addition, you can set the maximum number of messages that can be accumulated at a JMS consumer. All of these configuration aspects are explained in Chapter 8.

15.1.3.1 Optimizing paging and flow control

WebLogic's message paging feature lets you free up virtual memory during peak message loads. When the message loads reach (or go beyond) a configured threshold, WebLogic swaps out messages from the virtual memory to a persistent store. This prevents the JMS service from impeding the performance of other WebLogic services. In addition, WebLogic supports flow control features whereby the JMS server (or JMS destination) can instruct its message producers to limit their message flow when it's overloaded.

15.1.3.2 Optimizing persistence

In order to support persistent messaging, WebLogic allows you to configure a file or JDBC store. In general, file stores perform much better than JDBC stores. The scalability, performance, and reliability of the disk persistence are governed by the synchronous write policy that you choose for the file store. Choose the most appropriate for your application. Note that hardware and OS play a role here too, in the sense that the speed of the synchronous write may be determined by whether the system you use has a direct-write capability, and whether the disk cache can survive a power outage. Chapter 8 goes into more detail describing the pros and cons of the various synchronous write policies.

15.1.3.3 Monitoring the JMS servers

Finally, WebLogic captures a gamut of statistics for various JMS objects: JMS servers, connections, sessions, server session pools, destinations, durable subscribers, message producers, and consumers. By monitoring these statistics, you have enough information to properly tune the various resources available on JMS. To monitor JMS resources, choose a JMS server in the Administration Console and then select the Monitoring tab.

15.1.4 EJB Tuning

The WebLogic-specific deployment descriptors for an EJB describe the configuration settings for EJB pools, caching, concurrency strategy, clustering, and transaction isolation levels. Here, we examine the various performance-related settings that may be configured using the weblogic-ejb-jar.xml descriptor file.

15.1.4.1 Optimizing the EJB pool size

WebLogic establishes a free pool of EJBs for each stateless session bean, message-driven bean, or entity bean deployed to the server. The max-beans-in-free-pool element in the weblogic-ejb-jar.xml descriptor lets you specify the size of the free pool. By default, the max-beans-in-free-pool element has no upper limit, which means that the size of the free pool is limited only by the server's available memory. For best performance, you should use the default value for the max-beans-in-free-pool parameter, which lets the server run as many EJBs concurrently as possible, maximizing thread usage.

For stateless session and message-driven beans, you need to specify a value only if you need to limit the number of concurrent EJBs running on the server. If your application invokes a lot of finder and home methods or frequently creates entity EJB instances, you may wish to tune this parameter to ensure that the pool has enough anonymous entity EJB instances. If the cost of frequently creating and removing EJBs is quite significant, a fixed pool of EJBs also may improve application performance. If you find that your EJBs are quite database-intensive, it is best to let the size of the free pool be bound only by the server's available memory. Remember, even though you may have specified the maximum number of EJB instances held in the free pool, it still can be overridden by a WebLogic instance at runtime depending on the state of its memory resources.

Moreover, you can use the initial-beans-in-free-pool element in the weblogic-ejb-jar.xml descriptor file to specify the initial number of EJB instances that are created for the pool when the server boots up. By default, the free pool is empty when the server starts. This setting applies only to stateless session beans, message-driven beans, and entity beans. By establishing a ready pool of EJB instances, you can improve the initial response times for the server, as this will eliminate the delay of creating new EJB instances.

15.1.4.2 Optimizing the EJB cache size

The max-beans-in-cache element in the weblogic-ejb-jar.xml descriptor allows you to specify a maximum size for the EJB cache i.e., the cache of active EJB instances that are held in memory. When an EJB cache reaches its capacity, WebLogic passivates one or more EJB instances not currently used by any client. This behavior holds true for stateful session EJBs and entity beans. You should choose a suitable value for the size of the EJB cache so that excessive passivation of EJBs can be avoided otherwise, these actions negate any of the benefits of using an in-memory EJB cache. This is even more important when WebLogic needs to handle a large number of stateful session or entity EJBs.

Chapter 10 describes how you also can configure an application-level entity cache that is shared among all the EJBs that are packaged with the application.

15.1.4.3 EJB concurrency

WebLogic supports exclusive EJB locks, optimistic locks, and database locking. In general, WebLogic can provide better support for concurrent EJB access when it defers locking services to the underlying database. The underlying DBMS can offer finer granularity for locking data, and avoid deadlocks. The optimistic locking strategy often offers the best performance, as it minimizes the amount of locking that needs to occur. You can specify the locking mechanism for the EJB using the concurrency-strategy element in the weblogic-ejb-jar.xml descriptor. By default, WebLogic uses database locking.

15.1.4.4 Transaction isolation levels

The transaction isolation level for an EJB decides the extent to which multiple, interleaved transactions can interfere with each other. Thus, the transaction isolation level setting also affects the performance of your EJBs. In general, lower isolation levels result in better database concurrency but at the cost of reduced transaction isolation. Chapter 10 explains how to set the transaction isolation level for an EJB using the isolation-level element in the weblogic-ejb-jar.xml descriptor.

15.1.4.5 Optimizing CMP

EJB relationship caching can help improve the performance of accessing entity beans. WebLogic can automatically load related EJB instances into the cache by running a single join query to fetch the related EJBs and thereby avoid multiple queries. In addition, WebLogic allows you to defer cascade delete operations to the underlying database. In this way, the DBMS automatically handles the deletion of related EJB instances when the parent EJB instance has been removed. WebLogic also supports optimistic locking, which allows the EJB container to defer all database checks until the point of transaction commit. Fetch groups can also provide you with control over exactly which fields of an EJB are populated from the database.

In addition, batch inserts, updates, and deletes enable the EJB container to combine multiple database operations into a single JDBC operation. This helps reduce the number of network round trips for the EJB container and enhances the performance of your CMP entity beans. These and many other CMP optimizations are explained in Chapter 11. By carefully enabling these optimizations, you can improve the performance of an application that relies heavily on CMP entity beans.

15.1.4.6 Monitoring EJBs

However much you enhance your EJB's, you won't know if you are doing the right thing unless you monitor the statistics of your EJBs in action. WebLogic gathers a wealth of information to help guide you. To access this information, select an EJB module that is already deployed and then choose the Monitoring tab. Next, you must choose which types of beans in the JAR you wish to monitor either stateless, stateful, entity, or message-driven beans. WebLogic 8.1 provides a richer set of statistics than is available in WebLogic 7.0. We will mark each statistic that is available only in WebLogic 8.1 with the § symbol. All of the ratios quoted in the following list are made available to you in WebLogic 8.1's Administration Console. In WebLogic 7.0, you have to calculate the ratios yourself.

The first set of statistics relates to the free pool. These statistics are relevant to all bean types except stateful session beans because they don't have an associated pool.

Pool Beans Current Count[1]

This value indicates the number of bean instances available in the free pool.

Beans in Use Current Count

This value indicates the total number of bean instances currently in use.

Pool Waiter Total Count

This value indicates the total number of threads currently waiting for an available bean instance from the free pool.

Pool Timeout Total Count

This value indicates the total number of threads that have timed out waiting for an available bean instance from the free pool.

Access Total Count §

This value indicates the total number of times an attempt was made to get an instance from the free pool.

Miss Total Count §

This value indicates the total number of failures to get an instance from the free pool. For example, if the pool is empty, a failure will occur.

Pool Miss Ratio §

The pool miss ratio is calculated by dividing the Miss Total Count by the Access Total Count.

Pool Timeout Ratio §

This ratio is calculated by dividing the Pool Timeout Total Count by the Access Total Count.

Ideally, you never should get pool timeouts. By raising the value of the max-beans-in-free-pool element, you provide additional EJB instances for servicing requests; therefore, you reduce the number of pool timeouts. The transaction timeout configured for the EJBs also may contribute to a high number of pool timeouts. The amount of time a thread waits for an EJB from the pool is bound by the EJB's current transaction timeout. By increasing the transaction timeout setting for the EJB, threads wait longer for an EJB from the free pool.

If you limit the size of the free pool for stateless session beans, the miss ratio will indicate the ratio of calls that have had to wait for a bean to become available. Entity and message-driven beans do not wait for an instance to become available. If a pool miss occurs for these bean types, a new bean is created to service the request. Ideally, requests never should have to wait due to the pool being full, nor should they have to wait for a bean to be created. The pool timeout ratio provides an indication of the proportion of requests that timed out while waiting for a bean instance. If this value is high, you may need to increase the size of the pool.

Entity beans and stateful session beans have associated caches that hold active bean instances. At WebLogic's discretion, the beans in the cache can be passivated and activated again later. The following statistics are held for these bean types:

Cached Beans Current Count

This value indicates the total number of beans currently cached.

Cache Access Count

This value indicates the number of times that the cache has been accessed.

Cache Hit Count

This value indicates the number of times that a bean has been looked up and successfully found in the cache.

Activation Count

This value indicates the total number of beans that have been made active.

Passivation Count

This value indicates the total number of beans that have been passivated.

Cache Miss Ratio

This ratio can be calculated by dividing the cache hit count by the cache access count. This indicates the effectiveness of the cache.

The cache miss ratio is particularly useful. A high reading for the cache miss ratio may indicate that you need to resize the EJB cache. The amount of time saved by getting a bean from the cache depends on the cost of the bean's ejbActivate( ) method, as well as the bean's cache-between-transactions setting. Whenever a cache miss occurs, a bean will be obtained from the pool and its ejbActivate( ) method called. Obviously, the more expensive this method is to execute, the higher the penalty for a cache miss. If the EJB is configured with the cache-between-transactions set to true, a cache miss will also force the EJB container to make an extra call to the database to load the bean.

If certain EJB instances (primary keys) are used more frequently than others, you can resize the EJB cache to ensure that frequently used EJB instances remain in the cache, while less frequently used EJBs can enter or leave the cache on demand. By increasing the cache size, you could reduce the cache miss ratio. If a subset of EJB instances is not used more frequently than others, test your application setup against different EJB cache sizes to determine which gives you the lowest cache miss ratio. The server's memory is a valuable resource, so you need to balance the cache miss ratio against the memory requirements of the EJB cache.

Stateful and entity beans also have locking statistics associated with them:

Lock Entries Current Count

This value indicates the total number of beans that are currently locked.

Lock Manager Access Count

This value indicates the total number of attempts to obtain a lock on a bean, which includes attempts to obtain a lock on a bean that is locked already.

Lock Manager Waiter Total Count

This value indicates the total number of threads waiting for a lock on a bean.

Lock Manager Timeout Total Count

This value indicates the total number of transactions that have timed out while waiting for a lock on a bean.

Lock Waiter Ratio

This ratio is calculated by dividing the Lock Manager Waiter Total Count by the Lock Manager Access Count.

Lock Timeout Ratio

This ratio is calculated by dividing the Lock Manager Timeout Total Count by the Lock Manager Access Count.

There are a few caveats here. For entity beans, if you choose the optimistic concurrency strategy, no locks are held by the EJB container. Similarly, if you use the ReadOnly strategy, WebLogic activates a new instance for each transaction so that the requests can proceed in parallel and so that no locking occurs here. Finally, if you use the database concurrency strategy, locking is deferred to the underlying database. So, the only time an entity bean will be locked by the container is if you use the Exclusive locking strategy. The default concurrency strategy for entity beans is the database strategy.

The waiter ratio gives a good indication of how often a thread had to wait for a locked bean. A high reading for this ratio may indicate a less-than-optimal setting for the EJB's concurrency strategy. Choosing the database or optimistic locking guarantees greater EJB concurrency and eliminates the need for locking EJBs at the container level. Moreover, by reducing transaction times you can reduce the duration for which locks are held and improve EJB concurrency.

The timeout ratio gauges the number of times a thread timed out while waiting for a lock on an EJB. These timeouts are particularly harmful as they will result in an exception that will roll back the current transaction. A high timeout ratio may indicate an improperly configured transaction timeout, as the amount of time a thread waits for a lock is bound by the EJB's current transaction timeout. Ensure that the transaction timeout isn't set to a low value otherwise, a thread may not wait long enough for locks to be released and may prematurely time out. On the other hand, raising the transaction timeout value means your valuable server threads wait longer than usual for locks. Moreover, you may increase the request times because requests wait longer before timing out.

The final set of statistics that are gathered are those for transactions:

Transactions Committed Total Count

This value indicates the total number of transactions that have been committed for this EJB.

Transactions Rolled Back Total Count

This value indicates the total number of transactions that have been rolled back for this EJB.

Transactions Timed Out Total Count

This value indicates the total number of transactions that have timed out for this EJB.

Transaction Timeout Ratio

This ratio can be calculated by dividing the Transactions Timed Out Total Count by the Transactions Committed Total Count.

Transaction Rollback Ratio

This ratio can be calculated by dividing the Transactions Rolled Back Total Count by the Transactions Committed Total Count.

A high reading for the transaction timeout ratio could again indicate an improperly configured transaction timeout. Too low a value for the transaction timeout means your threads are timing out prematurely before the transaction can complete its work. Raising the transaction timeout value may reduce the number of transaction timeouts. Again, you need to exercise caution when raising the transaction timeout value, as this may generate additional bottlenecks for your server.

Before investigating a high transaction rollback ratio, first sort out any high transaction timeout ratios. Because timeouts cause rollbacks, it also will affect the rollback statistics. Otherwise, you should investigate which resources are causing the transactions to time out.

These statistics play a vital role in determining the ideal settings for an EJB's transaction timeout, concurrency strategy, and the size of the free pool and EJB cache. The complex interplay between these configuration settings determines the average request times, the degree of concurrency, and the efficiency of the server's threads. What's more, you can effectively determine the most appropriate settings only by running your application under its expected load and monitoring these performance figures.

15.1.5 Monitoring with JMX

Using the Administration Console to monitor your application can be somewhat tedious, especially because the performance of the system hinges on so many different variables. A nicer way to obtain the desired performance data is to directly access the runtime MBeans that capture the data. This process is described in Chapter 20.

Although you can inspect the runtime MBeans programmatically, you also can use WebLogic's administration tool to reach the data. The following example extracts the current count for the lock entries from all of the EJB locking runtime MBeans:

java weblogic.Admin -url t3://hostname:port -username system -password psst GET -pretty -type EJBLockingRuntime -property LockEntriesCurrentCount

For the full list of MBeans that are available to you, see the API documentation for the weblogic.management.runtime package.