Wednesday, August 14, 2013

Optimize And Tuning WebLogic Server Performance: Tuning JDBC Data Source Connection Pool



High performance WebLogicapplication and system influenced by many factors. One of important factor is good and properly JDBC data sources setting in WebLogicServer domain. Bellow check list can be considered while tuning a JDBC data source:

Setting WebLogic Connection Pool Size

Maximum Capacity

Maximum capacity is number of allocated session in database side for WebLogic Server divide by number of WebLogic server. Example, DBA allocated 500 session for WebLogic server which has 5 server, so this config value is 500/5 = 100

Initial Capacity

Initial capacity mean number of initial session created when data source started. Recommended value for Initial Capacity is equal to Maximum Capacity size database connection pools. In this case WebLogicnever increase the number of connections when it needed by application. Negative impact for this case is increase DB load although only few session use by WebLogic, but if impact in database side is not significant its work well.

Capacity Increment

There is no rule about how many session added when need more connection from application to database, but need to make sure that database can be handle this increment session login. This value should be less than 10 to increase speed of session increment.

Store the Prepared Statements in the Statement Cache

Every statement from an application or EJB, there is a processing cost at application server, database server and network communication between both. This cost can be minimizing using Statement Cacheapproach by cache statements used in application. In the next time WebLogic will be reuses the statement stored in the cache. Using cache will be improve application and system performance by reduce CPU usage in application server side and database server side.

Statement Cache Type

Two algorithm options available for maintaining the prepared statements stored in the statement cache
  • LRU (Least Recently Used)

LRU is default option for Statement Cache Type. In this algorithm, WebLogic Server caches statements used on the connection until the statement cache size is reached. When an application calls prepare Statement, WebLogic Server checks to see if the statement is stored in the statement cache. If cache is available (currently not being used), WebLogic Server returns the cached statement, else (statement is not in the cache, and the cache is full), WebLogic Server determines which existing statement in the cache was the least recently used and replaces that statement in the cache with the new statement.
  • Fixed

In Fixed algorithm, WebLogic Server caches statements used on the connection until the statement cache size is reached. When other statements are used, they are not cached, even though the statement is frequently used. Weakness of this algorithm is WebLogic will cache infrequently used statements.
In most of WebLogic implementation LRU algorithm is preferred because caches only save frequently used statements, so goal of Statement caching is reach.

Statement Cache Size

Statement Cache Size shows the total number of statements to cache for each connection in each instance of the data source. WebLogic Server can reuse statements in the cache without reloading the statements, which can increase server performance. Each connection in the connection pool has its own cache of statements.
Setting the statement cache size = 0 mean that disable statement caching. For example, if a data source with 10 connections deployed on 5servers, and setting Statement Cache Size = 10, Database maximum will be open 500 cursors (10 Conn x 2 server x size 10) on database server for the cached statements.
WebLogic statement caches related to database cursor. The statement may open a cursor in the database. If cache contains too many statements, database connection may reach maximum limit of open cursors for. Avoiding this case can be done in two approaches; increase maximum open cursors allowed for a connection or reduce Statement Cache Size setting.
The conclusion for ideal number of Statement Cache Size is depends on bellow factor:
  • Number of servers which data source deployed
  • Maximum connection for data source
  • Maximum cursor open allowed on database side

Connection Testing Options for a Data Source

The purpose of connection testing is to make sure that the database connections in a data source in a health status healthy. Connection testing should execute periodically in a specific interval time.
There are two methods to testing a data source connection in WebLogic Server:
  • Manual testing
Testing by manually execute connection testing. Usually do while trouble-shoot a data source for a problem. To do manual testing, go to Tab Monitoring, and then tab Testing. Choose data source which want to test and click “Test Data Source”. If there is no problem will be shown a notification Test of SUGI-DB on server MedRecServer was successful.
  • Automatic testing
Configure options on the data source so that WebLogic Server will be periodically test to makes sure that database connections remain healthy.

Test Connections on Reserve

This option will be enables WebLogic Server to test a connection before giving it to a client application. Test connection done by WebLogic using SQL statement based on value of Test Table Name. This connection test adds a small delay in serving the client’s request for a connection from the pool, but ensures that the client receives a reliable connection.

Test Frequency

Test Frequency is time interval in seconds WebLogic Server tests unused connections. Connections that fail the test are closed, re-establish a valid physical connection to database, and test the new connection before returning it to the pool. If the test fails again, the connection is closed. If value set to 0, periodic testing is disabled.

Test Table Name

Test table name is config for SQL Statement which used when testing physical database connections. This value is required when Test Frequency value more than 0 and Test Reserved Connections is enable. The SQL code used to test a connection is depends on database provider. Best practice for this value is fastest SQL statement to the database. Example in ORACLE database, SQL SELECT 1 FROM DUAL is super-fast SQL statement which can be used in this option. Test Table Name must begin with string “SQL “, and then followed by SQL statement.

Maintain Connection to Data Source

Seconds to Trust an Idle Pool Connection

This option can minimize the impact of testing reserved connections. The appropriate value depends on system environment. Too low setting will be poor because during heavy traffic connection testing is give more loads to database, but if value is set too high will be reduce effectiveness of connection testing itself. Default value for this config is 10.

Shrink Frequency

The number of seconds to wait before shrinking a connection pool that has incrementally increased to meet demand. Shrinking means that WebLogic drop some connections from the data source when a peak usage period has ended, freeing up WebLogic Server and DBMS resources. When set to 0, shrinking is disabled.

Init SQL

Init SQL statement will be execute while WebLogic initialize newly created physical database connections in a data source. Default value is blank, but for specific purposes, this value can be set based on Database Provider Script. For example in Oracle, you can set Init SQL with alter session. Init SQL must begin with string “SQL “, and then followed by SQL statement.

Connection Creation Retry Frequency

This value represent interval in seconds between attempts to establish connections to the database. If the database is unavailable when the data source is created, WebLogic Server will attempt to create connections in the pool again after the number of seconds you specify, and will continue to attempt to create the connections until it succeeds. When set to 0, connection retry is disabled. Recommended to set this value to a desired seconds. The exact value need suggest from Database Administrator because related to database environment behavior.

Login Delay

Login delay means interval in seconds before creating each physical database connection. Recommended to set the value with 0 (no delay), but if database cannot handle multiple connection requests it means that need delay during creating database connection.

Inactive Connection Timeout

Inactive connection in WebLogic Server will be releases and back into the connection pool if this value set more than 0. Recommendation for this config is specific value in seconds, so if found leaked connections that were not correctly closed by the application will be handle via this feature.

Maximum Waiting for Connection

The maximum number of connection requests that can concurrently block threads while waiting to reserve a connection from the data source’s connection pool. Leave this setting with default value (Maximum value).

Connection Reserve Timeout

When an application requests a connection from a data source, if all connections in the data source are in use and if the data source has expanded to its maximum capacity, the application will get a Connection Unavailable SQL Exception. Configure the Connection Reserve Timeout value (in seconds) will be affected that connection requests will wait for a connection to become available. After the Connection Reserve Timeout has expired, if no connection becomes available, the request will fail and the application will get a PoolLimitSQLException exception.  To set connection request will timeout immediately, set value = -1. To set a connection request will wait until connection available, set value = 0. Fair value is 10 seconds by default.

Statement Timeout

Statement currently being executed will time out if consume execution time more than this setting. Set timeout via this setting will impacted to every statement in this data source. It recommended that setting timeout in application adapter depends on every statement. But also recommended to set this value also with upper limit of timeout define by business process to anticipate application adapter with no timeout.

Ignore In-Use Connections

This option enables the data source to be shutdown even if connections obtained from the pool are still in use. Enable or disable this option depends on policy in the system, but default value is enable.

Pinned-To-Thread

PinnedToThread is an option that can improve performance by enabling execute threads to keep a pooled database connection even after the application closes the logical connection. Default value is disable. This option can be enable and will be useful if implemented with correct calculation between WebLogicresource and database resource because related to reserved thread.

Remove Infected Connections Enabled

Default and safe value is enable which means the physical connection is not returned to the connection pool after the application closes the logical connection. Instead, the physical connection is closed and recreated.

Saturday, August 10, 2013

WebLogic – Database Connection Pinned-to-Thread

Using Pinned-To-Thread Property to Increase Performance

To minimize the time it takes for an application to reserve a database connection from a data source and to eliminate contention between threads for a database connection, you can add the Pinned-To-Thread property in the connection Properties list for the data source, and set its value to true.
When Pinned-To-Thread is enabled, WebLogic Server pins a database connection from the data source to an execution thread the first time an application uses the thread to reserve a connection. When the application finishes using the connection and calls connection.close(), which otherwise returns the connection to the data source, WebLogic Server keeps the connection with the execute thread and does not return it to the data source. When an application subsequently requests a connection using the same execute thread, WebLogic Server provides the connection already reserved by the thread. There is no locking contention on the data source that occurs when multiple threads attempt to reserve a connection at the same time and there is no contention for threads that attempt to reserve the same connection from a limited number of database connections.
 
 
Note:
In this release, the Pinned-To-Thread feature does not work with multi data sources, Oracle RAC, and IdentityPool. These features rely on the ability to return a connection to the connection pool and reacquire it if there is a connection failure or connection identity does not match.
 
 

Changes to Connection Pool Administration Operations When PinnedToThread is Enabled

Because the nature of connection pooling behavior is changed when PinnedToThread is enabled, some connection pool attributes or features behave differently or are disabled to suit the behavior change:
  • Maximum Capacity is ignored. The number of connections in a connection pool equals the greater of either the initial capacity or the number of connections reserved from the connection pool.
  • Shrinking does not apply to connection pools with PinnedToThread enabled because connections are never returned to the connection pool. Effectively, they are always reserved.
  • When you Reset a connection pool, the reset connections from the connection pool are marked as Test Needed. The next time each connection is reserved, WebLogic Server tests the connection and recreates it if necessary. Connections are not tested synchronously when you reset the connection pool. This feature requires that Test Connections on Reserve is enabled and a Test Table Name or query is specified.

Additional Database Resource Costs When PinnedToThread is Enabled

When PinnedToThread is enabled, the maximum capacity of the connection pool (maximum number of database connections created in the connection pool) becomes the number of execute threads used to request a connection multiplied by the number of concurrent connections each thread reserves. This may exceed the Maximum Capacity specified for the connection pool. You may need to consider this larger number of connections in your system design and ensure that your database allows for additional associated resources, such as open cursors.
Also note that connections are never returned to the connection pool, which means that the connection pool can never shrink to reduce the number of connections and associated resources in use. You can minimize this cost by setting an additional driver parameter onePinnedConnectionOnly. WhenonePinnedConnectionOnly=true, only the first connection requested is pinned to the thread. Any additional connections required by the thread are taken from and returned to the connection pool as needed. Set onePinnedConnectionOnly using the Properties attribute, for example:
Properties="PinnedToThread=true;onePinnedConnectionOnly=true;user=examples"
If your system can handle the additional resource requirements, Oracle recommends that you use the PinnedToThread option to increase performance.
If your system cannot handle the additional resource requirements or if you see database resource errors after enabling PinnedToThread, Oracle recommends not using PinnedToThread.

Start Weblogic without user and password

When you want to launch your weblogic server at production, you need to use the startWeblogic.sh in your bin folder.
But each time it’s launched it’s asking you to authenticate yourself by typing a user and password. It’s a problem when you want to automate the execution of your server…
To avoid this problem all you need to do is creating a file named boot.properties“ and insert into the two following lines :
username=<yourUserName>
password=<yourPassword>
this file must be placed in each server security folder :$WLS_HOME/user_projects/domains/<domainName/servers/<serverName>/security

Then, start server, it shouldn’t ask you for anythin, then reopen your boot.properties file, password and username should be automatically encrypted !

Avoid java out of memory with Weblogic

This is common exception always exists if you install weblogic and doesn’t extend memory arguments in server.
The file “setDomainEnv.sh” in $WLS_HOME/user_projects/domains/<domainName>/bin/ has configuration of domain.
If you edit this file it will have the following default values.
MEM_ARGS=”-Xms256m -Xmx512m”
export MEM_ARGS
MEM_PERM_SIZE=”-XX:PermSize=48m”
export MEM_PERM_SIZE
MEM_MAX_PERM_SIZE=”-XX:MaxPermSize=128m”
export MEM_MAX_PERM_SIZE
you should modify the MEM_ARGS java memory value depending of your server, here is the suggested to increase.
MEM_ARGS=”-Xms2024m -Xmx3036m”
export MEM_ARGS
MEM_PERM_SIZE=”-XX:PermSize=128m”
export MEM_PERM_SIZE
MEM_MAX_PERM_SIZE=”-XX:MaxPermSize=512m”
export MEM_MAX_PERM_SIZE
Note : choosing the values depend on you server hardware.

Redirecting JVM Output to Server Logging Destinations

The JVM in which a WebLogic Server instance runs, sends messages to standard error and standard out. Server as well as application code write directly to these streams instead of using the logging mechanism. Through a configuration option, you can redirect the JVM output to all the registered log destinations, like the server terminal console and log file. When enabled, a log entry appears as a message of NOTICE severity. Redirecting the JVM output does not capture output from native code, for example thread dumps from the JVM.
1)To redirect the JVM standard out messages:
When you start the Administration Server, include the following Java option in the weblogic.Server command:
-Dweblogic.log.RedirectStdoutToServerLogEnabled=true
2)Redirecting Stdout to Server Logging Destinations
C:\>java weblogic.WLST
wls:/offline> connect(‘username’,'password’)
wls:/mydomain/serverConfig> edit()
wls:/mydomain/edit> startEdit()
wls:/mydomain/edit !> cd(“Servers/myserver/Log/myserver”)
wls:/mydomain/edit/Servers/myserver/Log/myserver !> cmo.setRedirectStdoutToServerLogEnabled(true)
wls:/mydomain/edit/Servers/myserver/Log/myserver !> save()
wls:/mydomain/edit/Servers/myserver/Log/myserver !> activate()

Thread Dump in WebLogic Server

Thread dumps are very useful to analyze and troubleshoot performance related issues such as server hang, deadlocks, slow running, idle or stuck applications,  etc.
Different ways to take thread dumps in WebLogic Server
Always prefer Operating system(OS) commands rather instead of Admin Console or Java Classes, because if the console is hanging, users won’t be able to connect to it to issue thread dumps.
1. OS Commands for Thread Dumps
i) On Windows,
<ctrl>+<break> — the thread dumps are generated in the server stdout
ii) On Solaris / Linux
first identify the process ID (pid) using   ps -ef | grep java, then run   kill -3 <pid> td_filename 2>&1
2. Using weblogic.WLST ( work only from WLS 9.x onwards)
First set CLASSPATH using setDomain.cmd or setDomain.sh (wlst.sh /wlst.cmd will also do in path C:\Oracle\Middleware\wlserver_12.1\common\bin ). Then run below command
java weblogic.WLST ThreadDumps.py
save below code in ThreadDumps.py file:
connect(“<username>”,”<password>”,”t3://<url>:<port>”)
cd(‘Servers’)
cd(‘AdminServer’)
threadDump()
disconnect()
exit()
threadDump()
The thread dumps get stored in the location from where you run it.
3. From Weblogic Administration Console
navigating to Server -> <server_name> -> Monitoring -> Dump threads stack.
4. From the JRockit Command line
jrcmd <pid> print_threads

How to enable the Heapdump in Weblogic Server

In order to get Heap-dump to be generated in Weblogic server  we have to set the following parameters in server start-up argument
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=C:\Heapdump