You are on page 1of 3

IBM - Determining which Java thread is consuming CPU cycles on Solaris systems

Page 1 of 3

Determining which Java thread is consuming CPU cycles on


Solaris systems

Technote (troubleshooting)
Problem(Abstract)
This technotes shows how to determine which thread(s) is consuming the
CPU cycles when a Java process has high CPU usage for an extended
period of time on a Solaris system.
This technote uses the data gathered from technote # 1115625, titled:
"MustGather: 100% CPU Usage on Solaris". For educational purposes, the
data that is used in this particular example is included at the end of the
document.
Cause
If the IBM WebSphere Application Server causes a spike in CPU
utilization for a period of greater than several minutes, this technote will
show you techniques to assist you in identifying the Java code that is
causing the problem.
Resolving the problem
Using the files attached to this technote, the following three steps
demonstrate how to find the problem thread and the corresponding Java
code.
1. Analyze the prstat information to determine the LWPID (light
weight process) that is consuming a significant amount of CPU.
The following prstat was generated with the following command:
prstat -mvL

Use of different prstat parameters will give different output but can be used in a similar
fashion as described below.
PID USRNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG
PROC/LWPID
16365 root
36 53 0.0 0.0 0.0 0.0 18 0.0
0 178 13K
prstat/1
16310 root
46 0.0 0.1 0.0 0.0 4.3 45 5.0
0 58
8
0
java/30
16310 root
5.8 0.2 0.0 0.0 0.0 0.0 94 0.0
0 11 56
java/11
16310 root
2.6 0.1 0.0 0.0 0.0 0.0 97 0.0
0
6 18
java/5
5158 root
0.5 0.1 0.0 0.0 0.0 0.0 99 0.0 49 30 1K
19 .netscape.bi/1
16310 root
0.3 0.0 0.0 0.0 0.0 0.0 100 0.0
2
1 11
java/23
16310 root
0.3 0.0 0.0 0.0 0.0 0.0 100 0.0
5
1
7
java/13
16310 root
0.3 0.0 0.0 0.0 0.0 0.0 100 0.0
1
0
2
java/3
16310 root
0.3 0.0 0.0 0.0 0.0 0.0 100 0.0
0
0
2
java/17
16310 root
0.2 0.0 0.0 0.0 0.0 0.0 100 0.0
4
2
6

http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21162381

0
0

0
0
0
0
0

3/04/2009

IBM - Determining which Java thread is consuming CPU cycles on Solaris systems

java/16
16310 root
0.2 0.0 0.0 0.0
java/18
16310 root
0.1 0.1 0.0 0.0
java/1
16310 root
0.2 0.0 0.0 0.0
java/28
16310 root
0.2 0.0 0.0 0.0
java/19
290 root
0.1 0.1 0.0 0.0
Xsun/1
Total: 90 processes, 233 lwps,

0.0 0.0 100 0.0

Page 2 of 3

99 0.5

99

87

0.0 0.0 100 0.0

12

0.0 0.0 100 0.0

0.0 0.0 100 0.0

21

0 132

0.0 0.0

load averages: 1.00, 0.84, 0.66

In the preceding prstat output, the third and fourth columns provide the amount of time the
process has spent in user and system mode. Ignoring the prstat command the most CPU time
is being consumed by LWPID=30.
2. Use the LWPID of 30 to find the thread in the pstack output. The
following is a snippet from the pstack output:
--------------------- lwp# 30 / thread# 50 ---------------------fe753aa8 ???????? (0, 5265c00, fa, 43a85d79, fe74e170,
e04803fc)
fb1025b4 ???????? (f60748, e04819b8, fe74e170, f60748,
fe763ca0, 16)
.....
fe505288 _start
(fe74e170, e1ef5d10, 0, 5, 1,
fe401000) + 20
ff36b6e0 _thread_start (f25078, 0, 0, 0, 0, 0) + 40
------------------------------------------------------------------Look for _thread_start at the bottom of the thread stack. The first number inside
the ( ) is the threadID or "tid". In this case the tid, in hex, is f25078.
3. Finally, search for f25078 in the Java thread dump output. This
output, usually gathered from issuing a kill -3 on the process in
question, is located in the native_stdout.log file. In the
following snippet of the thread dump you will see the tid of f25078:
"Servlet.Engine.Transports : 0" daemon prio=5 tid=0xf25078 nid=0x32 runnable
[0xe0480000..0xe04819d8]
at java.lang.System.currentTimeMillis(Native Method)
at java.util.Date.<init>(Date.java:161)
at org.apache.jsp._wtime._jspService(_wtime.java:96)
at com.ibm.ws.webcontainer.jsp.runtime.HttpJspBase.service
(HttpJspBase.java:89)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
com.ibm.ws.webcontainer.jsp.servlet.JspServlet$JspServletWrapper.service
...
Alternatively, you may use the thread# 50 from the pstack output (from the command run on the system
core file "/usr/proc/bin/pstack [core] >pstack.out") to correlate the prstat and the thread
dump. This is the decimal representation of the thread's "nid" in the thread dump. Using the preceding
example, since decimal 50 is 0x32, thread# 50 can be used to find the thread by locating "nid=0x32"in the
thread dumps. This is the only method that works if the application is using the alternate threading libraries.
To determine which library is being used, please refer to technote Determining which library is currently in

http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21162381

3/04/2009

IBM - Determining which Java thread is consuming CPU cycles on Solaris systems

Page 3 of 3

use.
Analyzing the thread stack for this tid will help lead you to the code
causing the high CPU usage. In this case the issue most likely comes from
the JSP:
_wtime._jspService(_wtime.java:96)
Related information
MustGather: 100% CPU Usage on Solaris
Changing to the alternate one-to-one threading library

Cross Reference information


Segment

Product

Component Platform

Version

Edition

Runtimes
Application
Java SDK
for Java
Servers
Technology

Copyright and trademark information


IBM, the IBM logo and ibm.com are trademarks of International Business
Machines Corp., registered in many jurisdictions worldwide. Other product
and service names might be trademarks of IBM or other companies. A
current list of IBM trademarks is available on the Web at "Copyright and
trademark information" at www.ibm.com/legal/copytrade.shtml.

http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21162381

3/04/2009

You might also like