Introduction to Thread Dump - Java Thread Dump Analyzer

1. What is a Thread Dump

A Thread Dump is a snapshot of all threads in a Java application at a specific moment. It captures the current state information of all threads in the Java Virtual Machine (JVM). This is an important diagnostic tool for analyzing the operational status of Java applications, especially when encountering performance issues or deadlock situations.

A Thread Dump typically includes the following information:

Thread ID and name
Thread state (such as RUNNABLE, BLOCKED, WAITING, TIMED_WAITING, etc.)
Thread priority
Whether the thread is a daemon thread
Thread execution stack trace, showing the code path being executed
Information about locks (monitors) held by the thread
Information about locks the thread is waiting to acquire
Thread creation time and CPU usage time

2. How to Export Thread Dumps on Linux Systems

There are several methods to generate Thread Dumps for Java applications on Linux systems:

Using the jstack Tool

jstack is a command-line tool included with the JDK, specifically designed for generating thread dumps:

# First find the Java process PID
jps

# Use jstack to generate a thread dump
jstack <pid> > thread_dump.txt

If you need to capture multiple thread dumps to analyze an issue:

# Generate a thread dump every 5 seconds, 3 times
for i in 1 2 3; do jstack <pid> > thread_dump_$i.txt; sleep 5; done

Using the kill Command to Send a Signal

You can also generate a thread dump by sending a SIGQUIT signal to the Java process:

kill -3 <pid>

This will output the thread dump to the application's standard output (usually the console or log file).

Using the jcmd Tool

jcmd is another JDK tool with more comprehensive functionality:

jcmd <pid> Thread.print > thread_dump.txt

3. How to Use Thread Dumps to Solve Problems

Solution Approaches

Scenario 1: High CPU, High Load, Slow Response

Approach:

Capture multiple dumps during a single request process
Compare the runnable threads across multiple dump files. If there are significant changes in the methods being executed, the situation is relatively normal. If threads are executing the same method repeatedly, there may be a problem.

Scenario 2: Finding Threads Consuming the Most CPU

Approach:

Use the command: top -H -p pid (where pid is the process ID of the system being tested) to identify the thread ID causing high CPU usage.
The thread ID found by the top command corresponds to the nid in the thread dump information, except one is in decimal and the other in hexadecimal.
In the thread dump, locate the corresponding thread stack information based on the thread ID found by the top command.
Capture multiple dumps and compare the method calls.

Scenario 3: Low CPU Usage but Long Response Times

Approach:

Capture a dump and check if many threads are stuck in I/O, database operations, etc., to identify the bottleneck.
1. Waiting on condition: Waiting for a resource or condition to wake itself up. For example, a thread is sleeping or waiting due to busy network I/O.
2. Blocked: Waiting for monitor entry.

Scenario 4: Requests Not Responding

Approach:

Capture multiple dumps and check if all runnable threads are continuously executing the same method. If so, the system is likely locked!

Case Studies

Case Study 1: Deadlock Analysis

Thread Dump characteristics of a deadlock issue:

Found one Java-level deadlock:
=============================
"Thread-1":
  waiting to lock monitor 0x000000001d840878 (object 0x00000000eb4231d0, a java.lang.Object),
  which is held by "Thread-0"
"Thread-0":
  waiting to lock monitor 0x000000001d843208 (object 0x00000000eb4231e0, a java.lang.Object),
  which is held by "Thread-1"

Case Study 2: Connection Pool Exhaustion Analysis

When database or HTTP connection pools are exhausted, multiple threads will be in a WAITING state, waiting for connection resources:

"http-nio-8080-exec-10" #49 daemon prio=5 os_prio=0 tid=0x00007f3b14337000 nid=0x7f8c WAITING [0x00007f3aa9bfe000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c46ff608> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:478)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
    at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)

Case Study 3: Infinite Loop Causing High CPU Usage

Characteristics of high CPU usage caused by infinite loops or long calculations:

"high-cpu-thread" #23 prio=5 os_prio=0 tid=0x00007fa3c4173800 nid=0x1234 runnable [0x00007fa3bc5fc000]
   java.lang.Thread.State: RUNNABLE
    at com.example.HighCpuUsage.computeEndlessly(HighCpuUsage.java:42)
    at com.example.HighCpuUsage.process(HighCpuUsage.java:29)
    at com.example.HighCpuUsage.run(HighCpuUsage.java:18)

4. Thread Dump Analysis Best Practices

Regular Sampling: Capture multiple Thread Dumps at intervals of a few seconds when problems occur
Comparative Analysis: Compare multiple Thread Dumps to observe changes in thread states
Focus on Hotspots: Pay special attention to threads in the RUNNABLE state, especially those that remain in the same method for a long time
State Distribution: Analyze the distribution ratio of threads in different states to assess the overall health of the system
Complementary Tools: Use system tools such as top, vmstat, etc., to gain a comprehensive understanding of system status
Automated Analysis: Utilize professional tools like theadanalyzer.online, FastThread, IBM Thread Analyzer. to assist with analysis

By regularly capturing Thread Dumps and performing comparative analysis, you can more effectively diagnose intermittent issues and performance bottlenecks, thereby improving the stability and performance of Java applications.