WSO2 ESB is an asynchronous high performing messaging engine which uses Java NIO technology for its internal implementations. You can find more information about the implementation details about the WSO2 ESB’s high performing http transport known as Pass-Through Transport (PTT) from the links given below.
From this tutorial, I am going to discuss about various threads created when you start the ESB and start processing requests with that. This would help you to troubleshoot critical ESB server issues with the usage of a thread dump. You can monitor the threads created by using a monitoring tool like Jconsole or java mission control (java 1.7.40 upwards). Given below is a list of important threads and their stack traces from an active ESB server.
PassThroughHTTPSSender ( 1 Thread )
This thread is the reactor(acceptor) thread which is responsible for handling new connection requests to https endpoints from ESB. When there is a <send/> or <call/> mediator within the mediation flow to and https endpoint, it will eventually executes this thread. This thread needs to be running all the time during the server is up and running (unless https transport is disabled).
Name: PassThroughHTTPSSender
State: RUNNABLE
Total blocked: 1 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@2bef6f1
- locked java.util.Collections$UnmodifiableSet@578b8a47
- locked sun.nio.ch.KQueueSelectorImpl@1a8f39bb
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpSender$2.run(PassThroughHttpSender.java:195)
java.lang.Thread.run(Thread.java:744)
HTTPS-Sender I/O dispatcher-1 (4 Threads in 4 Core)
This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to sending messages to https endpoints are handled by these threads. These threads need to be running all the time during the server is up and running (unless https transport is disabled).
Name: HTTPS-Sender I/O dispatcher-1
State: RUNNABLE
Total blocked: 1 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@25484cc9
- locked java.util.Collections$UnmodifiableSet@3f6342dd
- locked sun.nio.ch.KQueueSelectorImpl@66ac838c
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)
PassThroughHTTPSender ( 1 Thread )
This thread is the reactor(acceptor) thread which is responsible for handling new connection requests to http endpoints from ESB. When there is a <send/> or <call/> mediator within the mediation flow to http endpoint, it will eventually executes this thread and create a new socket channel. This thread needs to be running all the time during the server is up and running (unless http transport is disabled).
Name: PassThroughHTTPSender
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@c15e5c9
- locked java.util.Collections$UnmodifiableSet@7e49d31e
- locked sun.nio.ch.KQueueSelectorImpl@9052336
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpSender$2.run(PassThroughHttpSender.java:195)
java.lang.Thread.run(Thread.java:744)
HTTP-Sender I/O dispatcher-1 (4 Threads in 4 Core)
This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to sending messages to http endpoints are handled by these threads. These threads need to be running all the time during the server is up and running (unless http transport is disabled).
Name: HTTP-Sender I/O dispatcher-1
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@327d74e9
- locked java.util.Collections$UnmodifiableSet@460208f5
- locked sun.nio.ch.KQueueSelectorImpl@33f55d67
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)
PassThroughHTTPSListener (1 Thread)
This thread is the reactor(acceptor) thread which is responsible for handling new connection requests from https requests from clients. When there is a https request towards ESB, it will eventually executes this thread. When this request is received, it will create a new socket channel and register with a selection key and a channel. Further IO events are handled by the dispatcher threads. This thread needs to be running all the time during the server is up and running (unless https transport is disabled).
Name: PassThroughHTTPSListener
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@50cff7e
- locked java.util.Collections$UnmodifiableSet@271d259c
- locked sun.nio.ch.KQueueSelectorImpl@2282db4d
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpListener$4.run(PassThroughHttpListener.java:251)
java.lang.Thread.run(Thread.java:744)
HTTPS-Listener I/O dispatcher-1(4 Threads in 4 Core)
This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to receiving messages from https requests are handled by these threads. These threads need to be running all the time during the server is up and running (unless https transport is disabled).
Name: HTTPS-Listener I/O dispatcher-1
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@48c97759
- locked java.util.Collections$UnmodifiableSet@8ac860c
- locked sun.nio.ch.KQueueSelectorImpl@ff4fe7c
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)
PassThroughHTTPListener (1 Thread)
This thread is the reactor(acceptor) thread which is responsible for handling new connection requests from http requests from clients. When there is a http request towards ESB, it will eventually executes this thread. When this request is received, it will create a new socket channel and register with a selection key and a channel. Further IO events are handled by the dispatcher threads. This thread needs to be running all the time during the server is up and running (unless http transport is disabled).
Name: PassThroughHTTPListener
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@1f6438be
- locked java.util.Collections$UnmodifiableSet@152987f9
- locked sun.nio.ch.KQueueSelectorImpl@11d60796
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpListener$4.run(PassThroughHttpListener.java:251)
java.lang.Thread.run(Thread.java:744)
HTTP-Listener I/O dispatcher-1(4 Threads in 4 Core)
This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to receiving messages from http requests are handled by these threads. These threads need to be running all the time during the server is up and running (unless http transport is disabled).
Name: HTTP-Listener I/O dispatcher-1
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@34095061
- locked java.util.Collections$UnmodifiableSet@60e8e6e5
- locked sun.nio.ch.KQueueSelectorImpl@5fbe8e73
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)
PassThroughMessageProcessor-1(400)
This is the worker thread which is responsible for processing requests coming in to the ESB when the request message is received. Passthrough transport will start these worker threads once the HTTP headers are received from the clients. Then these threads will run independently from PTT threads which are described earlier. There will be n number of threads created where n is the worker_pool_size_core parameter defined in the passthrough-http.properties file.
Name: PassThroughMessageProcessor-1
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7050afc9
Total blocked: 0 Total waited: 1
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
SynapseWorker-1(20)
This is the worker thread which is responsible for handling iterate/clone mediator executions. There will be a separate thread pool to handle such executions. Number of threads in this thread pool can be configured in synapse.properties file with the parameter synapse.threads.core.
Name: SynapseWorker-1
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22d3b7d9
Total blocked: 1 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Following are threads related to embedded tomcat instance run within the ESB. It will be used to serve requests coming in to ESB management console, admin services. You can configure the thread pool and other parameters from ESB_HOME/repository/conf/tomcat/catalina-server.xml file.
NioBlockingSelector.BlockPoller-1(2)
Name: NioBlockingSelector.BlockPoller-1
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@5d630f69
- locked java.util.Collections$UnmodifiableSet@59cdfa64
- locked sun.nio.ch.KQueueSelectorImpl@510b6d29
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:327)
http-nio-9763-ClientPoller-0(2)
Name: http-nio-9763-ClientPoller-0
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@5f185036
- locked java.util.Collections$UnmodifiableSet@1b5f14d
- locked sun.nio.ch.KQueueSelectorImpl@5b8b369f
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1146)
java.lang.Thread.run(Thread.java:744)
http-nio-9763-Acceptor-0(2)
Name: http-nio-9763-Acceptor-0
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked java.lang.Object@525f8df5
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:787)
java.lang.Thread.run(Thread.java:744)
http-nio-9443-ClientPoller-0(2)
Name: http-nio-9443-ClientPoller-0
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@3d8e1f2f
- locked java.util.Collections$UnmodifiableSet@2f3ecb19
- locked sun.nio.ch.KQueueSelectorImpl@113dc8a9
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1146)
java.lang.Thread.run(Thread.java:744)
http-nio-9443-Acceptor-0(2)
Name: http-nio-9443-Acceptor-0
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked java.lang.Object@38fcf802
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:787)
java.lang.Thread.run(Thread.java:744)
http-nio-9443-exec-1(50)
Name: http-nio-9443-exec-1
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a4e4fb2
Total blocked: 1 Total waited: 6
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thanks for the post. Got a good idea about threads running in the WSO2 ESB.
ReplyDeletedigital transformation services thanks for this information
ReplyDelete