Skip to main content

Understanding Threads created in WSO2 ESB

WSO2 ESB is an asynchronous high performing messaging engine which uses Java NIO technology for its internal implementations. You can find more information about the implementation details about the WSO2 ESB’s high performing http transport known as Pass-Through Transport (PTT) from the links given below.

[1] http://soatutorials.blogspot.com/2015/05/understanding-wso2-esb-pass-through.html

[2] http://wso2.com/library/articles/2013/12/demystifying-wso2-esb-pass-through-transport-part-i/

From this tutorial, I am going to discuss about various threads created when you start the ESB and start processing requests with that. This would help you to troubleshoot critical ESB server issues with the usage of a thread dump. You can monitor the threads created by using a monitoring tool like Jconsole or java mission control (java 1.7.40 upwards). Given below is a list of important threads and their stack traces from an active ESB server. 



PassThroughHTTPSSender ( 1 Thread )

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests to https endpoints from ESB. When there is a <send/> or <call/> mediator within the mediation flow to and https endpoint, it will eventually executes this thread. This thread needs to be running all the time during the server is up and running (unless https transport is disabled).

Name: PassThroughHTTPSSender
State: RUNNABLE
Total blocked: 1  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@2bef6f1
   - locked java.util.Collections$UnmodifiableSet@578b8a47
   - locked sun.nio.ch.KQueueSelectorImpl@1a8f39bb
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpSender$2.run(PassThroughHttpSender.java:195)
java.lang.Thread.run(Thread.java:744)


HTTPS-Sender I/O dispatcher-1 (4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to sending messages to https endpoints are handled by these threads. These threads need to be running all the time during the server is up and running (unless https transport is disabled).


Name: HTTPS-Sender I/O dispatcher-1
State: RUNNABLE
Total blocked: 1  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@25484cc9
   - locked java.util.Collections$UnmodifiableSet@3f6342dd
   - locked sun.nio.ch.KQueueSelectorImpl@66ac838c
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)


PassThroughHTTPSender ( 1 Thread )

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests to http endpoints from ESB. When there is a <send/> or <call/> mediator within the mediation flow to http endpoint, it will eventually executes this thread and create a new socket channel. This thread needs to be running all the time during the server is up and running (unless http transport is disabled).

Name: PassThroughHTTPSender
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@c15e5c9
   - locked java.util.Collections$UnmodifiableSet@7e49d31e
   - locked sun.nio.ch.KQueueSelectorImpl@9052336
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpSender$2.run(PassThroughHttpSender.java:195)
java.lang.Thread.run(Thread.java:744)


HTTP-Sender I/O dispatcher-1 (4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to sending messages to http endpoints are handled by these threads. These threads need to be running all the time during the server is up and running (unless http transport is disabled).

Name: HTTP-Sender I/O dispatcher-1
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@327d74e9
   - locked java.util.Collections$UnmodifiableSet@460208f5
   - locked sun.nio.ch.KQueueSelectorImpl@33f55d67
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)


PassThroughHTTPSListener (1 Thread)

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests from https requests from clients. When there is a https request towards ESB, it will eventually executes this thread. When this request is received, it will create a new socket channel and register with a selection key and a channel. Further IO events are handled by the dispatcher threads. This thread needs to be running all the time during the server is up and running (unless https transport is disabled).

Name: PassThroughHTTPSListener
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@50cff7e
   - locked java.util.Collections$UnmodifiableSet@271d259c
   - locked sun.nio.ch.KQueueSelectorImpl@2282db4d
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpListener$4.run(PassThroughHttpListener.java:251)
java.lang.Thread.run(Thread.java:744)




HTTPS-Listener I/O dispatcher-1(4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to receiving messages from https requests are handled by these threads. These threads need to be running all the time during the server is up and running (unless https transport is disabled).

Name: HTTPS-Listener I/O dispatcher-1
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@48c97759
   - locked java.util.Collections$UnmodifiableSet@8ac860c
   - locked sun.nio.ch.KQueueSelectorImpl@ff4fe7c
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)


PassThroughHTTPListener (1 Thread)

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests from http requests from clients. When there is a http request towards ESB, it will eventually executes this thread. When this request is received, it will create a new socket channel and register with a selection key and a channel. Further IO events are handled by the dispatcher threads. This thread needs to be running all the time during the server is up and running (unless http transport is disabled).

Name: PassThroughHTTPListener
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@1f6438be
   - locked java.util.Collections$UnmodifiableSet@152987f9
   - locked sun.nio.ch.KQueueSelectorImpl@11d60796
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpListener$4.run(PassThroughHttpListener.java:251)
java.lang.Thread.run(Thread.java:744)



HTTP-Listener I/O dispatcher-1(4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to receiving messages from http requests are handled by these threads. These threads need to be running all the time during the server is up and running (unless http transport is disabled).

Name: HTTP-Listener I/O dispatcher-1
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@34095061
   - locked java.util.Collections$UnmodifiableSet@60e8e6e5
   - locked sun.nio.ch.KQueueSelectorImpl@5fbe8e73
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)


PassThroughMessageProcessor-1(400)

This is the worker thread which is responsible for processing requests coming in to the ESB when the request message is received. Passthrough transport will start these worker threads once the HTTP headers are received from the clients. Then these threads will run independently from PTT threads which are described earlier. There will be n number of threads created where n is the worker_pool_size_core parameter defined in the passthrough-http.properties file. 

Name: PassThroughMessageProcessor-1
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7050afc9
Total blocked: 0  Total waited: 1

Stack trace: 
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)


SynapseWorker-1(20)

This is the worker thread which is responsible for handling iterate/clone mediator executions. There will be a separate thread pool to handle such executions. Number of threads in this thread pool can be configured in synapse.properties file with the parameter synapse.threads.core. 

Name: SynapseWorker-1
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22d3b7d9
Total blocked: 1  Total waited: 2

Stack trace: 
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)



Following are threads related to embedded tomcat instance run within the ESB. It will be used to serve requests coming in to ESB management console, admin services. You can configure the thread pool and other parameters from ESB_HOME/repository/conf/tomcat/catalina-server.xml file.



NioBlockingSelector.BlockPoller-1(2)

Name: NioBlockingSelector.BlockPoller-1
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@5d630f69
   - locked java.util.Collections$UnmodifiableSet@59cdfa64
   - locked sun.nio.ch.KQueueSelectorImpl@510b6d29
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:327)


http-nio-9763-ClientPoller-0(2)

Name: http-nio-9763-ClientPoller-0
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@5f185036
   - locked java.util.Collections$UnmodifiableSet@1b5f14d
   - locked sun.nio.ch.KQueueSelectorImpl@5b8b369f
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1146)
java.lang.Thread.run(Thread.java:744)


http-nio-9763-Acceptor-0(2)

Name: http-nio-9763-Acceptor-0
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
   - locked java.lang.Object@525f8df5
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:787)
java.lang.Thread.run(Thread.java:744)


http-nio-9443-ClientPoller-0(2)

Name: http-nio-9443-ClientPoller-0
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@3d8e1f2f
   - locked java.util.Collections$UnmodifiableSet@2f3ecb19
   - locked sun.nio.ch.KQueueSelectorImpl@113dc8a9
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1146)
java.lang.Thread.run(Thread.java:744)


http-nio-9443-Acceptor-0(2)

Name: http-nio-9443-Acceptor-0
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
   - locked java.lang.Object@38fcf802
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:787)
java.lang.Thread.run(Thread.java:744)


http-nio-9443-exec-1(50)

Name: http-nio-9443-exec-1
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a4e4fb2
Total blocked: 1  Total waited: 6

Stack trace: 
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)





Labels:

Comments

Post a Comment

Popular posts from this blog

WSO2 ESB tuning performance with threads

I have written several blog posts explaining the internal behavior of the ESB and the threads created inside ESB. With this post, I am talking about the effect of threads in the WSO2 ESB and how to tune up threads for optimal performance. You can refer [1] and [2] to understand the threads created within the ESB. [1] http://soatutorials.blogspot.com/2015/05/understanding-threads-created-in-wso2.html [2] http://wso2.com/library/articles/2012/03/importance-performance-wso2-esb-handles-nonobvious/ Within this blog post, I am discussing about the "worker threads" which are used for processing the data within the WSO2 ESB. There are 2 types of worker threads created when you start sending the requests to the server 1) Server Worker/Client Worker Threads 2) Mediator Worker (Synapse-Worker) Threads Server Worker/Client Worker Threads These set of threads will be used to process all the requests/responses coming to the ESB server. ServerWorker Threads will be used to pr
4 comments
Read more

How puppet works in your IT infrstructure

What is Puppet? Puppet is IT automation software that helps system administrators manage infrastructure throughout its lifecycle, from provisioning and configuration to orchestration and reporting. Using Puppet, you can easily automate repetitive tasks, quickly deploy critical applications, and proactively manage change, scaling from 10s of servers to 1000s, on-premise or in the cloud. How the puppet works? It works like this..Puppet agent is a daemon that runs on all the client servers(the servers where you require some configuration, or the servers which are going to be managed using puppet.) All the clients which are to be managed will have puppet agent installed on them, and are called nodes in puppet. Puppet Master: This machine contains all the configuration for different hosts. Puppet master will run as a daemon on this master server. Puppet Agent: This is the daemon that will run on all the servers, which are to be managed using p
7 comments
Read more

How to configure timeouts in WSO2 ESB to get rid of client timeout errors

WSO2 ESB has defined some configuration parameters which controls the timeout of a particular request which is going out of ESB. In a particular  scneario, your client sends a request to ESB, and then ESB sends a request to another endpoint to serve the request. CLIENT->WSO2 ESB->BACKEND The reason for clients getting timeout is that ESB timeout is larger than client's timeout. This can be solved by either increasing the timeout at client side or by decreasing the timeout in ESB side. In any of the case, you can control the timeout in ESB using the below properties. 1) Global timeout defined in synapse.properties (ESB_HOME\repository\conf\) file. This will decide the maximum time that a callback is waiting in the ESB for a response for a particular request. If ESB does not get any response from Back End, it will drop the message and clears out the call back. This is a global level parameter which affects all the endpoints configured in ESB. synapse.global_timeout_inte
3 comments
Read more