Skip to main content

Understanding Threads created in WSO2 ESB

WSO2 ESB is an asynchronous high performing messaging engine which uses Java NIO technology for its internal implementations. You can find more information about the implementation details about the WSO2 ESB’s high performing http transport known as Pass-Through Transport (PTT) from the links given below.

[1] http://soatutorials.blogspot.com/2015/05/understanding-wso2-esb-pass-through.html

[2] http://wso2.com/library/articles/2013/12/demystifying-wso2-esb-pass-through-transport-part-i/

From this tutorial, I am going to discuss about various threads created when you start the ESB and start processing requests with that. This would help you to troubleshoot critical ESB server issues with the usage of a thread dump. You can monitor the threads created by using a monitoring tool like Jconsole or java mission control (java 1.7.40 upwards). Given below is a list of important threads and their stack traces from an active ESB server. 



PassThroughHTTPSSender ( 1 Thread )

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests to https endpoints from ESB. When there is a <send/> or <call/> mediator within the mediation flow to and https endpoint, it will eventually executes this thread. This thread needs to be running all the time during the server is up and running (unless https transport is disabled).

Name: PassThroughHTTPSSender
State: RUNNABLE
Total blocked: 1  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@2bef6f1
   - locked java.util.Collections$UnmodifiableSet@578b8a47
   - locked sun.nio.ch.KQueueSelectorImpl@1a8f39bb
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpSender$2.run(PassThroughHttpSender.java:195)
java.lang.Thread.run(Thread.java:744)


HTTPS-Sender I/O dispatcher-1 (4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to sending messages to https endpoints are handled by these threads. These threads need to be running all the time during the server is up and running (unless https transport is disabled).


Name: HTTPS-Sender I/O dispatcher-1
State: RUNNABLE
Total blocked: 1  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@25484cc9
   - locked java.util.Collections$UnmodifiableSet@3f6342dd
   - locked sun.nio.ch.KQueueSelectorImpl@66ac838c
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)


PassThroughHTTPSender ( 1 Thread )

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests to http endpoints from ESB. When there is a <send/> or <call/> mediator within the mediation flow to http endpoint, it will eventually executes this thread and create a new socket channel. This thread needs to be running all the time during the server is up and running (unless http transport is disabled).

Name: PassThroughHTTPSender
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@c15e5c9
   - locked java.util.Collections$UnmodifiableSet@7e49d31e
   - locked sun.nio.ch.KQueueSelectorImpl@9052336
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpSender$2.run(PassThroughHttpSender.java:195)
java.lang.Thread.run(Thread.java:744)


HTTP-Sender I/O dispatcher-1 (4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to sending messages to http endpoints are handled by these threads. These threads need to be running all the time during the server is up and running (unless http transport is disabled).

Name: HTTP-Sender I/O dispatcher-1
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@327d74e9
   - locked java.util.Collections$UnmodifiableSet@460208f5
   - locked sun.nio.ch.KQueueSelectorImpl@33f55d67
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)


PassThroughHTTPSListener (1 Thread)

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests from https requests from clients. When there is a https request towards ESB, it will eventually executes this thread. When this request is received, it will create a new socket channel and register with a selection key and a channel. Further IO events are handled by the dispatcher threads. This thread needs to be running all the time during the server is up and running (unless https transport is disabled).

Name: PassThroughHTTPSListener
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@50cff7e
   - locked java.util.Collections$UnmodifiableSet@271d259c
   - locked sun.nio.ch.KQueueSelectorImpl@2282db4d
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpListener$4.run(PassThroughHttpListener.java:251)
java.lang.Thread.run(Thread.java:744)




HTTPS-Listener I/O dispatcher-1(4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to receiving messages from https requests are handled by these threads. These threads need to be running all the time during the server is up and running (unless https transport is disabled).

Name: HTTPS-Listener I/O dispatcher-1
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@48c97759
   - locked java.util.Collections$UnmodifiableSet@8ac860c
   - locked sun.nio.ch.KQueueSelectorImpl@ff4fe7c
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)


PassThroughHTTPListener (1 Thread)

This thread is the reactor(acceptor) thread which is responsible for handling new connection requests from http requests from clients. When there is a http request towards ESB, it will eventually executes this thread. When this request is received, it will create a new socket channel and register with a selection key and a channel. Further IO events are handled by the dispatcher threads. This thread needs to be running all the time during the server is up and running (unless http transport is disabled).

Name: PassThroughHTTPListener
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@1f6438be
   - locked java.util.Collections$UnmodifiableSet@152987f9
   - locked sun.nio.ch.KQueueSelectorImpl@11d60796
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:366)
org.apache.synapse.transport.passthru.PassThroughHttpListener$4.run(PassThroughHttpListener.java:251)
java.lang.Thread.run(Thread.java:744)



HTTP-Listener I/O dispatcher-1(4 Threads in 4 Core)

This thread is the reactor thread which is responsible for handling IO events and trigger different states of the state machine. There will be n number of this threads where n equals to number of cores in the machine. All the I/O events related to receiving messages from http requests are handled by these threads. These threads need to be running all the time during the server is up and running (unless http transport is disabled).

Name: HTTP-Listener I/O dispatcher-1
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@34095061
   - locked java.util.Collections$UnmodifiableSet@60e8e6e5
   - locked sun.nio.ch.KQueueSelectorImpl@5fbe8e73
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:259)
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
java.lang.Thread.run(Thread.java:744)


PassThroughMessageProcessor-1(400)

This is the worker thread which is responsible for processing requests coming in to the ESB when the request message is received. Passthrough transport will start these worker threads once the HTTP headers are received from the clients. Then these threads will run independently from PTT threads which are described earlier. There will be n number of threads created where n is the worker_pool_size_core parameter defined in the passthrough-http.properties file. 

Name: PassThroughMessageProcessor-1
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7050afc9
Total blocked: 0  Total waited: 1

Stack trace: 
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)


SynapseWorker-1(20)

This is the worker thread which is responsible for handling iterate/clone mediator executions. There will be a separate thread pool to handle such executions. Number of threads in this thread pool can be configured in synapse.properties file with the parameter synapse.threads.core. 

Name: SynapseWorker-1
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22d3b7d9
Total blocked: 1  Total waited: 2

Stack trace: 
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)



Following are threads related to embedded tomcat instance run within the ESB. It will be used to serve requests coming in to ESB management console, admin services. You can configure the thread pool and other parameters from ESB_HOME/repository/conf/tomcat/catalina-server.xml file.



NioBlockingSelector.BlockPoller-1(2)

Name: NioBlockingSelector.BlockPoller-1
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@5d630f69
   - locked java.util.Collections$UnmodifiableSet@59cdfa64
   - locked sun.nio.ch.KQueueSelectorImpl@510b6d29
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:327)


http-nio-9763-ClientPoller-0(2)

Name: http-nio-9763-ClientPoller-0
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@5f185036
   - locked java.util.Collections$UnmodifiableSet@1b5f14d
   - locked sun.nio.ch.KQueueSelectorImpl@5b8b369f
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1146)
java.lang.Thread.run(Thread.java:744)


http-nio-9763-Acceptor-0(2)

Name: http-nio-9763-Acceptor-0
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
   - locked java.lang.Object@525f8df5
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:787)
java.lang.Thread.run(Thread.java:744)


http-nio-9443-ClientPoller-0(2)

Name: http-nio-9443-ClientPoller-0
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@3d8e1f2f
   - locked java.util.Collections$UnmodifiableSet@2f3ecb19
   - locked sun.nio.ch.KQueueSelectorImpl@113dc8a9
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1146)
java.lang.Thread.run(Thread.java:744)


http-nio-9443-Acceptor-0(2)

Name: http-nio-9443-Acceptor-0
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
   - locked java.lang.Object@38fcf802
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:787)
java.lang.Thread.run(Thread.java:744)


http-nio-9443-exec-1(50)

Name: http-nio-9443-exec-1
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a4e4fb2
Total blocked: 1  Total waited: 6

Stack trace: 
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)





Labels:

Comments

Post a Comment

Popular posts from this blog

How to protect your APIs with self contained access token (JWT) using WSO2 API Manager and WSO2 Identity Server

In a typical enterprise information system, there is a high chance that people will use different types of systems built by different vendors to implement certain types of functionalities. The APIs might be hosted in an API Manager developed by vendor A and the user management can be implemented using a different vendor (vendor B). In this type of a situation, one system will not be able to directly contact the other system but they want to use both systems in tandem. Self-contained access tokens are used in these types of situations where applications can get the token from one system and use that in another system to access protected resources. In this scenario, the second system does not need to make a contact to the first system over the network to validate the user information since the token is self-contained and it has relevant details about the user. This will improve the token processing time significantly since it completely removes the network interaction. The below fig...
1 comment
Read more

WSO2 ESB creating a response for a "GET" request

When you are creating REST APIs with WSO2 ESB, you may need to send some response message back to the user when something goes wrong. In this kind of scenario, you can create a payload inside the ESB and send it back. For doing this, you can use the below configuration. <api xmlns=" http://ws.apache.org/ ns/synapse " name="LoopBackProxy" context="/loopback">    <resource methods="POST GET">       <inSequence>          <property name="NO_ENTITY_BODY" scope="axis2" action="remove"></property>          <log level="full"></log>          <payloadFactory media-type="xml">             <format>                <m:messageBeforeLoopBack xmlns:m=" http://services...
Post a Comment
Read more

WSO2 ESB usage of get-property function

What are Properties? WSO2 has a huge set of mediators but property mediator is mostly used mediator for writing any proxy service or API. Property mediator is used to store any value or xml fragment temporarily during life cycle of a thread for any service or API. We can compare “Property” mediator with “Variable” in any other traditional programming languages (Like: C, C++, Java, .Net etc). There are few properties those are used/maintained by ESB itself and on the other hand few properties can be defined by users (programmers). In other words, we can say that properties can be define in below 2 categories: ESB Defined Properties User Defined Properties. These properties can be stored/defined in different scopes, like: Transport Synapse or Default Axis2 Registry System Operation Generally, these properties are read by get-properties() function. This function can be invoked with below 2 variations. get-property(String propertyName) get-property(String scop...
Post a Comment
Read more