Performance Tuning WSO2 ESB with a practical example

WSO2 ESB is arguably the highest performing open source ESB available in the industry. With the default settings, it will provide a really good performance OOTB. You can find the official performance tuning guide in the below link.

https://docs.wso2.com/display/ESB481/Performance+Tuning

Above document includes lot of parameters and how to change them for best performance. Even though it provides some recommended values for the performance tuning, that is not straightforward. The values you put in the configuration files will highly reliant on your use case. Idea of this blog post is to provide a practical example of tuning these parameters with a sample use case.

For the performance testing, I am using a simple proxy service which iterate through set of xml elements and send request to a sample server and aggregate the responses back to the client.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="SplitAggregateProxy"
       startOnLoad="true">
   <target>
      <inSequence>
         <iterate xmlns:m0="http://services.samples"
                  preservePayload="true"
                  attachPath="//m0:getQuote"
                  expression="//m0:getQuote/m0:request">
            <target>
               <sequence>
                  <send>
                     <endpoint>
                        <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
                     </endpoint>
                  </send>
               </sequence>
            </target>
         </iterate>
      </inSequence>
      <outSequence>
         <aggregate>
            <completeCondition>
               <messageCount/>
            </completeCondition>
            <onComplete xmlns:m0="http://services.samples" expression="//m0:getQuoteResponse">
               <send/>
            </onComplete>
         </aggregate>
      </outSequence>
   </target>
</proxy>

The reason for selecting this type of proxy is that it will allow us to test different thread groups within the WSO2 ESB.  Request for this proxy is given below.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://services.samples" xmlns:xsd="http://services.samples/xsd">
   <soapenv:Header/>
   <soapenv:Body>
      <ser:getQuote>
          <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>
           <!--Optional:-->
         <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>
          <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>  
          <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>
          <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>
          <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>    
          <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>
           <!--Optional:-->
         <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>
          <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>  
          <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>
          <ser:request>
            <!--Optional:-->
            <xsd:symbol>IBM</xsd:symbol>
         </ser:request>        
      </ser:getQuote>    
   </soapenv:Body>
</soapenv:Envelope>

We have used the apache jmeter as the test client. There we have created a Thread group with following configurations.

Number of Threads (Users) - 50
Ramp up period (seconds) - 10
Loop count - 200
Total requests - 10000

With this information in hand, we have used WSO2 ESB 4.8.1 within a ubuntu linux machine with 8GB memory and 4 core cpu.

First, let's talk about the parameters which we have used to carry out this performance test.

(ESB_HOME/bin/wso2server.sh)
Memory (Xmx) - Maximum heap memory allocated for JVM. We kept both Xmx and Xms the same.

(ESB_HOME/repository/conf/synapse.properties)
synapse.threads.core - Core number of threads allocated for ThreadPoolExecutor used for creating threads for iterate mediator executions

synapse.threads.qlen - Task queue length used for ThreadPoolExecutor used for creating threads for iterate mediator executions

(ESB_HOME/repository/conf/passthru-http.properties)
worker_pool_size_core - Core number of threads allocated for ThreadPoolExecutor used for processing incoming requests to the ESB

io_buffer_size - Size of the memory buffer used for reading/writing data from NIO channels


Performance tests were carried out while monitoring the server using Java Mission Control tool and the server load is kept at a perfect value around 60-70 % CPU usage and Load average value around 3-4 (4 core). There were no GC related issues observed during this testing.

You can read the following article to get an understanding about the performance of a server and the methods for tuning servers.

http://www.infoq.com/articles/Tuning-Java-Servers

We have captured both Latency and the TPS value for monitoring the performance of the server


Performance variation with Memory allocation

Theoretically, the performance of the server should be increased with the allocated memory. By performance, we consider both Latency and the TPS value of the server. According to the below results, we can see that TPS is increased with the memory and the Latency has reduced (increased performance).

Performance variation with number of Server Worker/Client Worker Threads

WSO2 ESB uses the ThreadPoolExecutor to create threads when there is data to be processed from the client requests. worker_pool_size_core parameter controls the number of core threads for this executor pool. By increasing the thread pool, we would expect that we see a performance improvement. According to the below graphs, we can see that latency is reduced and the TPS is slightly improved with this parameter (performance increased with number of threads)

Performance variation with Synapse Worker Threads count

When using iterate or clone mediator within the ESB, it will use a separate thread pool to create new threads when there is data to be iterated and processed. The size of this thread pool is configured with the parameter synapse.threads.core. By increasing the value here, we would expect a better performance when iterate mediator is used. According to the test results, we can see that performance is increased when the value is changed from 20 to 100. But after that, we can see some performance degradation since when the number of threads in the system increases, that will degrade the performance due to the high load in the thread scheduler in the operating system.

Performance variation with Synapse Worker Queue length

When using iterate or clone mediator within the ESB, it will use a separate thread pool to create new threads when there is data to be iterated and processed. We can configure the task queue length of this thread pool with the synapse.threads.qlen parameter. By giving a finite value to this task queue, it will make the system to create new threads when all the core number of threads are created and task queue is full. This is the only time that max value of thread pool is used. If the queue length is infinite (-1), max value is never used and there will only be core number of threads at any given time. According to the results, we can see that, when there is a finite value for queue length, we can see increased performance. One thing to note here is that, when we have a limited value for this queue length, there can be situations that requests will be rejected when the task queue is full and all the threads are occupied. Therefore, you need to make a decision according to your actual load. One additional thing with this parameter is that if there is some thread blocking happens when the queue length is infinite, server can go in to OOM situation.

Performance variation with IO Buffer size

IO buffer size parameter(io_buffer_size) decides the value of the memory buffer allocated for reading data in to the memory from the underlying socket/file channels. According to the average value of the payloads, we can configure this parameter. According to the results we observed from the testing, we cannot clearly come to a conclusion for this scenario since the request/response size is <4k during this testing.

According to the above results, we can see that when tuning the WSO2 ESB is not straightforward and you need to have a proper idea about your use cases.