Skip to main content

WSO2 ESB Error Handling Tutorial - Part I (Client side error handling)

Recently, I found a nice video on facebook which was shared by Sanjiva Weerawarana (CEO @ WSO2), which was a narration by Matt Damon. The original paragraph was taken from a speech by Howard Zinn's 1970 speech.

https://vimeo.com/48834336

According to that, world is topsy turvy (upside down). Wrong people are in power, Wrong people are out of power. But there is one thing missing in that speech. Which is that wrong people are using software, wrong people are not using software :).

Sorry about going out of the topic. But this speech has really moved me. Anyway, let's start talking about the subject. WSO2 ESB is the central hub of your SOA architecture. It will communicate with all kinds of heterogenous systems. These systems can go mad sometimes. In such a scenarios, WSO2 ESB should not go mad. If you haven't done proper error handling at WSO2 ESB, even though it does not go mad, people will feel that it has gone mad by looking at lengthy error logs and exceptions. So why do you let people to think in that way. Rather we can implement a proper error handling mechanism and make the WSO2 ESB looks solid at all time.

Here is a typical message flow in your enterprise system which involves WSO2 ESB.
In the above message flow, things can go wrong in all 3 components. 
  • Client Error
  • ESB Error
  • Server Error
In all 3 scenarios, we need to have a proper handling mechanism to identify the error scenarios as soon as they occur. Otherwise, it will cost your organization's business. This can be ranged from hundreds of dollars to millions of dollars. Let's discuss about error handling at each and every component depicted above.

Handling Client errors

Anything can go wrong at any time. That is a fact in this world. So is the Clients who are using the services/APIs exposed through WSO2 ESB. Here are some example scenarios where clients go mad during the message execution.

  • Sending wrong messages (non-allowed content)
  • Closing connections early
  • Sending requests to wrong URLs
Let's discuss these scenarios one by one and learn about the error handling mechanisms which we can take to get over these.

Sending wrong messages

Let's say your client is sending XML messages to the ESB. Due to a mistake in the client code, let's say client sends the following message.


<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://services.samples" xmlns:xsd="http://services.samples/xsd">
   <soapenv:Header/>
   <soapenv:Body>
      <ser:getQuote>
         <!--Optional:-->
         <ser:request>
            <!--Optional:-->
            <xsd:symbol>WSO2&&&&</xsd:symbol>
         </ser:request>
      </ser:getQuote>
   </soapenv:Body>
</soapenv:Envelope>

In the above message, client is sending '&' character within the XML message (This is only an example). Let's say we have a very simple PassThrough Proxy service defined in the ESB. 

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="PassThroughProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <outSequence>
         <send/>
      </outSequence>
      <endpoint>
         <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
      </endpoint>
   </target>
   <description/>
</proxy>

In this case, ESB will simply pass-through the message to the backend without worrying about the content. According to the capabilities of the back end server, it will respond with some message and ESB will pass the response to the client. No worries at this point. 

All good with the above proxy service. Let's add a log mediator with log level "full". Once we have this content-aware mediator in the message flow, ESB tries to convert the incoming data stream into a canonical XML message. Here comes the exception. Since this message contains wrong XML characters, ESB will fail during the message building process. Now the proxy looks like below.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="PassThroughProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <log level="full"/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
      <endpoint>
         <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
      </endpoint>
   </target>
   <description/>
</proxy>

Once we have this content-aware mediator in place, ESB will try to build the message and it will fail with an exception similar to this.

ERROR - NativeWorkerPool Uncaught exception
org.apache.axiom.om.OMException: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[8,29]
Message: The entity name must immediately follow the '&' in the entity reference.
at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:296)
at org.apache.axiom.om.impl.llom.OMElementImpl.buildNext(OMElementImpl.java:653)
at org.apache.axiom.om.impl.llom.OMElementImpl.getFirstOMChild(OMElementImpl.java:670)

This is fine, since the message is wrong. But what is wrong here is that client did not get any message from ESB related to this error scenario. Then client will timeout the connection after waiting for the configured time duration and you can see the following log in the ESB log console.

[2015-06-27 17:26:08,885]  WARN - SourceHandler Connection time out after request is read: http-incoming-2

We need to handle this situation with a proper error handler. We can define a fault sequence at the proxy service level. Then the message will go through this fault handler sequence and we can send a fault message to the client if we encounter this kind of message. Let's add the fault sequence to the proxy service.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="PassThroughProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <log level="full"/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
      <faultSequence>
         <makefault version="soap11">
            <code xmlns:soap11Env="http://schemas.xmlsoap.org/soap/envelope/"
                  value="soap11Env:Client"/>
            <reason expression="$ctx:ERROR_MESSAGE"/>
            <role/>
         </makefault>
         <send/>
      </faultSequence>
      <endpoint>
         <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
      </endpoint>
   </target>
   <description/>
</proxy>
                               
Once we have the fault sequence, client will get a fault message as the response for this message.


Closing connections early

Another common use case is that client closes the connection before the server respond back with the message. When this happens, you can observe the following error message in the server log file.

[2015-06-28 19:28:37,524]  WARN - SourceHandler Connection time out after request is read: http-incoming-1

This slowness can be occurred due to back end slowness or due to ESB server contention. In both scenarios, you need to increase the client timeout to get rid of this error. You can configure the client timeout to a considerable value which is greater than the maximum response time of the server. But configuring the client timeout only will not make this scenario work for you. The reason is that, even though the client has increased the timeout, ESB will close the connection after 60 seconds (default value). Therefore, you need to configure the client side HTTP connection timeout in the ESB_HOME/repository/conf/passthru-http.properties file with the following parameter. Add this parameter if it is not already there.

http.socket.timeout=120000

Sending Requests to wrong URL

Sometimes, client may send requests to non existing URLs. For example, let's say you have an API defined in the ESB like below.

<api xmlns="http://ws.apache.org/ns/synapse" name="test" context="/test">
   <resource methods="POST GET" url-mapping="/echo">
      <inSequence>
         <log level="full"></log>
         <send>
            <endpoint>
               <address uri="http://localhost:9000/services/SimpleStockQuoteService"></address>
            </endpoint>
         </send>
      </inSequence>
   </resource>
</api>

According to the definition, you need to send the request to following URL.

http://localhost:8280/test/echo

But due to a mistake by the client, it sends a request to the following URL

http://localhost:8280/test/echo2

Here what happens is, ESB will respond with 202 accepted message to the client. That is not the correct message ESB should send to the client since ESB is not processing this message correctly. What it should do is that, it needs to respond to the client with some error message such that client can go though error response and identify the root cause.

We need to define a special sequence for handling this kind of failed requests. You can define this sequence as given below.

<sequence xmlns="http://ws.apache.org/ns/synapse" name="_resource_mismatch_handler_">
   <payloadFactory media-type="xml">
      <format>
         <tp:fault xmlns:tp="http://test.com">
            <tp:code>404</tp:code>
            <tp:type>Status report</tp:type>
            <tp:message>Not Found</tp:message>
            <tp:description>The requested resource (/$1) is not available.</tp:description>
         </tp:fault>
      </format>
      <args>
         <arg xmlns:ns="http://org.apache.synapse/xsd" xmlns:ns3="http://org.apache.synapse/xsd" expression="$axis2:REST_URL_POSTFIX" evaluator="xml"></arg>
      </args>
   </payloadFactory>
   <property name="RESPONSE" value="true" scope="default"></property>
   <property name="NO_ENTITY_BODY" action="remove" scope="axis2"></property>
   <property name="HTTP_SC" value="404" scope="axis2"></property>
   <header name="To" action="remove"></header>
   <send></send>
   <drop></drop>
</sequence>

In the above sequence, you can change the internal mediators as per your wish. But the name of the sequence should be as it is (_resource_mismatch_handler_). One you have this sequence in place, clients will get the following error message if they send requests to non-existing API resources.

 <tp:fault>
<tp:code>404</tp:code>
<tp:type>Status report</tp:type>
<tp:message>Not Found</tp:message>
<tp:description>The requested resource (//echo-test) is not available.</tp:description>
 </tp:fault>

I will be discussing about the rest of the 2 scenarios in a future blog post.

Handling back end Server errors

Handling ESB errors

Comments

  1. Hi Chanaka , It was really a wonderful article i'm looking forward for your part - II in error handling . Thank you .

    ReplyDelete
  2. This syllabus of the Note 4 is a guided way to learn this technology. Those who provide education for machine learning should get inspiration from the course module. Click here to learn more about machine learning companies.

    ReplyDelete
  3. I found your blog looking for updates, I'm happy to be here. Very useful and also easily understandable content.
    Mulesoft Online Training
    Mulesoft Training in Hyderabad

    ReplyDelete
  4. Hi @chanakaudaya,

    This blog is useful for WSO2 Beginners especially, I request you to share remaining 2 component's ERROR Handling Mechanism.

    Thank you

    ReplyDelete

Post a Comment

Popular posts from this blog

How to configure timeouts in WSO2 ESB to get rid of client timeout errors

WSO2 ESB has defined some configuration parameters which controls the timeout of a particular request which is going out of ESB. In a particular  scneario, your client sends a request to ESB, and then ESB sends a request to another endpoint to serve the request. CLIENT->WSO2 ESB->BACKEND The reason for clients getting timeout is that ESB timeout is larger than client's timeout. This can be solved by either increasing the timeout at client side or by decreasing the timeout in ESB side. In any of the case, you can control the timeout in ESB using the below properties. 1) Global timeout defined in synapse.properties (ESB_HOME\repository\conf\) file. This will decide the maximum time that a callback is waiting in the ESB for a response for a particular request. If ESB does not get any response from Back End, it will drop the message and clears out the call back. This is a global level parameter which affects all the endpoints configured in ESB. synapse.global_timeout_inte

Understanding Threads created in WSO2 ESB

WSO2 ESB is an asynchronous high performing messaging engine which uses Java NIO technology for its internal implementations. You can find more information about the implementation details about the WSO2 ESB’s high performing http transport known as Pass-Through Transport (PTT) from the links given below. [1] http://soatutorials.blogspot.com/2015/05/understanding-wso2-esb-pass-through.html [2] http://wso2.com/library/articles/2013/12/demystifying-wso2-esb-pass-through-transport-part-i/ From this tutorial, I am going to discuss about various threads created when you start the ESB and start processing requests with that. This would help you to troubleshoot critical ESB server issues with the usage of a thread dump. You can monitor the threads created by using a monitoring tool like Jconsole or java mission control (java 1.7.40 upwards). Given below is a list of important threads and their stack traces from an active ESB server.  PassThroughHTTPSSender ( 1 Thread )

How puppet works in your IT infrstructure

What is Puppet? Puppet is IT automation software that helps system administrators manage infrastructure throughout its lifecycle, from provisioning and configuration to orchestration and reporting. Using Puppet, you can easily automate repetitive tasks, quickly deploy critical applications, and proactively manage change, scaling from 10s of servers to 1000s, on-premise or in the cloud. How the puppet works? It works like this..Puppet agent is a daemon that runs on all the client servers(the servers where you require some configuration, or the servers which are going to be managed using puppet.) All the clients which are to be managed will have puppet agent installed on them, and are called nodes in puppet. Puppet Master: This machine contains all the configuration for different hosts. Puppet master will run as a daemon on this master server. Puppet Agent: This is the daemon that will run on all the servers, which are to be managed using p