WSO2 ESB Error Handling Tutorial - Part I (Client side error handling)

Recently, I found a nice video on facebook which was shared by Sanjiva Weerawarana (CEO @ WSO2), which was a narration by Matt Damon. The original paragraph was taken from a speech by Howard Zinn's 1970 speech.

https://vimeo.com/48834336

According to that, world is topsy turvy (upside down). Wrong people are in power, Wrong people are out of power. But there is one thing missing in that speech. Which is that wrong people are using software, wrong people are not using software :).

Sorry about going out of the topic. But this speech has really moved me. Anyway, let's start talking about the subject. WSO2 ESB is the central hub of your SOA architecture. It will communicate with all kinds of heterogenous systems. These systems can go mad sometimes. In such a scenarios, WSO2 ESB should not go mad. If you haven't done proper error handling at WSO2 ESB, even though it does not go mad, people will feel that it has gone mad by looking at lengthy error logs and exceptions. So why do you let people to think in that way. Rather we can implement a proper error handling mechanism and make the WSO2 ESB looks solid at all time.

Here is a typical message flow in your enterprise system which involves WSO2 ESB.

In the above message flow, things can go wrong in all 3 components. 

• Client Error
• ESB Error
• Server Error

In all 3 scenarios, we need to have a proper handling mechanism to identify the error scenarios as soon as they occur. Otherwise, it will cost your organization's business. This can be ranged from hundreds of dollars to millions of dollars. Let's discuss about error handling at each and every component depicted above.

Handling Client errors

Anything can go wrong at any time. That is a fact in this world. So is the Clients who are using the services/APIs exposed through WSO2 ESB. Here are some example scenarios where clients go mad during the message execution.

• Sending wrong messages (non-allowed content)
• Closing connections early
• Sending requests to wrong URLs

Let's discuss these scenarios one by one and learn about the error handling mechanisms which we can take to get over these.

Sending wrong messages

Let's say your client is sending XML messages to the ESB. Due to a mistake in the client code, let's say client sends the following message.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://services.samples" xmlns:xsd="http://services.samples/xsd">

   <soapenv:Header/>

   <soapenv:Body>

      <ser:getQuote>

         <!--Optional:-->

         <ser:request>

            <!--Optional:-->

            <xsd:symbol>WSO2&&&&</xsd:symbol>

         </ser:request>

      </ser:getQuote>

   </soapenv:Body>

</soapenv:Envelope>

In the above message, client is sending '&' character within the XML message (This is only an example). Let's say we have a very simple PassThrough Proxy service defined in the ESB. 

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"

       name="PassThroughProxy"

       transports="https,http"

       statistics="disable"

       trace="disable"

       startOnLoad="true">

   <target>

      <outSequence>

         <send/>

      </outSequence>

      <endpoint>

         <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>

      </endpoint>

   </target>

   <description/>

</proxy>

In this case, ESB will simply pass-through the message to the backend without worrying about the content. According to the capabilities of the back end server, it will respond with some message and ESB will pass the response to the client. No worries at this point. 

All good with the above proxy service. Let's add a log mediator with log level "full". Once we have this content-aware mediator in the message flow, ESB tries to convert the incoming data stream into a canonical XML message. Here comes the exception. Since this message contains wrong XML characters, ESB will fail during the message building process. Now the proxy looks like below.

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"

       name="PassThroughProxy"

       transports="https,http"

       statistics="disable"

       trace="disable"

       startOnLoad="true">

   <target>

      <inSequence>

         <log level="full"/>

      </inSequence>

      <outSequence>

         <send/>

      </outSequence>

      <endpoint>

         <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>

      </endpoint>

   </target>

   <description/>

</proxy>

Once we have this content-aware mediator in place, ESB will try to build the message and it will fail with an exception similar to this.

ERROR - NativeWorkerPool Uncaught exception

org.apache.axiom.om.OMException: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[8,29]

Message: The entity name must immediately follow the '&' in the entity reference.

at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:296)

at org.apache.axiom.om.impl.llom.OMElementImpl.buildNext(OMElementImpl.java:653)

at org.apache.axiom.om.impl.llom.OMElementImpl.getFirstOMChild(OMElementImpl.java:670)

This is fine, since the message is wrong. But what is wrong here is that client did not get any message from ESB related to this error scenario. Then client will timeout the connection after waiting for the configured time duration and you can see the following log in the ESB log console.

[2015-06-27 17:26:08,885]  WARN - SourceHandler Connection time out after request is read: http-incoming-2

We need to handle this situation with a proper error handler. We can define a fault sequence at the proxy service level. Then the message will go through this fault handler sequence and we can send a fault message to the client if we encounter this kind of message. Let's add the fault sequence to the proxy service.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="PassThroughProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <log level="full"/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
      <faultSequence>
         <makefault version="soap11">
            <code xmlns:soap11Env="http://schemas.xmlsoap.org/soap/envelope/"
                  value="soap11Env:Client"/>
            <reason expression="$ctx:ERROR_MESSAGE"/>
            <role/>
         </makefault>
         <send/>
      </faultSequence>
      <endpoint>
         <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
      </endpoint>
   </target>
   <description/>
</proxy>
                               
Once we have the fault sequence, client will get a fault message as the response for this message.


Closing connections early

Another common use case is that client closes the connection before the server respond back with the message. When this happens, you can observe the following error message in the server log file.

[2015-06-28 19:28:37,524]  WARN - SourceHandler Connection time out after request is read: http-incoming-1

This slowness can be occurred due to back end slowness or due to ESB server contention. In both scenarios, you need to increase the client timeout to get rid of this error. You can configure the client timeout to a considerable value which is greater than the maximum response time of the server. But configuring the client timeout only will not make this scenario work for you. The reason is that, even though the client has increased the timeout, ESB will close the connection after 60 seconds (default value). Therefore, you need to configure the client side HTTP connection timeout in the ESB_HOME/repository/conf/passthru-http.properties file with the following parameter. Add this parameter if it is not already there.

http.socket.timeout=120000

Sending Requests to wrong URL

Sometimes, client may send requests to non existing URLs. For example, let's say you have an API defined in the ESB like below.

<api xmlns="http://ws.apache.org/ns/synapse" name="test" context="/test">
   <resource methods="POST GET" url-mapping="/echo">
      <inSequence>
         <log level="full"></log>
         <send>
            <endpoint>
               <address uri="http://localhost:9000/services/SimpleStockQuoteService"></address>
            </endpoint>
         </send>
      </inSequence>
   </resource>
</api>

According to the definition, you need to send the request to following URL.

http://localhost:8280/test/echo

But due to a mistake by the client, it sends a request to the following URL

http://localhost:8280/test/echo2

Here what happens is, ESB will respond with 202 accepted message to the client. That is not the correct message ESB should send to the client since ESB is not processing this message correctly. What it should do is that, it needs to respond to the client with some error message such that client can go though error response and identify the root cause.

We need to define a special sequence for handling this kind of failed requests. You can define this sequence as given below.

<sequence xmlns="http://ws.apache.org/ns/synapse" name="_resource_mismatch_handler_">
   <payloadFactory media-type="xml">
      <format>
         <tp:fault xmlns:tp="http://test.com">
            <tp:code>404</tp:code>
            <tp:type>Status report</tp:type>
            <tp:message>Not Found</tp:message>
            <tp:description>The requested resource (/$1) is not available.</tp:description>
         </tp:fault>
      </format>
      <args>
         <arg xmlns:ns="http://org.apache.synapse/xsd" xmlns:ns3="http://org.apache.synapse/xsd" expression="$axis2:REST_URL_POSTFIX" evaluator="xml"></arg>
      </args>
   </payloadFactory>
   <property name="RESPONSE" value="true" scope="default"></property>
   <property name="NO_ENTITY_BODY" action="remove" scope="axis2"></property>
   <property name="HTTP_SC" value="404" scope="axis2"></property>
   <header name="To" action="remove"></header>
   <send></send>
   <drop></drop>
</sequence>

In the above sequence, you can change the internal mediators as per your wish. But the name of the sequence should be as it is (_resource_mismatch_handler_). One you have this sequence in place, clients will get the following error message if they send requests to non-existing API resources.

 <tp:fault>
<tp:code>404</tp:code>
<tp:type>Status report</tp:type>
<tp:message>Not Found</tp:message>
<tp:description>The requested resource (//echo-test) is not available.</tp:description>
 </tp:fault>

I will be discussing about the rest of the 2 scenarios in a future blog post.

Handling back end Server errors

Handling ESB errors