Skip to main content

Monitoring WSO2 ESB Mediation Statistics with WSO2 BAM


Monitoring mediation statistics data with BAM

Setting up Mediation Data Agent

BAM Mediation Data Agent is used to collect statistics related to mediation data from WSO2 ESB and send to WSO2 BAM. The BAM server receives data events via its Thrift API. This is a binary protocol that enables fast data transmission between the ESB and BAM. It is possible to configure BAM Mediation Data Agent early so that you can predefine the statistics to be extracted from mediation data of ESB. In the configuration panel, specify the BAM server (Thrift server) related information and properties to be extracted from the Configuration Context of ESB. Also specify the Event Stream related parameters uniquely identified by a name and a version.

Installing BAM Mediation Data Agent in the ESB

The following installation instructions apply to WSO2 ESB version 4.5.x onwards.
  • Download WSO2 ESB from http://wso2.com/products/enterprise-service-bus.
When running both ESB and BAM servers concurrently, you must change the port of one server in order to prevent port conflicts. This is done by applying an offset in file

<ESB_HOME>/repository/conf/carbon.xml.

<!-- Ports offset. This entry will set the value of the ports defined below to the define value + Offset. e.g. Offset=2 and HTTPS port=9443 will set the effective HTTPS port to 9445 -->
<Offset>2</Offset>

  • Start the ESB server by executing the following commands from <ESB_HOME>/bin.
on Linux wso2server.sh
on MS Windows wso2server.bat
  • After starting the server, log in to ESB's management console ( https://localhost:9445/carbon )

In the management console, go to Configure > Features menu.
In the Repository Management tab, select and add a P2 repository either from a URL or a file. For instructions, see Managing the Feature Repository.
WSO2 features are available in Equinox P2 repository at http://dist.wso2.org/p2/carbon/releases. Repositories are listed there by WSO2 Carbon platform versions. Check the WSO2 Release Matrix, identify the WSO2 Carbon version that your product is based on and get the repository that is based on that Carbon version.
After adding the repository, go to Available Features tab. From the repository drop-down list, select the one you added above. The, click Find Features.
From the list of features that appears, select BAM Mediation Data Agent Aggregate feature and Install.
After the installation is done, restart the ESB server.

(Note: From ESB 4.6.0 onwards, this feature is bundled with ESB, you can find it from installed features section)

Configuring and using the BAM Mediation Data Agent

The sections below describe, through an example scenario, how to use the BAM Mediation Data Agent to log mediation statistics data into a Cassandra Database:
  • Configuring the BAM Mediation Data Agent
Enable statistics in <ESB_HOME>/repository/conf/carbon.xml file as follows:
<StatisticsReporterDisabled>false</StatisticsReporterDisabled>

  • After changing the carbon.xml file you need to restart the ESB server.
  • Log in to the ESB management console and select Mediation Data Publishing in the Configure menu.
The Mediation Data Publisher Configuration window opens. Fill in the field. By default the Stream name must be bam_mediation_stats_data_publisher. For example,

Enable Mediation Stats: Select this option to enable the agent.
Stream Definition Configuration:
Stream Name : Any string value with alpha-numeric characters.
Version : Stream version distinguishes different streams with the same stream name. Default version is 1.0.0.
Nick Name : This is a user-preferred nick name to the stream name in alpha-numeric characters. Useful when stream name is lengthy and you want to keep a shorter alias for convenience.
Description : A description about the stream identified by Stream Name, Stream Version pair. Description should also consist of alpha-numeric characters.
BAM Credentials:
BAM URL : IP address of the BAM server. Port must be the thrift port (e.g., tcp://127.0.0.1:7611). Add a comma-separated list of server URLs here to ensure that the load is distributed among many servers. It minimizes effects of a failure of one node.
Username and Password: BAM Thrift login user name and password. Default values are admin/admin.
Properties: BAM receives events from different data agents. You can associate a property to uniquely identify the events triggering from one agent. This feature is particularly useful in a clustered environment.
Now you have configured the BAM mediation data agent successfully.

Setting up the backend server for ESB endpoint

In order to test the functionality of the BAM Mediation Data Agent, create an Endpoint service as the WSDL proxy service and define a sample WSDL proxy service as follows:
Navigate to <ESB_HOME>/samples/axis2Server/src/SimpleStockQuoteService and build the backend service with Apache Ant (Install Ant 1.7.0 or higher from http://ant.apache.org).

just type “ant” inside the folder <ESB_HOME>/samples/axis2Server/src/SimpleStockQuoteService . This will build the sample service and place it in the axis2server.

Go to <ESB_HOME>/samples/axis2Server and run the Axis2 server.

In Linux, axis2server.sh
In Windows, axis2server.bat

Test whether the proxy service WSDL exists in http://localhost:9000/services/SimpleStockQuoteService?wsdl

Adding a proxy service to WSO2 ESB

These are the steps to create a sample WSDL based Proxy Service to test the BAM Mediation Data Agent.
Log in to ESB management console and select Add > Proxy Service from the Main menu.
In the Create Proxy Service from Template page that opens, click WSDL Based Proxy link to create a new WSDL-based proxy.
The WSDL Based Proxy page opens. Fill the form and click Create. For example,
The fields are described below. The ones marked with * are mandatory.

Proxy Service Name: Name of the proxy. e.g., Simple_Stock_Quote_Service_Proxy.
WSDL URI: http://localhost:9000/services/SimpleStockQuoteService?wsdl
WSDL Service: SimpleStockQuoteService
WSDL Port: SimpleStockQuoteServiceHttpSoap11Endpoint

The Deployed Services page opens. Select the newly-created proxy (e.g., Simple_Stock_Quote_Service_Proxy) to access its service dashboard.
In the proxy service's dashboard that opens, click Enable Statistics in category Specific Configuration.

Sending messages to the ESB

When you come to this stage, make sure all previous steps are completed in order to test the BAM Mediation Data Agent. In this last step, you can send custom messages through the previously-created WSDL Proxy Service and test whether they are correctly stored in the Cassandra database. This is only an example, but the steps to use the BAM Mediation Data Agent in a production environment are similar.
Navigate to <ESB_HOME>/samples/axis2Client and build the backend service with Apache Ant as follows:
ant stockquote -Daddurl=https://localhost:8244/services/Simple_Stock_Quote_Service_Proxy -Dmode=fullquote -Dsymbol=testString
Daddurl value must be given from the HTTPS endpoint located in Simple_Stock_Quote_Service_Proxy service's dashboard. To access its dashboard, log in to ESB's management console and click Services > List menu in the Main menu. Then, click the Simple_Stock_Quote_Service_Proxy service from the Deployed Services page. Instead of the testString for symbol, you can use any preferred string.
Once done, the BAM Mediation Data Agent in the Simple_Stock_Quote_Service_Proxy service sends statistics from the ESB to the column family with the given stream name in key-space EVENT_KS in the Cassandra database. Use Cassandra Explorer in BAM to see the data in the Cassandra database. For example,

Analyzing the mediation statistics

BAM mediation statistics metrics are similar to the metrics used by General ESB mediation statistics component. Mediation statistics are widely used monitoring features where functional components of the ESB, sequences, proxy services and endpoints directly get involved in processing messages. The BAM Mediation Statistics feature enables a server/tenant administrator to collect runtime statistics on these three types of functional components and view them through an interactive BAM Dashboard. This feature provides a simple but effective way to determine the runtime performance of the ESB. The collected statistical data can be used to analyze network traffic patterns and draw rough estimates on how the service bus will be used in the future. One of the most enticing attributes of the mediation statistics feature is that it can be configured in a fine-grained manner. In other words, it allows a user to collect data on only a specified set of sequences, proxy services and endpoints.
All statistics collected by the mediation component is aggregated for time duration and published to BAM as an event. That is, mediation statistics events are not sent for every request. Rather, the server aggregates request measures for a period of time and sends an event to BAM. For more information on the measures of mediation statistics, see this article in WSO2 library:

Viewing the data with Mediation Statistics Toolbox

The "Mediation Statistics Toolbox" is a default, out-of-the-box toolbox provided by WSO2 BAM for point-and-click installation. It includes the necessary artifacts (hive scripts, data stream definitions, properties files, gadgets etc.) required to collect and monitor activity related to services deployed in any WSO2 service-hosting product like the WSO2 Enterprise Service Bus, WSO2 Application Server, WSO2 Data Services Server etc.
1. Log in to BAM's management console http://localhost:9443/carbon and select "BAM Toolbox -> Add" in the "Main" menu.
2. The "Add Tool Box" page appears. Select the "Mediation Stats Monitoring Toolbox" and click "Install" button. With this simple step, all required artifacts will be deployed to BAM.
3. The "Available BAM Tool Boxes" page opens with the installation status as "Installing". Wait a few seconds and refresh the page to see the status changed to "Installed".

Executing the Analytics Scripts

After installing the toolbox, navigate to "Analytics -> List" in the "Main" menu.
When installing the Toolbox, an analytics script is added for execution. It can simply be executed to display statistics on the dashboard. Click the "Execute" link associated with it.

Viewing the mediation data graphs

After running the hive script, navigate to "Dashboard" in the "Main" menu.
You should be able to see visualized data of a particular service deployed in the WSO2 ESB. For example,

You can view Proxy services, Sequences and End point data in the graphs.




Comments

Popular posts from this blog

WSO2 ESB tuning performance with threads

I have written several blog posts explaining the internal behavior of the ESB and the threads created inside ESB. With this post, I am talking about the effect of threads in the WSO2 ESB and how to tune up threads for optimal performance. You can refer [1] and [2] to understand the threads created within the ESB. [1] http://soatutorials.blogspot.com/2015/05/understanding-threads-created-in-wso2.html [2] http://wso2.com/library/articles/2012/03/importance-performance-wso2-esb-handles-nonobvious/ Within this blog post, I am discussing about the "worker threads" which are used for processing the data within the WSO2 ESB. There are 2 types of worker threads created when you start sending the requests to the server 1) Server Worker/Client Worker Threads 2) Mediator Worker (Synapse-Worker) Threads Server Worker/Client Worker Threads These set of threads will be used to process all the requests/responses coming to the ESB server. ServerWorker Threads will be used to pr

How puppet works in your IT infrstructure

What is Puppet? Puppet is IT automation software that helps system administrators manage infrastructure throughout its lifecycle, from provisioning and configuration to orchestration and reporting. Using Puppet, you can easily automate repetitive tasks, quickly deploy critical applications, and proactively manage change, scaling from 10s of servers to 1000s, on-premise or in the cloud. How the puppet works? It works like this..Puppet agent is a daemon that runs on all the client servers(the servers where you require some configuration, or the servers which are going to be managed using puppet.) All the clients which are to be managed will have puppet agent installed on them, and are called nodes in puppet. Puppet Master: This machine contains all the configuration for different hosts. Puppet master will run as a daemon on this master server. Puppet Agent: This is the daemon that will run on all the servers, which are to be managed using p

How to configure timeouts in WSO2 ESB to get rid of client timeout errors

WSO2 ESB has defined some configuration parameters which controls the timeout of a particular request which is going out of ESB. In a particular  scneario, your client sends a request to ESB, and then ESB sends a request to another endpoint to serve the request. CLIENT->WSO2 ESB->BACKEND The reason for clients getting timeout is that ESB timeout is larger than client's timeout. This can be solved by either increasing the timeout at client side or by decreasing the timeout in ESB side. In any of the case, you can control the timeout in ESB using the below properties. 1) Global timeout defined in synapse.properties (ESB_HOME\repository\conf\) file. This will decide the maximum time that a callback is waiting in the ESB for a response for a particular request. If ESB does not get any response from Back End, it will drop the message and clears out the call back. This is a global level parameter which affects all the endpoints configured in ESB. synapse.global_timeout_inte