Building Integration Solutions : A Rethink

History of Enterprise Integration

The history of enterprise integration goes back to early computer era where we had computers only in large enterprises. The early requirements came from the concept of Material Requirement Planning (MRP) where it requires a system to plan and control production and material flows. With the growth of the businesses and the interactions among different 3rd party organizations, MRP has been evolved into an Enterprise Resource Planning (ERP) systems where they are responsible for much more functionalities to bridge the different departments of the enterprise like accounting, finance, engineering, product management and many more. Proprietary ERP solutions were dealing with so many complex use cases and failed really big in some cases. With these lessons, people realized that there should be a better to way to build the enterprise IT infrastructure beyond the ERP systems.

Integration and SOA

Service Oriented Architecture (SOA) comes into the picture in a time where the world is searching for a proper way to handle their complex enterprise IT requirements. The Wikipedia definition of the SOA is like below.

“A service-oriented architecture (SOA) is an architectural pattern in computer software design in which application components provide services to other components via a communications protocol, typically over a network. The principles of service-orientation are independent of any vendor, product or technology”

Rather than having a proprietary system in your enterprise, SOA has built a set of loosely coupled independent services to interact with each other and provide the business functionality to other systems/users. With the concepts of loosely coupled services came the concept of integration where we need to connect with other services to provide the business functionality. At the early stages, it was only a peer to peer communication between services. This has lead to the complex “spaghetti” integration pattern.  

If you have 10 services in your system, you may need 45 point to point connections to communicate with all of the other services. Rather than connecting the services point to point, we can connect them to a central “Bus” and do the communication over that.

The Integration Era

Once people realized the value of SOA and the integration, enterprises started moving into that space more and more than the ERP systems and it became a common architectural pattern in most enterprises. Then came the Enterprise Service Bus (ESB) concept where you connect your all the disparate systems to the central bus and made the interaction possible across different services.

Same type of service has been provided by so many different vendors and the standards around SOA has been emerged. People started thinking about common standards in a more serious manner and all the monopolies existed on the world of software has been little by little converged into common standards. Innovative ideas came into the picture and became standards and the integration space has been emerged as a challenging technology domain. Different wire level protocols, messaging formats, enterprise messaging patterns evolved with the heavy usage of SOA and integration in the enterprises. Almost all the big software vendors  has released their own products for application integration and this has become a billion dollar business.

Beyond Integration

The technology industry has been a moving target since its inception and the pace of the movement might have been varied time to time. At the moment, we are in a time where that pace has been increased and there are lot of new concepts taking over the technology industry. Integration has been pushed to the backyards and the new technology concepts like Micro Services Architecture (MSA), Internet Of Things (IOT), Big Data and Analytics have  been taking over the world of technology.  But any of these concepts are not going to fill the same bucket as integration. They are independent concepts which has surfaced with the increased usage of technology in people’s day to day activity. But the important thing is that Integration cannot live without thinking about these trends. The below diagram depicts the interaction between MSA and Integration Bus in a real enterprise IT system. This was captured from the blog post written by Kasun Indrasiri at [1].

Figure 4: MSA and Integration Server in modern enterprise

Integration for the future

Integration has been a complex subject from the beginning and it has been able to tackle most of the integration requirements popped up in enterprise IT infrastructures. But with the  advancement of other areas, integration solutions need to pay more attention to the following emerging concepts in the future and become more and more “lean”.

1. Enterprise architects looking for vendor neutral solutions  - Integration has been an area where you need to have not only domain experts but vendor experts to succeed. But the world is more and more moving towards domain expertise and vendor neutrality. Which means that enterprise architects always looking for solutions which can be easily replaceable with a different vendor.
2. Integration solutions needs to be more user friendly - Architects want to see their integrations more clearly and with a more visually pleasing manner. They don’t want to read through thousands of XML files to understand a simple integration flow.
3. Internet Of Things (IOT) will hit you very soon - Your solution needs to be able to accommodate IOT protocols and concepts as first class features.
4. No longer sitting inside enterprise boundary -  Enterprises are moving more and more towards cloud based solutions and your solution needs to be run on the cloud while interacting with other cloud services
5. Ability to divide will matter - Users will want to replace parts of your system with some other components which they have been using for longer time and worked for them. Your system should be able to compose into different independent components and be able to work in tandem with other systems.
6. There will be more than “systems” to integrate - Integration has been dealing with different systems in the past and the future would be much different with the concepts of MSA where you have business functions exposed as services and there can be some other components like data, IOT gateways, smart cars you need to integrate. Better to prepare as early as possible.
7. Make room to inject “intelligence” into your solution - Enterprises would like to inject some intelligence through the concepts like analytics and predictions to your integration solution which is the core of your enterprise IT infrastructure.

References:

[1] Microservices In modern enterprise architecture