Quite a few organisations have central groups responsible for integration between systems.
I’ve never seen this approach work effectively.
The concept is not inherently flawed, but the implementation can cause more problems than it solves.
But firstly, what is the motivation for creating these groups? Usually:
- Consistency of design and implementation of integration points
- Visibility and governance around integration
None of these objectives are bad ones: in fact, they are all important and necessary in any reasonably complex IT architecture.
And where is the place of middleware and the ESBs of the world in all of this, you might say? Well, there does seem to be a high correlation between the existence of these groups and their becoming the custodian of enterprise integration middleware and the mandate that all integration will be done via said tooling. But the presence or absence of such tools doesn’t really make the problems associated with central integration groups any more or less, as you’ll soon see.
The Root Cause
The main pitfall I see repeatedly with these groups is when the group makes itself (or is made) responsible for both the design and implementation of the integration points. By centralising both the thinking and doing functions, a couple of downstream issues often occur…
The Problem of Distributed Responsibility
Agile delivery teams like stories representing vertical slices through all the technical layers of a system. More importantly, they like to be both accountable and responsible for the delivery of those stories. If a team cannot implement all the layers because integration must be done by another team, then it’s impossible for them to complete their stories end-to-end.
The normal solution is to split out the integration aspect of the story and link the two – as both must be implemented before any business value is delivered.
However, the split stories solution to this problem often causes further complications…
The Problem of Differing Capacity
With a central team responsible for implementing integration points between systems based on requirements supplied by delivery teams, there are three likely scenarios expressing the relative capacity between the two parties:
Here, the integration group has the availability and capacity to deliver the integration functionality in a timely manner. Life is good, although the likelihood of this scenario (remember, there are actually multiple delivery teams consuming this capapcity) is low.
In this case, the integration group has excess capacity and wants to deliver all the integration points as soon as possible, even if the consuming services aren’t ready. There be dragons here due to the lack of feedback from the consuming team that the implemented services are suitable.
Finally, the integration group has insufficient capacity to deliver integration points in time for the delivery groups and becomes a bottleneck and/or a dependency that needs to be satisfied significantly in advance of time to mitigate the risk. Typically, many teams under schedule pressure simply work around any process and standards established by the integration group and write their own connections in a manner which is optimised for their specific needs.
But let’s not dwell too much on the problem – the root cause is not allowing delivery teams to be masters of their own destiny and write their own integration services.
Therefore, the obvious solution is to allow them to do exactly that in compliance with the design established established by the integration team. In other words, narrow the responsibility for these integration teams to one of high level design and trust the delivery teams to implement in line with these. Ideally, make the design use open standards (e.g., SOAP, REST) to provide open source library support for delivery teams and reduce the need for the integration group to police compliance against arbitrary, internal standards.
A logical first step in pushing development responsibility back to the delivery teams would be to second developers from the Integration group into the delivery teams to assist with their first couple of integration stories.
So Integration groups are not necessarily a bad idea, but keep these things in mind:
- Provide guidance to the delivery teams about the overall integration approach, preferably based on open standards
- Give delivery teams the responsibility to implement the integration themselves
- Listen to the feedback from the delivery teams on how suitable the integration design is in practice
Many thanks to Evan Botcher for helping me sort out my thoughts around this topic.