Everyone’s doing integration nowadays. It’s not just the grizzled vet who reminisces about EDI or the seasoned DBA who can design snowflake schemas for a data warehouse in their sleep. No, now we have data scientists mashing up data sources, developers processing streams and connecting things via APIs, and “citizen integrators” (non-technical users) building event-driven actions on their own. It’s wild.
Here, I’ll take a look at a few things to keep an eye on, and the implications for you.
iPaaS
Research company Gartner coined the term Integration Platform-as-a-Service (iPaaS) to represent services that offer application integration capabilities in the cloud. Gartner delivers an annual assessment of these vendors in the form of a Magic Quadrant, and released the 2016 version back in March. While revenue in this space is still relatively small, the market is growing by 50%, and Gartner predicts that by 2019, iPaaS will be the preferred option for new projects. Kent Weare recently conducted an excellent InfoQ virtual panel about iPaaS with representatives from SnapLogic, Microsoft, and Mulesoft. I found a number of useful tidbits in there, and drew a few conclusions:
- The vendors are pushing their own endpoint connectors, but all seem to (somewhat grudgingly) recognize the value of consume “raw” APIs without a forced abstraction.
- An iPaaS model won’t take off unless it’s seen as viable for existing, on-premises systems. Latency and security matter, and it still seems like there’s work to be done here to ensure that iPaaS products can handle all the speed and connectivity requirements.
- Elasticity is an increasingly important value proposition of iPaaS. Instead of trying to build out a complete integration stack themselves that handle peak traffic, companies want something that dynamically scales. This is especially true given that Internet-of-Things is seem as a huge driver of iPaaS in the years ahead.
- User experience is more important than ever, and these vendors are paying special attention to the graphical UI. At the same time, they’ll need to keep working on the technical interface for things like automated testing. They seemed well positioned, however, to work with new types of transient microservices and short-lived containers.
There’s some cool stuff in the iPaaS space. It’s definitely worth your time to read Kent’s panel and explore some of these technologies more closely.
Microservices-driven integration
Have you heard of “microservices”? Of course you have, unless you’ve been asleep for the past eighteen months. This model of single-purpose, independently deployable services has taken off as groups rebel against the monolithic apps (and teams!) they’re saddled with today.
You’ll often hear microservices proponents question the usefulness of an Enterprise Service Bus. Why? They point out that ESBs are typically managed in organization silos, centralize too much of the processing, and offer much more functionality than most teams need. If you look at your application components arranged as a graph instead of a stack, then you realize a different integration toolset is needed.
Back in May, I was at Integrate 2016 where I delivered a talk on the Open Source Messaging Landscape (video now online). Lightweight messaging is BACK, baby! I’m seeing more and more teams looking to (open source) distributed messaging solutions when connecting their disparate services. This means software like Kafka, RabbitMQ, ZeroMQ, and NATS are going to continue to increase in relevance in the months and years ahead.
How are you supposed to orchestrate all these microservices integrations? That’s not an easy answer. One technology that’s trying to address this is Spring Cloud Data Flow. I saw a demo a couple week back, and walked away very impressed.
Spring Cloud Data Flow is software that helps you create and run pipelines of data microservices. These microservices are loosely coupled but linked through a shared messaging layer and sit atop a variety of runtimes including Kubernetes, Apache Mesos, and Cloud Foundry. This gives you a very cool way to design and run modern system integration.
Serverless and “citizen integrators”
Microservices is a hot trend, but “serverless” is probably even hotter! A more accurate name is “function as a service” and engineer Mike Roberts wrote a great piece that covers criteria and use cases. Basically, it’s about running short-lived, often asynchronous, single operations without any knowledge about the underlying infrastructure.
This matters for integration people not just because you’ll see more and more messaging-oriented scenarios interacting with serverless engines, but because it’s opened up the door for many citizen developers to build event-driven integrations. These integration services meet the definition of “serverless” with their pay-pay-use, short-lived actions that abstract the infrastructure.
Look at the crazy popularity of IFTTT. “Regular” people can design pretty darn powerful integrations that start with an external trigger and end with an action. This stuff isn’t just for making automatic updates to Pinterest. Have you investigated Zapier? Their directory of connectors contains an impressive array of leading CRM, Finance, and Support systems that anyone can use. Microsoft’s in the game now with Flow for simple cloud-based workflows. Developers can take advantage of serverless products like AWS Lambda and Webtask (from Auth0) when custom code is needed.
Implications
What does all this mean to you? First and foremost, integration is hot again! I’m willing to bet that you’d benefit from investing some time in learning new and emerging tech. If you haven’t learned anything new in the integration space over the past two years, you’ve missed a lot. Take a course, pick up a book, or just hack around.
Recognize the growth of microservices and think about how it impacts your team. What tools will developers use to connect their services? What needs to be upgraded? What does this type of distributed integration do to your tracing and troubleshooting procedures? How can you break down the organizational silos and keep the “integration team” from being a bottleneck? Take a look at messaging software that can complement existing ESB software.
Finally, don’t miss out on this “citizen integrator” trend. How can you help your less technical colleagues connect their systems in novel ways? The world will always need integration specialists, but it’s important to support the growing needs of those who shouldn’t have to queue up for help from the experts.
What do you think? Any integration trends that stand out to you?
Richard Seroter's Architecture Musings 
2 thoughts