ESB Toolkit: Executing Multiple Maps In Sequence

There are a few capabilities advertised in the Microsoft ESB Toolkit for BizTalk Server that I have yet to try out.  One thing that seemed possible, although I hadn’t seen demonstrated, was the ability to sequentially call a set of BizTalk maps.

Let’s say that you have maps from “Format1 to Format2″ and “Format2 to Format3.”  These are already deployed and running live in production.  Along comes a new scenario where a message comes in and must be transformed from Format1 to Format3. 

There are a few “classic BizTalk” ways to handle this.  First, you could apply one map on the receive port and another on the send.  Not bad, but this definitely means that this particular receive port can’t be one reused from another solution as this could cause unintended side effects on others.  Second, you could write an orchestration that takes the inbound message and applies consecutive maps.  This is common, but also requires new bits to be deployed into production.  Thirdly, you could write a new map that directly transforms from Format1 to Format3.  This also requires new bits, and, may force you to consolidate transformation logic that was unique to each map.

So what’s the ESB way to do it?  If we see BizTalk as now just a set of services, we can build up an itinerary that directs the bus to execute a countless set of consecutive maps, each as a distinct service.  This is a cool paradigm that allows me to reuse existing content more freely than before by introducing new ways to connect components that weren’t originally chained together.

First, we make sure our existing maps are deployed.  In my case, I have two maps that follow the example given above.

I’ve also gone ahead and created a new receive port/location and send port for this demonstration.  Note that I could have also added a new receive location to an existing receive port.  The ESB service execution is localized to the specific receive location, unlike the “classic BizTalk” model where maps are applied across all of the receive locations.  My dynamic send port has a ESB-friendly subscription.

We’ll look at the receive location settings in a moment.  First, let’s create the itinerary that makes this magic happen.  The initial shape in our itinerary is the On-Ramp.  Here, I tell the itinerary to use my new receive port.

Next, I set up a messaging service that the Off-Ramp will use to get its destination URI.  In my  case, I used a STATIC resolver that exploits the FILE adapter and specifies a valid file path.

Now the games begin.  I next added a new messaging service which is used for transformation.  I set another STATIC resolver, and chose the “Format1 to Format2″ map deployed in my application.

Then we add yet another transformation messaging service, this time telling the STATIC resolver to apply the “Format2 to Format3″ map.

Great.  Finally, we need an Off-Ramp.  We then associate the three previous shapes (messaging service and two transformation services) with this Off-Ramp.  Be sure to verify that the order of transformation resolvers is correct in the Off-Ramp.  You don’t want to accidentally execute the “Format2 to Format3″ map first!

Once our itinerary is connected up and ready to roll, we switch the itinerary status to “deployed” in the itinerary’s property window.  This ensures that ESB runtime can find this itinerary when it needs it.  To publish the itinerary to the common database, simply chose “Export Model.”

Fantastic.  Now let’s make sure our BizTalk messaging components are up to snuff.  First, open the FILE receive location and make sure that the ItinerarySelectReceiveXml pipeline is chosen.  Then open the pipeline configuration window and set the resolver key and resolver string.  The itinerary factkey is usually “Resolver.Itinerary” (which tells the pipeline in which resolver object property to find the XML itinerary content) and the resolver connection string itself is ITINERARY-STATIC:\\name=DoubleMap;  The ITINERARY-STATIC directive enables me to do server-side itinerary lookup.  It’ll use the name provided to find my itinerary record in the database and yank out the XML content.  Note that I used a FILE receive location here.  These ESB pipeline components can be used with ANY inbound adapter which really increases the avenues for publishing itinerary-bound messages to the bus.

Finally, go to the dynamic send port and make sure the ItinerarySendPassthrough pipeline is chosen.  We need to ensure that the ESB services (like transformation) have a context in which to run.  If you only had the standard passthrough pipeline selected here, you’d be subtracting the environment (pipelines) in which the ESB components do much of their work.

That is it.  If we drop a “Format1″ message in, we get a “Format3″ message out.  And all of this, POTENTIALLY, without deploying a single new BizTalk component.  That said, you may still need to create a new dynamic send port if you don’t already have one reuse, and would probably want to create a new receive location, OR, if the itinerary was being looked up via the business rules engine (BRI resolver), then you could just update the existing business rule.  Either way, this is a pretty quick and easy way to do something that wasn’t quick and easy before.

 Technorati Tags: BizTalk, ESB

Can Software Architecture Attributes Also Be Applied to Business Processes?

I’m in San Diego attending the Drug Information Association conference with the goal of getting smarter in the functional areas that make up a bio-pharma company.  I’m here with two exceptional architecture colleagues which means that most meals have consisted of us talking shop. 

During dinner tonight, we were discussing the importance (or imperative, really) of having a central business champion that can envision what they need and communicate that vision to the technical team.  The technical team shouldn’t be telling the business what their vision is.

Within that conversation, we talked about the value of having good business analysts who deeply understand the business and are in the position to offer actual improvements to the processes they uncover and document.  I then asked if it’s valid to hijack many of the attributes that architects think about in the confines of a technical solution, but also have them applied by a business analyst to a business process.  Maybe it’s crazy, but on first pass, most of the solutions architecture things I spend my day thinking about have direct correlation to what a good business process should address or mitigate as well:

• Scalability.  How well does my process handle an increase in input requests?  Is it built to allow for us to ramp up personnel or are there eventual bottlenecks we need to consider now?
• Flexibility.  Can my process support modifications in sequencing or personnel?  Or did we define a process that only works in a rigid order with little room for the slightest tweak?
• Reusability. Is the process modular enough that an entire series of steps could be leveraged by another organization that has an identical need?
• Encapsulation.  If I’ve chained processes together, have I insulated each one from another so that fundamental internal modifications to one process doesn’t necessarily force a remodeling of a connected process?
• Security.  Have I defined the explicit roles of the users in my process and identified who can see (or act on) what information as the process moves through its lifecycle?
• Maintainability.  Is the process efficient and supportable in the long term?
• Availability.  If someone is sick for two weeks, does the process grind to a halt?  What if a key step in the process itself cannot be completed for a given period of time?  What’s the impact of that?
• Concurrency.  What happens if multiple people want to work on different pieces of the same process simultaneously?  Should the process support this or does it require a sequential flow?
• Globalization/localization.  Can this process be applied to a global work force or conversely, does the process allow for local nuances and modifications to be added?

Just like with solutions architecture, where you often may trade one attribute for another (e.g. “I’ll pick a solution which give up efficiency because I demand extreme flexibility”), the same can apply to a well-considered business process. 

So what do you think?  Do the business analysts you work with think along these lines?  Are we properly “future-proofing” our business processes or are we simply documenting the status quo without enough rigor around quality attributes and a vision around the inevitable business/industry changes?  I’ll admit that I haven’t done a significant amount of business process modeling in my career so maybe everyone already does this.  But, I haven’t seen much of this type of analysis in my current environment.

Or, I just ate too much chicken tikka masala tonight and am completely whacked out.

Books I’ve Recently Finished Reading

Other obligations have quieted down over the past two months and I’ve been able to get back to some voracious reading.  I thought I’d point out a few of the books that I’ve recently knocked out, and let you know what I think of them.

• SOA Governance.  This is a book from Todd Biske and published by my book’s publisher, Packt.  It follows a make-believe company through their efforts to establish SOA best practices at their organization.  Now, that doesn’t mean that the book reads like a novel, but, this isn’t a “reference book” to me as much as an “ideas” book.  When I finished it, I had a better sense of the behavioral changes, roles required and processes that I should consider when evangelizing SOA behavior in my own company.  Todd does a good job identifying the underlying motivations of the people that will enable SOA to succeed or fail within a company.  You’ll find some useful thinking around identifying the “right” services, versioning considerations, SLA definition, and even some useful checklists to verify if you’re asking the right questions at each phase of the service lifecycle.  Whether you’re “doing SOA” or not, this is a easy read that can help you better digest the needs of stakeholders in an enterprise software solution.
• Mashup Patterns : Designs and Examples for the Modern Enterprise.  I’ve been spending a fair amount of time digging into mashups lately, and it was great to see a book on the topic come out.  The author breaks down the key aspects of designing a mashup (harvesting data, enriching data, assembling results and managing the deliverable).  Each of the 30+ patterns is comprised of: (a) a problem statement that describes the issue at hand, (b) a conceptual solution to the problem, (c) a “fragility score” which indicates how brittle the solution is, (d) and finally 2 or more examples where this solution is applied to a very specific case.  The examples for each pattern are where I found the most value.  This helped drive home the problem being solved and provided a bit more meat on the conceptual solution being offered.  That said, don’t expect this book to tell you WHAT can help you create these solutions.  There is very much the tone of “we just need to get this data from here, combine it with this, and even our business analyst can do it!” However, nowhere does the author dig into how all this MAGIC really happens (e.g. products, tools, etc).  That was the only weakness of the book to me.  Otherwise, this was quite a well put together book that added a few things to my arsenal of options when architecting solutions.
• Thank You for Arguing: What Aristotle, Lincoln, and Homer Simpson Can Teach Us About the Art of Persuasion.  I really enjoyed reading this.  In essence, it’s a look at the lost art of rhetoric and covers a wide set of tools we can use to better frame an argument and win it.  The author has a great sense of humor and I found myself actually taking notes while reading the book (which I never really do).  There are a mix of common sense techniques for setting up your own case, but I also found the parts outlining how to spot a bad argument quite interesting.  So, if you want to get noticeably better at persuading others and also become more effective at identifying when someone’s trying to bamboozle you, definitely pick this up.
• Leaving Microsoft to Change the World.  A co-worker suggested this book to me.  It’s the story of John Wood, a former Microsoft executive during the 90s glory days, who chucked his comfortable lifestyle and started a non-profit organization (Room to Read) with the mission of improving education in the poorest countries in the world.  John’s epiphany came during a backpacking trip through Nepal and seeing the shocking lack of reading materials available to kids who desperately wanted to learn and lift themselves out of poverty.  Even if the topic doesn’t move you, this book has a fascinating look at how to start up a global organization with a focused objective and a shoestring budget.  This is one of those “perspective books” that I try and make sure I read from time to time.
• Microsoft .NET: Architecting Applications for the Enterprise.  I actually had this book sent to me by a friend at Microsoft.  Authored by Dino Esposito and Andrea Saltarello, this is an excellent look at software architecture.  It starts off with a very clear summary of what architecture really is, and raised  a point that struck home for me: architecture should be about the “hard decisions.”  An architect isn’t going to typically get into the weeds on every project, but instead should be seeking out the trickiest or most critical parts of a proposed solution and focus their energies there.  The book contains a good summary of core architecture patterns and spends much of the time digging into how to design a business layer, data access layer, service layer, and presentation layer.  Clearly this book has a Microsoft bent, but, don’t discount it as a valid introduction to architecture for any technologist.  They address a wide set of core principles that are technology agnostic in a well-written fashion.

I’m trying to queue up some books for my company’s annual “summer shutdown” and always looking for suggestions.   Technology, sports, erotic thrillers, you name it.

Four Ways to Accept Any XML Data Into BizTalk Web Services

I knew of three techniques for creating generic service on-ramps into BizTalk, but last night learned of a fourth.

So what if you want to create an untyped web service endpoint that can accept any valid XML message?  I previously knew of three choices:

• Orchestration with XmlDocument message type.  Here you create a throw-away orchestration which takes in an XmlDocument.  Then you walk through the service publishing wizard and create a service out of this orchestration.  Once you have the service, you can discard the originating orchestration.  I seem to recall that this only works with the ASMX publishing wizard.

• Create wrapper schema around “any” node.  In this case, you build a single schema that has a child node of type “any.”  Then, you can use the “publish schemas as web service” option of the publishing wizards to create a general purpose service on ramp.  If you’re using WCF receive locations, you can always strip out the wrapper node before publishing to the MessageBox.

• Custom WSDL on generated service.  For BizTalk WCF-based services, you can now attach custom WSDLs to a given endpoint.  I cover this in my book, but in a nutshell, you can create any WCF endpoint using the publishing wizard, and then set the “externalMetadataLocation” property on the Metadata behavior. 

So that’s all well and good.  BizTalk service endpoints in WCF are naturally type-less, so it’s a bit easier to muck around with those exposed interface definitions than when dealing with ASMX services. 

That said, last night I was watching the third in Peter Kelcey’s video series on the ESB Toolkit, and he slipped in a new way (to me) for building generic service endpoints.  Simply start up the BizTalk WCF Service Publishing Wizard, choose to publish schemas as a service, and when choosing the message type of the contract, you browse to C:\Program Files\Microsoft BizTalk Server 2009 and pick the Microsoft.XLANGs.BaseTypes.dll.

Once you do that, you can actually pick the “any” schema type that BizTalk defines.

Once you finish the wizard (assuming you chose to create a metadata endpoint), you’ll have a WSDL that has your custom-defined operation which accepts an “any” message.

So there you go.  Maybe this was common knowledge, but it was news to me.  That’s a pretty slick way to go.  Thanks Peter.

Technorati Tags: BizTalk

ESB Toolkit Out and About

Congrats to the BizTalk team for getting the ESB Toolkit out the door.    This marks a serious milestone in this package.  No longer just a CodePlex set of bits (albeit a rich one), but now a supported toolkit (download here) with real Microsoft ownership.  Check out the MSDN page for lots more on what’s in the Toolkit.

I’ve dedicated a chapter to the Toolkit in my book, and also recently recorded a webcast on it.  You’ll see that online shortly.  Also, the upcoming Pro BizTalk 2009 book, which I’m the technical reviewer for, has a really great chapter on it by the talented Peter Kelcey.

The main message with this toolkit is that you do NOT have to install and use the whole thing.  Want dynamic transformation as a standalone service?  Go for it.  Need to resolve endpoints and metadata on the fly?  Try the resolver service.  Looking for a standard mechanism to capture and report on exceptions?  Take a look at the exception framework.  And so on. 

Technorati Tags: BizTalk, ESB

Interview Series: Four Questions With … Charles Young

This month’s chat in my ongoing series of discussions with “connected systems” thought leaders is with Charles Young.  Charles is a steady blogger, Microsoft MVP, consultant for Solidsoft Ltd,  and all-around exceptional technologist. 

Those of you who read Charles’ blog regularly know that he is famous for his articles of staggering depth which leave the reader both exhausted and noticeably smarter.  That’s a fair trade off to me.

Let’s see how Charles fares as he tackles my Four Questions.

Q: I was thrilled that you were a technical reviewer of my recent book on applying SOA patterns to BizTalk solutions.  Was there anything new that you learned during the read of my drafts?  Related to the book’s topic, how do you convince EAI-oriented BizTalk developers to think in a more “service bus” sort of way?

A: Well, actually, it was very useful to read the book.    I haven’t really had as much real-world experience as I would like of using the WCF features introduced in BTS 2006 R2.   The book has a lot of really useful tips and potential pitfalls that are, I assume, drawn from real life experience.    That kind of information is hugely valuable to readers…and reviewers.

With regard to service buses, developers tend to be very wary of TLAs like ‘ESB’.  My experience has been that IT management are often quicker to understand the potential benefits of implementing service bus patterns, and that it is the developers who take some convincing.   IT managers and architects are thinking about overall strategy, whereas the developers are wondering how they are going to deliver on the requirements of the current project   I generally emphasise that ‘ESB’ is about two things –first, it is about looking at the bigger picture, understanding how you can exploit BizTalk effectively alongside other technologies like WCF and WF to get synergy between these different technologies, and second, it is about first-class exploitation of the more dynamic capabilities of BizTalk Server.   If the BizTalk developer is experienced, they will understand that the more straight-forward approaches they use often fail to eliminate some of the more subtle coupling that may exist between different parts of their BizTalk solution.    Relating ESB to previously-experienced pain is often a good way to go.

Another consideration is that, although BizTalk has very powerful dynamic capabilities, the basic product hasn’t previously provided the kind of additional tooling and metaphors that make it easy to ‘think’ and implement ESB patterns.   Developers have enough on their plates already without having to hand-craft additional code to do things like endpoint resolution.   That’s why the ESB Toolkit (due for a new release in a few weeks) is so important to BizTalk, and why, although it’s open source, Microsoft are treating it as part of the product.   You need these kinds of frameworks if you are going to convince BizTalk developers to ‘think’ ESB.

Q: You’ve written extensively on the fundamentals of business rules and recently published a thorough assessment of complex event processing (CEP) principles.  These are two areas that a Microsoft-centric manager/developer/architect may be relatively unfamiliar given Microsoft’s limited investment in these spaces (so far).  Including these, if you’d like, what are some industry technologies that interest you but don’t have much mind share in the Microsoft world yet?  How do you describe these to others?

A: Well, I’ve had something of a focus on rules for some time, and more recently I’ve got very interested in CEP, which is, in part, a rules-based approach.    Rule processing is a huge subject.   People get lost in the detail of different types of rules and different applications of rule processing.   There is also a degree of cynicism about using specialised tooling to handle rules.   The point, though, is that the ability to automate business processes makes little sense unless you have a first-class capability to externalise business and technical policies and cleanly separate them from your process models, workflows and integration layers.    Failure to separate policy leads directly to the kind of coupling that plagues so may solutions.   When a policy changes, huge expense is incurred in having to amend and change the implemented business processes, even though the process model may not have changed at all.   So, with my technical architect’s hat on, rule processing technology is about effective separation of concerns.

If readers remain unconvinced about the importance of rules processing, consider that BizTalk Server is built four-square on a rules engine – we call it the ‘pub-sub’ subscription model which is exploited via the message agent.   It is fundamental to decoupling of services and systems in BizTalk.    Subscription rules are externalised and held in a set of database tables.   BizTalk Server provides a wide range of facilities via its development and administrative tools for configuring and managing subscription rules.   A really interesting feature is that way that BizTalk Server injects subscription rules dynamically into the run-time environment to handle things like correlation onto existing orchestration instances.

Externalisation of rules is enabled through the use of good frameworks, repositories and tooling.    There is a sense in which rule engine technology itself is of secondary importance.   Unfortunately, no one has yet quite worked out how to fully separate the representation of rules from the technology that is used to process and apply rules.   MS BRE uses the Rete algorithm.   WF Rules adopts a sequential approach with optional forward chaining.    My argument has been that there is little point in Microsoft investing in a rules processing technology (say WF Rules) unless they are also prepared to invest in the frameworks, tooling and repositories that enable effective use of rules engines.

As far as CEP is concerned, I can’t do justice to that subject here.   CEP is all about the value bound up in the inferences we can draw from analysis of diverse events.   Events, themselves, are fundamental to human experience, locked as we are in time and space.   Today, CEP is chiefly associated with distinct verticals – algorithmic trading systems in investment banks, RFID-based manufacturing processes, etc.   Tomorrow, I expect it will have increasingly wider application alongside various forms of analytics, knowledge-based systems and advanced processing.   Ironically, this will only happen if we figure how to make it really simple to deal with complexity.  If we do that, then with the massive amount of cheap computing resource that will be available in the next few years all kinds of approaches that used to be niche interests, or which were pursed only in academia, will begin to come together and enter the mainstream.   When customers start clamouring for CEP facilities and advanced analytics in order to remain competitive, companies like Microsoft will start to deliver.   It’s already beginning to happen.

Q: If we assume that good architects (like yourself) do not live in a world of uncompromising absolutes, but rather understand that the answer to most technical questions contain “it depends”, what is an example of a BizTalk solution you’ve built that might raise the eyebrows of those without proper context, but make total sense given the client scenario.

A: It would have been easier to answer the opposite question.   I can think of one or two BizTalk applications where I wish I had designed things differently, but where no one has ever raised an eyebrow.   If it works, no one tends to complain!

To answer your question, though, one of the more complex designs I worked on was for a scenario where the BizTalk system had only to handle a few hundred distinct activities a day, but where an individual message might represent a transaction worth many millions of pounds (I’m UK-based).   The complexity lay in the many different processes and sub-processes that were involved in handling different transactions and business lines, the fact that each business activity involved a redemption period that might extend for a few days, or as long as a year, and the likelihood that parts of the process would change during that period, requiring dynamic decisions to be made as to exactly which version of which sub-process must be invoked in any given situation.   The process design was labyrinthine, but we needed to ensure that the implementation of the automated processes was entirely conformant to the detailed process designs provided by the business analysts.   That meant traceability, not just in terms of runtime messages and processing, but also in terms of mapping orchestration implementation directly back to the higher-level process definitions.    I therefore took the view that the best design was a deeply layered approach in which top-level orchestrations were constructed with little more that Group and Send orchestration shapes, together with some Decision and Loop shapes, in order to mirror the highest-level process definition diagrams as closely as possible.   These top-level orchestrations would then call into the next layer of orchestrations which again closely resembled process definition diagrams at the next level of detail.   This pattern was repeated to create a relatively deep tree structure of orchestrations that had to be navigated in order to get to the finest-level of functional granularity.    Because the non-functional requirements were so light-weight (a very low volume of messages with no need for sub-second responses, or anything like that), and because the emphasis was on correctness and strict conformance process definition and policy, I traded the complexity of this deep structure against the ability to trace very precisely from requirements and design through to implementation and the facility to dynamically resolve exactly which version of which sub-process would be invoked in any given situation using business rules.

I’ve never designed any other BizTalk application in quite the same way, and I think anyone taking a casual look at it would wonder which planet I hail from.   I’m the first to admit the design looked horribly over-engineered, but I would strongly maintain that it was the most appropriate approach given the requirements.   Actually, thinking about it, there was one other project where I initially came up with something like a mini-version of that design.   In the end, we discovered that the true requirements were not as complex as the organisation had originally believed, and the design was therefore greatly simplified…by a colleague of mine…who never lets me forget!

Q [stupid question]: While I’ve followed Twitter’s progress since the beginning, I’ve resisted signing up for as long as I can. You on the other hand have taken the plunge.  While there is value to be extracted by this type of service, it’s also ripe for the surreal and ridiculous (e.g. Tweets sent from toilets, a cat with 500,000 followers).  Provide an example of a made-up silly use of a Twitter account.

A: I resisted Twitter for ages.   Now I’m hooked.   It’s a benign form of telepathy – you listen in on other people’s thoughts, but only on their terms.    My suggestion for a Twitter application?   Well, that would have to be marrying Wolfram|Alpha to Twitter, using CEP and rules engine technologies, of course.    Instead of waiting for Wolfram and his team to manually add enough sources of general knowledge to make his system in any way useful to the average person, I envisage a radical departure in which knowledge is derived by direct inference drawn from the vast number of Twitter ‘events’ that are available.   Each tweet represents a discrete happening in the domain of human consciousness, allowing us to tap directly into the very heart of the global cerebral cortex.   All Wolfram’s team need to so is spend their days composing as many Twitter searches as they can think of and plugging them into a CEP engine together with some clever inferencing rules.   The result will be a vast stream of knowledge that will emerge ready for direct delivery via Wolfram’s computation engine.   Instead of being limited to comparative analysis of the average height of people in different countries whose second name starts with “P”, this vastly expanded knowledge base will draw only on information that has proven relevance to the human race – tweet epigrams, amusing web sites to visit, ‘succinct’ ideas for politicians to ponder and endless insight into the lives of celebrities.

Thanks Charles.  Insightful as always.

Technorati Tags: BizTalk