Archive for November, 2009

Validating Incoming Data Using the BizTalk Business Rules Engine

A project team recently asked me if they could use the BizTalk BRE to validate incoming data.  I asked what sort of validation we were talking about, and it came down to four areas:

  • Setting default values when the incoming fields were empty
  • Doing look-ups based on existing data and populating related fields
  • Doing concatenation of fields
  • Catching any business validation errors and recording them

Before answering “of course it can”, I figured I’d quickly roll up my sleeves and prove that the BRE can do those things fairly easily.

The Setup

First, I had to get my environment set up.  This started with a BizTalk XSD schema to represent the data coming into the BRE.  In this scenario, the content is actually metadata about a physical file that arrives into the company.

Notice that I have a “validation errors” record at the bottom with a repeating “error” element.

Next, I have a really simple database table that can be used for lookups.  Based on providing a “study” and “product” value (which comes in via the inbound XML), I want to be able to figure out which “investigator” to add to the message.

Next I need a .NET class to perform a few functions that the BRE doesn’t provide for me out of the box.  First, I need to be able to concatenate two strings.  Yes, it’s ridiculous that I have to write an operation to do this vs. the BRE having something built in.  Deal with it.

public class RuleFunctions
{
   public string ConcatValues(string input1, string input2)
   {
      return input1 + input2;
   }
}

Next, I need a function that can add nodes to my XML message when validation errors are discovered.  After adding references to the Microsoft.RulesEngine.dll and System.Xml, I wrote the following quick-and-dirty code.

public void AddValidationError(TypedXmlDocument activeDoc, string err)
{
   XmlDocument doc = activeDoc.Document.OwnerDocument;

   XmlNode errorRoot = doc.SelectSingleNode("//ValidationErrors");

   XmlElement newError = doc.CreateElement("Error");
   newError.InnerText = err;

   errorRoot.AppendChild(newError);
}

I accept a “TypedXmlDocument”, yank out the underlying XmlDocument, grab a pointer to the “ValidationErrors” record, and append a new node containing whatever error message was defined within the BRE.

After GAC-ing that helper DLL, I’m ready to roll.  Before building my rules in the BRE, I wanted to establish a friendly vocabulary for my XML nodes, database columns and helper functions.  So, I defined a new vocabulary and created definitions for all the “get”, “set” and “function” operations that my rules required.

Setting Default Values

The first requirement was to be able to set default values on nodes.  This one’s pretty easy.  I just check for either an empty string or null value, and set the default value.

Doing Lookups

The next requirement was to take values from the inbound message, perform a lookup and fill in an empty node from the message.  In the rule below, I say that if the value in the XML message equals a value from the database, and another value also equals a value from the database, then set the XML node (Investigator Name) equal to another column in the database.

What this generates behind the scenes when you execute the rule is this:

So the proper T-SQL is built and executed to grab my lookup value.

Performing Concatenation

There’s no out-of-box string concatenation function in the BRE, so here I use the custom class I built earlier.  I want this rule to fire at all times, so my condition is set to something that will always be true (unless math itself starts failing).  I take a static file path location and then append the file name of the document coming into my company and turn that whole thing into an absolute path to the file.

 

Catching and Recording Business Exceptions

The final requirement was to detect business validation errors and record them.  So, we can use my custom built function from earlier to create rules that look for business errors and add nodes to the message itself which outline the error details.

Remember that my custom function takes in a “TypedXmlDocument” type, so I pass the document itself into the function by dragging the root node of my schema from the XML Schemas Fact Explorer view into the operation parameter.

The Result

So what happens when I pass in a document that flexes all these conditions? I start off with an instance document that only has a few values.


So I should see some default values get set, the “investigator” node set based on database lookups, the “file path” node be populated with the concatenated value, and I should see some errors at the end because nodes like “document sub type” and “site” are empty.

To test this rule set, I’m using the built in rule tester, so I have to pass in a valid XML instance, a database connection, and a fact creator that provides the BRE with an instance of my custom .NET class.

 

When the rule policy is finished executing, my input XML is changed to this:

 

You can see that all the rules we built earlier got applied successfully.  So yes, you can use the BRE to do some pretty easy-to-maintain business validation.  This may not be a fit if you’re dealing with a nightly batch of 25,000 records, but if you are doing a messaging based solution, the BRE can do some nice work for you.

Share

My Presentation from Sweden on BizTalk/SOA/Cloud is on Channel9

So my buddy Mikael informs me (actually all of us), that my presentation on “BizTalk, SOA and Leveraging the Cloud” from my visit to the Sweden User Group is finally available for viewing on Microsoft’s Channel9.

In Part 1 of the presentation, I lay the groundwork for doing SOA with BizTalk, and, try to warm up the crowd with stupid American humor.  Then, in Part II I explain how to leverage the Google, Salesforce.com, Azure and Amazon clouds in a BizTalk solution.  Also, either out of sympathy or because my material was improving, you may hear a few more audible chuckles.  I take it where I can get it.

I had lots of fun over there, and will start openly petitioning for a return visit in 6 months or so.  Consider yourself warned, Mikael.

Share

Interview Series: Four Questions With … Lars Wilhelmsen

Welcome to the 14th edition of my interview series with thought leaders in the “connected technology” space.  This month, we are chatting with Lars Wilhelmsen, development lead for his employer KrediNor (Norway), blogger, and Connected Systems MVP.  In case you don’t know, Connected Systems is the younger, sexier sister of the BizTalk MVP, but we still like those cats.  Let’s see how Lars holds up to my questions below.

Q: You recently started a new job where you have the opportunity to use a host of the “Connected System” technologies within your architecture.  When looking across the Microsoft application platform stack, how do you begin to align which capabilities belong in which bucket, and lay out a logical architecture that will make sense for you in the long term?

A: I’m Development Lead. Not a Lead Developer, Solution Architect or Develop  Manager, but a mix of all those three, plus that I put on a variety of other “hats” during a normal day at work. I work close with both the Enterprise Architect and the development team. The dev team consists of “normal” developers, a project manager, a functional architect, an information architect, an tester, a designer and a “man-in-the-middle” whose only task is to “break down” the design into XAML.

We’re on a multi-year mission to turn the business around to meet new legislative challenges & new markets. The current IT system is largely centered around a mainframe-based system, that (at least as we like to think today, in 2009) has too many responsibilities. We seek to use “Components of the Shelf” where we can, but we’ve identified a good set of subsystems that needs to be built from scratch. The strategy defined by the top-level management states that we should seek to use primarily Microsoft technology to implement our new IT platform, but we’re definitely trying to be pragmatic about it. Right now, a lot of the ALT.NET projects gains a lot of usage and support, so even though Microsoft brushes up bits like Entity Framework and Workflow Foundation, we haven’t ruled out the possibility to use non-Microsoft components where we need to. A concrete example is in a new Silverlight –based application we’re developing right now; we evaluated some third party control suites, and in the end we landed on RadControls from Telerik.

Back to the question, I think over time, we will see a lot of the current offerings from Microsoft, either it targets developers, IT Pro’s or the rest of the company in general (Accounting, CRM etc. systems) implemented in our organization, if we find the ROI acceptable. Some of the technologies used by the current development projects include; Silverlight 3, WCF, SQL Server 2008 (DB, SSIS, SSAS) and BizTalk. As we move forward, we will definitely be looking into the next-generation Windows Application Server / IIS 7.5 / “Dublin”, as well as WCF/WF 4.0 (one of the tasks we’ve defined in the near future is a light-weight service bus), and codename “Velocity”.

So,the capabilities we’ve applied so far (and planned) in our enterprise architecture is a mix of both thoroughly tested and bleeding edge technology.

Q: WCF offers a wide range of transport bindings that developers can leverage.  What are you criteria for choosing an appropriate binding, and which ones do you think are the most over-used and under-used?

A: Well, I normally follow a simple set of “Rules of thumbs”;

  • Inter-process: NetNamedPipeBinding
  • Homogenous intranet communication: NetTcpBinding
  • Heterogeneous intranet communication: WSHttpBinding BasicHttpBinding
  • Extranet/Internet communication: WSHttpBinding or BasicHttpBinding

Now, one of the nice thing with WCF is that is possible to expose the same service with multiple endpoints, enabling multi-binding support that is often needed to get all types of consumers to work. But, not all types of binding are orthogonal; the design is often leaky (and the service contract often need to reflect some design issues), like when you need to design a queued service that you’d eventually want to expose with an NetMsmqBinding-enabled endpoint.

Often it boils down to how much effort you’re willing to put in the initial design, and as we all (hopefully) know by now; architectures evolves and new requirements emerge daily.

My first advice to teams that tries to adapt WCF as a technology and service-orientation, is to follow KISS – Keep It Simple, Stupid. There’s often room to improve things later, but if you do it the other way around, you’ll end up with unfinished projects that will be closed down by management.

When it comes to what bindings that are most over- and under-used, it depends. I’ve seen someone that has exposed everything with BasicHttpBinding and no security, in places where they clearly should have at least turned on some kind of encryption and signing.

I’ve also seen highly optimized custom bindings based on WSHttpBinding, with every small little knob adjusted. These services tends to be very hard to consume from other platforms and technologies.

But, the root cause of many problems related to WCF services is not bindings; it is poorly designed services (e.g. service, message, data and fault contracts). Ideally, people should probably do contract-first (WSDL/XSD), but being pragmatic I tend to advice people to design their WCF contracts right (if in fact, they’re using WCF). One of the worst thing I see, is service operations that accepts more than one input parameter. People should follow the “At most one message in – at most one message out” pattern. From a versioning perspective, multiple input arguments are the #1 show stopper. If people use message & data contracts correctly and implements the IExtensibleDataObject, it is much easier in the future to actually version the services.

Q: It looks like you’ll be coming to Los Angeles for the Microsoft Professional Developers Conference this year.  Which topics are you most keen to hear about and what information do you hope to return to Norway with?

A: It shouldn’t come as a surprise, but as a Connected Systems MVP, I’m most excited about the technologies from that department (Well, they’ve merged now with the Data Platform people, but I still refer to that part of MSFT as Connected Systems Div.). WCF/WF 4.0 will definitely get a large part of my attention, as well as Codename “Dublin” and Codename “Oslo”. I will also try to watch the ADFSv2 [Formerly known as Codename “Geneva”] sessions. Apart from that, I hope to use a lot of time talking to other people. MSFTies, MVPs and other people. To “fill up” the schedule, I will probably try to attend some of the (for me) more exoteric sessions about Axum, Rx framework, parallelization etc.

Workflow 3.0/3.5 was (in my book) a more or less a complete failure, and I’m excited that it seems like Microsoft has taken the hint from the market again. Hopefully WF 4.0, or WF 3.0 as it really should be called (Microsoft product seems to reach maturity first at version 3.0), will hopefully be a useful technology that we’ll be able to utilize in some of our projects. Some processes are state machines, some places do we need to call out in parallel to multiple services – and be able to compensate if something goes wrong, and other places do we need a rule engine.

Another thing we’d like to investigate more thorough, is the possibility to implement claims-based security in many of our services, so (for example) can federate with our large partners. This will enable “self service” of their own users that access our Line of Business applications via the Internet.

A more long term goal (of mine, so far) is definitely to use the different part of codename “Oslo” – the modeling capabilities, the repository and MGrammar – to create custom DSLs in our business. We try to be early adopters of a lot of the new Microsoft technologies, but we’re not about to try to push things into production without a “Go-Live” license.

Q [stupid question]: This past year you received your first Microsoft MVP designation for your work in Connected Systems.  There are a surprising number of technologies that have MVPs, but they could always use a few more such as a Notepad MVP, Vista Start Menu MVP or Microsoft Word “About Box” MVP.  Give me a few obscure/silly MVP possibilities that Microsoft could add to the fold.

A: Well, I’ve seen a lot of middle-aged++ people during my career that could easily fit into a “Solitaire MVP” category :-) Fun aside, I’m a bit curious why Microsoft have Zune & XBox MVP titles. Last time I checked, the P was for “Professional” I can hardly imagine anyone who gets paid for listening to their Zune or playing on their XBox. Now, I don’t mean  to offend the Zune & XBox MVPs, because I know they’re brilliant at what they do, but Microsoft should probably have a different badge to award people that are great at leisure activities, that’s all.

Thanks Lars for a good chat.


Disclaimer

Entries and comments here do not necessarily reflect the opinions, attitudes, and statements of my employer, my friends, or anyone associated with me.

Syndication

Publications

Order my new book SOA Patterns with BizTalk Server 2009 (Amazon.com, Packt Publishing)

Contact Me

Categories

Twitter Feed

Blog Stats

  • 225,044