The Good, Bad and Ugly of Integrating Dynamics CRM 2011 and BizTalk Server 2010

Microsoft Dynamics CRM 2011 is the latest version of Microsoft’s CRM platform.  The SaaS version is already live and the on-site version will likely be released within a couple weeks.  Unlike previous versions of Dynamics CRM, the 2011 release does NOT have a BizTalk-specific send adapter.  The stated guidance is to use the existing SOAP endpoints through the BizTalk WCF adapter.  So what is this experience like?  In a word, mixed.  In this post, I’ll show you what it takes to perform both “query” and “create” operations against Dynamics CRM 2011 using BizTalk Server.

Before I start, I’ll say that I really like using Dynamics CRM 2011.  It’s a marked improvement over the previous version (CRM 4) and is a very simple to use application platform.  I’m the architect of a project that is leveraging it and am a fan overall.  It competes directly with Salesforce.com, which I also like very much, and has areas where it is better and areas where it is worse.  I’ll say up front that I think the integration between Salesforce.com and BizTalk is MUCH cleaner than the integration between Dynamics CRM 2011 and BizTalk, but see if you agree with me after this post.

Integration Strategies

Right up front, you have a choice to make.  Now, I’m working against a Release Candidate, so there’s a chance that things change by the formal release but I doubt it.  Dynamics CRM 2011 has a diverse set of integration options (see MSDN page on Web Service integration here).  They have a very nice REST interface for interacting with standard and custom entities in the system.  BizTalk Server can’t talk “REST”, so that’s out.  They have (I think it’s still in the RC) as ASMX endpoint for legacy clients, and that is available for BizTalk consumers.  The final option is their new WCF SOAP endpoint.  Microsoft made a distinct choice to build an untyped interface into their SOAP service.  That is, the operations like Create or Update take in a generic Entity object.  An Entity has a name and a property bag of name/value pairs that hold the record’s columns and values.  If you are a building a .NET client to call Dynamics CRM 2011, you can use the rich SDK provided and generate some early bound classes which can be passed to a special proxy class (OrganizationServiceProxy) which hides the underlying translation between typed objects and the Entity object. There’s a special WCF behavior (ProxyTypesBehavior) in play there too.  So for .NET WCF clients, you don’t know you’re dealing with an untyped SOAP interface.  For non-.NET clients, or software that can’t leverage their SDK service proxy, you have to use the untyped interface directly.

So in real life, your choice as a BizTalk developer will have to be either (a) deal with messiness of creating and consuming untyped messages, or (b) build proxy services for BizTalk to invoke that take in typed objects and communicate to Dynamics CRM.  Ideally the Microsoft team would ship a WCF behavior that I could add to the BizTalk adapter that would do this typed-to-untyped translation both inbound and outbound, but I haven’t heard any mention of anything like that.

In this post, I’ll show option A which includes dealing directly with the bare Entity message type.  I’m scared.  Hold me.

Referencing the Service

First off, we need to add a reference to the SOAP endpoint.  Within Dynamics CRM, all the links to service endpoints can be found in the Customization menu under Developer Resources.  I’ve chosen the Organization Service which has a WSDL to point to.

Within a BizTalk project in Visual Studio.NET, I added a generated item, and chose to consume a WCF service.  After added the reference, I get a ton of generated artifacts.

Now in an ideal world, these schemas would be considered valid.  Alas, that is not the case.  When opening the schemas, I got all sorts of “end of the world” errors claiming that types couldn’t be found.  Apparently there is a lot of cross-schema-referencing missing from the schemas.  Wonderful.  So, I had to manually add a bunch of import statements to each schema.  To save someone else the pain, I’ll list out what I did:

• To OrganizationService_schemas_datacontract_org_2004_07_System_Collections_
  Generic.xsd schema, I added an Import directive to OrganizationService_schemas_microsoft_com_xrm_2011_Contracts.xsd.
• To OrganizationService_schemas_microsoft_com_2003_10_Serialization_Arrays.xsd schema I added an Import directive to OrganizationService_schemas_microsoft_com_2003_10_Serialization.xsd.
• To OrganizationService_schemas_microsoft_com_crm_2011_Contracts.xsd schema I added Import directives to both OrganizationService_schemas_microsoft_com_2003_10_Serialization_Arrays.xsd and OrganizationService_schemas_microsoft_com_xrm_2011_Contracts.xsd.
• To OrganizationService_schemas_microsoft_com_xrm_2011_Contracts.xsd schema, I added an Import directive to OrganizationService_schemas_microsoft_com_2003_10_Serialization_Arrays.xsd, OrganizationService_schemas_microsoft_com_xrm_2011_Metadata.xsd and OrganizationService_schemas_datacontract_org_2004_07_System_Collections_
  Generic.xsd.
• To OrganizationService_schemas_microsoft_com_xrm_2011_Contracts_Services.xsd schema I added Import directives to both OrganizationService_schemas_microsoft_com_2003_10_Serialization_Arrays.xsd and OrganizationService_schemas_microsoft_com_xrm_2011_Contracts.xsd.
• To OrganizationService_schemas_microsoft_com_xrm_2011_Metadata.xsd schema I added an Import directive to OrganizationService_schemas_datacontract_org_2004_07_System_Collections_
  Generic.xsd and OrganizationService_schemas_microsoft_com_xrm_2011_Contracts.xsd.

Ugh.  Note that even consuming their SOAP service from a custom .NET app required me to add some KnownType directives to the generated classes in order to make the service call work.  So, there is some work to do on interface definitions before the final launch of the product.

UPDATE (2/17/11): The latest CRM SDK version 5.0.1 includes compliant BizTalk Server schemas that can replace the ones added by the service reference.

For my simple demo scenario, I have a single message that holds details used for both querying and creating CRM records.  It holds the GUID identifier for a record in its Query node and in its Create node, it has a series of record attributes to apply to a new record.

Mapping the Query Message

Retrieving a record is pretty simple.  In this case, all you need to populate is the name of the entity (e.g “contact”, “account”, “restaurant”), the record identifier, and which columns to retrieve.  In my map, I’ve set the AllColumns node to true which means that everything comes back. Otherwise, I’d need some custom XSLT in a functoid to populate the Columns node.

Mapping the Create Message

The “create” message is more complicated as we need to successfully build up a set of name/value pairs.  Let’s walk through the steps.

The first “page” of my map links the entity’s name and sets a few unused elements to null.

Now it gets fun. You see a node there named KeyValuePairOfstringanyType.  This node is repeated for each column that I want to populate in my created Entity.  I’m going to show one way to populate it; there are others.  On this map page, I’ve connected each source node (related to a column) to a Looping functoid.  This will allow me to create one KeyValuePairOfstringanyType for each source node.

Got that?  Now I have to actually map the name and value across.  Let’s break this into two parts.  First, I need to get the node name into the “key” field.  We can do this by dragging each source node to the “key” field, and setting the map link’s Source Links property to Copy Name. This copies the name of the node across, not the value.

So far so good.  Now I need the node’s value.  You might say, “Richard, that part is easy.”  I’ll respond with “Nothing is easy.”  No, the node’s name, KeyValuePairOfstringanyType, gives it away. I actually need to set an XSD “type” property on the “value” node itself.  If I do a standard mapping and call the service, I get a serialization error because the data type of the “value” node is xsd:anyType and Dynamic CRM expects us to tell it which type the node is behaving like for the given column.  Because of this, I’m using a Scripting functoid to manually define the “value” node and attach a type attribute.

My functoid uses the Inline XSLT Call Template script type and contains the following:

<xsl:template name="SetNameValue">
<xsl:param name="param1" />
<value xmlns="http://schemas.datacontract.org/2004/07/System.Collections.Generic" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <xsl:attribute name="xsi:type">
    <xsl:value-of select="'xs:string'" />
   </xsl:attribute>
   <xsl:value-of select="$param1" />
  </value>
</xsl:template>

I also built an orchestration that calls the service and spits the result to disk, but there’s not much to that.  At this point, I deployed the solution.

Configuring the Send Port

Now within the BizTalk Admin Console, I imported one of the bindings that the WCF Service Consuming Wizard produced.  This makes life simple since there’s virtually nothing you have to change in the BizTalk send port that this binding produces.

The WCF-Custom adapter uses a custom WCF binding.

The only thing I added was on the Credentials tab, I added my Windows credentials for calling the service.  After creating the necessary receive port/location to pick up my initial file, send port to emit the service result to disk, and bound my orchestration, I was ready to go.

Executing the Query

In my Dynamics CRM environment, I added a customer account record for “Contoso”.  You can see a few data points which should show up in my service result when querying this record.

After calling the “Query” operation, I can see the result of the service call.  Not particularly pretty.  In reality, you’d have to build some mapping between this result and a canonical schema.

As for creating the record, when I send my command message in to create a new record, I see the new (Fabrikam) record in Dynamics CRM and a file on disk with the unique identifier for the new record.

Summary

So what’s “good”?  Dynamics CRM 2011 is an excellent application platform for building relationship-based solutions and has a wide range of integration options.  The REST interface is great and the SOAP interface will be useful for those that can leverage the CRM SDK.  What’s “bad”?  I don’t like the untyped interface.  I know it makes future flexibility easier (“add an attribute to an entity, don’t change the interface!”), but it really handicaps BizTalk and other tools that can’t leverage their SDK components.  I can’t see that many people choosing to build these functoid heavy maps just to create key/value pairs.  I’d probably opt to just use a custom XSLT stylesheet every time.  What’s “ugly”?  Not thrilled with the shape of the software, from an integration perspective, this close to general release.  Adding a simple WCF service reference to a .NET app should work.  It doesn’t.   Generated BizTalk schemas should be valid XSD.  They aren’t.  I don’t like the required “typing” of a node that forces me to do custom XSLT, even on a simple mapping.

I suspect that we’ll either see partner solutions, or even Microsoft ones, that make the integration story from BizTalk a tad simpler.  And for all I know, I’m missing something here.  I’ve vetted my concerns with the Microsoft folks, and I think I’ve got the story straight, however.

Thoughts from you all?  Are you a fan of untyped interfaces and willing to deal with the mapping sloppiness that ensues?  Other suggestions for how to make this process easier for developers?