The first challenge we face is how to actually interact with our Iron Foundry SQL Server service. At this point, Iron Foundry (and Cloud Foundry) doesn’t support direct tunneling to the application services. That means that I can’t just point the SQL Server 2008 Management Studio to a cloud database and use the GUI to muck with database properties. SQL Azure supports this, and hopefully we’ll see this added to the Cloud Foundry stack in the near future.

But one man’s challenge is … well, another man’s challenge. But, it’s an entirely solvable one. I decided to use the Microsoft Entity Framework to model a data structure, generate the corresponding database script, and run that against the Iron Foundry environment. I can do all of this locally (with my own SQL Server) to test it before deploying to Iron Foundry. Let’s do that.

Step 1: Generate the Data Model

To start with, I created a new, empty ASP.NET web application. This will hold our Entity model, ASP.NET web page for creating the database tables and populating them with data, and the WCF Data Service that exposes our data sets. Then, I added a new ADO.NET Data Entity Model to the project.

We’re not starting with an existing database here, so I chose the Empty Model option after creating this file. I then defined a simple set of entities representing Pets and Owners. The relationship indicates that an Owner may have multiple Pets.

Now, to make my life easier, I generated the DDL script that would build a pair of tables based on this model. The script is produced by right-clicking the model and selecting the Generate Database from Model option.

When walking through the Generate Database Wizard, I chose a database (“DemoDb”) on my own machine, and chose to save a connection entry in my web application’s configuration file. Note that the name used here (“PetModelContainer”) is the same name of the connection string the Entity Model expects to use when inflating the entities.

When this wizard finished, we got a SQL script that can generate the tables and relationships.

Before proceeding, open up that file and comment out all the GO statements. Otherwise, the SqlCommand object will throw an error when trying to execute the script.

Step 2: Add WCF Data Service

With the data model complete, I then added the WCF Data Service which exposes an OData endpoint for our entity model.

These services are super-easy to configure. There are really only two things you HAVE to do in order to get this service working. First the topmost statement (class declaration) needs to be updated with the name of the data entity class. Secondly, I uncommented/added statements for the entity access rules. In the case below, I provided “Read” access to all entities in the model.

public class PetService : DataService
    {
        // This method is called only once to initialize service-wide policies.
        public static void InitializeService(DataServiceConfiguration config)
        {
            // TODO: set rules to indicate which entity sets and service operations are visible, updatable, etc.
            // Examples:
            config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
            // config.SetServiceOperationAccessRule("MyServiceOperation", ServiceOperationRights.All);
            config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
        }
    }

Our service is now completed! That was easy.

Step 3: Create a Web Form that Creates the Database and Loads Data

I could not yet test this application since I haven’t physically constructed the underlying data structure. Since I cannot run the database script directly against the Iron Foundry database, I needed a host that can run this script. I chose an ASP.NET Web Form that could execute the script AND put some sample data in the tables.

Before creating the web page, I added an entry in my web.config file. Specifically, I added a new connection string entry that holds the details I need to connect to my LOCAL database.

<connectionStrings>
<add name="PetModelContainer" connectionString="metadata=res://*/PetModel.csdl|res://*/PetModel.ssdl|res://*/PetModel.msl;provider=System.Data.SqlClient; provider connection string=&quot;data source=.; initial catalog=DemoDb; integrated security=True; multipleactiveresultsets=True; App=EntityFramework&quot;" providerName="System.Data.EntityClient" />
<add name="PetDb" connectionString="data source=.; initial catalog=DemoDb; integrated security=True;" />
</connectionStrings>

I was now ready to consume the SQL script and create the database tables. The follow code instantiates a database connection, loads the database script from the file system into a SqlCommand object, and executes the command. Note that unlike Windows Azure, an Iron Foundry web application CAN use file system operations.

//create connection
            string connString = ConfigurationManager.ConnectionStrings["PetDb"].ConnectionString;
            SqlConnection c = new SqlConnection(connString);

            //load generated SQL script into a string
            FileInfo file = new FileInfo(Server.MapPath("PetModel.edmx.sql"));
            string tableScript = file.OpenText().ReadToEnd();

            c.Open();
            //execute sql script and create tables
            SqlCommand command = new SqlCommand(tableScript, c);
            command.ExecuteNonQuery();
            file.OpenText().Close();
            c.Close();

            command.Dispose();
            c.Dispose();

            lblStatus.Text = "db table created";

Cool. So after this runs, we should have real database tables in our LOCAL database. Next up, I wrote the code necessary to add some sample data into our tables

 //create connection
            string connString = ConfigurationManager.ConnectionStrings["PetDb"].ConnectionString;
            SqlConnection c = new SqlConnection(connString);
            c.Open();

            string commandString = "";
            SqlCommand command;
            string ownerId;
            string petId;

            //owner command
            commandString = "INSERT INTO Owners VALUES ('Richard Seroter', '818-232-5454', 0);SELECT SCOPE_IDENTITY();";
            command = new SqlCommand(commandString, c);
            ownerId = command.ExecuteScalar().ToString();

            //pet command
            commandString = "INSERT INTO Pets VALUES ('Watson', 'Dog', 'Corgador', '31 lbs', 'Do not feed wet food', " + ownerId + ");SELECT SCOPE_IDENTITY();";
            command = new SqlCommand(commandString, c);
            petId = command.ExecuteScalar().ToString();

 		//add more rows

		c.Close();
            command.Dispose();
            c.Dispose();

            lblStatus.Text = "rows added";

Step 4: Local Testing

I’m ready to test this application. After pressing F5 in Visual Studio 2010 and running this web application in a local web server, I saw my Web Form buttons for creating tables and seeding data. After clicking the Create Database button, I checked my local SQL Server. Sure enough, I found my new tables.

Next, I clicked the Seed Data button on my form and saw three rows added to each table. With my tables ready and data loaded, I could now execute the OData service. Hitting the service address resulted in a list of entities that the service makes available.

And then, per typical OData queries, I could drill into the various entities and relationship. With this simple query, I can show all the pets for a particular owner.

At this point, I had a fully working, LOCAL version of the this application.

Step 5: Deploy to Iron Foundry

Here’s where the rubber meets the road. Can I take this app, as is, and have it work in Iron Foundry? This answer is “pretty much.” The only thing that I really need to do is update the connection string for my Iron Foundry instance of SQL Server, but I’m getting ahead of myself. I first had to get this application up to Iron Foundry so that I could associate it with a SQL instance. Since I’ve had some instability with the Visual Studio plugin for Iron Foundry, I went ahead and “published” my ASP.NET application to my file system and ran the vmc client to upload the application.

With my app uploaded, I then bound my application to a SQL Server application service. I used the bind-service command to bind my SQL Server service to my application.

Now I needed to view my web.config file that was modified by the Iron Foundry engine. When this binding occurred, Iron Foundry provisioned a SQL Server space for me and updated my web.config file with the valid connection string. I’m going to need those connection string values (server name, database name, credentials) for my application as well. I wasn’t sure how to access my application files from the vmc tool, so I switched back to the Cloud Explorer where I can actually browse an app.

My web.config file now contained a “Default” connection string added by Iron Foundry.

<connectionStrings>
    <add name="PetModelContainer" connectionString="metadata=res://*/PetModel.csdl|res://*/PetModel.ssdl|res://*/PetModel.msl;provider=System.Data.SqlClient;provider connection string=&quot;data source=.;initial catalog=DemoDb;integrated security=True;multipleactiveresultsets=True;App=EntityFramework&quot;"
      providerName="System.Data.EntityClient" />
    <add name="PetDb" connectionString="data source=.;initial catalog=DemoDb;integrated security=True;" />
    <add name="Default" connectionString="Data Source=XXXXXX;Initial Catalog=YYYYYYY;Integrated Security=False;User ID=ABC;Password=DEF;Connect Timeout=30" />
  </connectionStrings>

Step 6: Update Application with Iron Foundry Connection Details and then Test the Solution

With these connection string values in hand, I had two things to update. First, I updated my generated T-SQL script to “use” the appropriate database.

Finally, I had to update the two previously created connection strings. I updated my ORIGINAL web.config and not the one that I retrieved back from Iron Foundry. The first (“PetDb”) connection string was used by my code to run the T-SQL script and create the tables, and the second connection string (“PetModelContainer”) is leveraged by the Entity Framework and the WCF Data Service. Both were updated with the Iron Foundry connection string details.

<connectionStrings>
    <add name="PetModelContainer" connectionString="metadata=res://*/PetModel.csdl|res://*/PetModel.ssdl|res://*/PetModel.msl;provider=System.Data.SqlClient;provider connection string=&quot;data source=XXXXX;initial catalog=YYYYYY;Integrated Security=False;User ID=ABC;Password=DEF;multipleactiveresultsets=True;App=EntityFramework&quot;"
      providerName="System.Data.EntityClient" />
    <add name="PetDb" connectionString="data source=XXXXX;initial catalog=YYYYYY;Integrated Security=False;User ID=ABC;Password=DEF;" />
   </connectionStrings>

With these updates in place, I rebuilt the application and pushed a new version of my application up to Iron Foundry.

I was now ready to test this cat out. As expected, I could now hit the public URL of my “setup” page (which I have since removed so that you can’t create tables over and over!).

After creating the database (via Create Database button), I then clicked the button to load a few rows of data into my database tables.

For the grand finale, I tested my OData service which should allow me to query my new SQL Server database tables. Hitting the URL http://seroterodata.gofoundry.net/PetService.svc/Pets returns a list of all the Pets in my database.

As with any OData service, you can now mess with the data in all sorts of ways. This URL (http://seroterodata.gofoundry.net/PetService.svc/Pets(2)/Owner) returns the owner of the second pet. If I want to show the owner and pet in a single result set, I can use this URL (http://seroterodata.gofoundry.net/PetService.svc/Owners(1)?$expand=Pets). Want the name of the 3rd pet? use this URL (http://seroterodata.gofoundry.net/PetService.svc/Pets(3)/Name).

Summary

Overall, this is fairly straightforward stuff. I definitely felt a bit handicapped by not being able to directly use SQL Server Management Studio, but at least it forced me to brush up on my T-SQL commands. One interesting item was that it APPEARS that I am provisioned a single database when I first bind to an application service and that same database is used for subsequent bindings. I had built a previous application that used the SQL Server application service and later deleted the app. When I deployed the application above, I noticed that the tables I had created earlier were still there! So, whether intentionally or not, Iron Foundry points me to the same (personal?) database for each app. Not a big deal, but this could have unintended side effects if you’re not aware of it.

Right now, developers can use either the SQL Server application service or MongoDB application service. Expect to see more show up in the near future. While you need to programmatically provision your database resources, that doesn’t seem to be a big deal. The Iron Foundry application services are a critical resource in building truly interesting web applications and I hope you enjoyed this walkthrough.

If you’ve been curious about what StreamInsight is, or have an odd interest in hearing me speak, now’s the time to indulge yourself. If you like where the course is going, I’d strongly encourage you to sign up for a Pluralsight subscription, which is one of the best investments that a developer can make.

Thanks Paul!

In this post, I’m going to highlight some of my favorite blog posts and books from 2011.

First off, these are a few blog posts that I enjoyed writing this year.

• Is BizTalk Server Going Away At Some Point? Yes. Dead? Nope. I think this was the first blog post that I’ve written that ever went viral. I wrote this in my head on the way to work one morning and apparently it resonated with a lot of people.
• The Good, Bad and Ugly of Integrating Dynamics CRM 2011 and BizTalk Server 2010. This has inexplicably become one of the most popular posts that I’ve ever written.
• Using the BizTalk Adapter Pack and AppFabric Connect in a Workflow Service. This was one of the demonstrations from my New Zealand trip and I thought it was a cool way to do lightweight, stateful LOB communication.
• 6 Quick Steps for Windows/.NET Folks to Try Out Cloud Foundry. Lots of changes to cloud offerings in 2011 and in hindsight, I’m glad I looked into Cloud Foundry. It’s a cool platform and we’ll likely be hearing much more about it in 2012.
• It was fun to dig into the WCF Routing Service in a pair of posts. I’m not sure if this is heavily used software, but it definitely adds value in simple routing scenarios.
• Sending Messages from Salesforce.com to BizTalk Server Through Windows Azure AppFabric. I built this demo for the last book, and still think it’s a really neat way to run cloud apps but leverage onsite services.
• I had a good time exploring a pre-release version of StreamInsight Austin (“StreamInsight in the cloud”) and showed how to build an internet-friendly adapter and then how to deploy a cloud CEP app.
• Integration in the Cloud. This four-part series was the basis for my presentation at QCon Hangzhou and I enjoyed applying well-known patterns to cloud scenarios.
• First Look: Deploying .NET Web Apps to Cloud Foundry via Iron Foundry. If we assume that PaaS is the future of cloud computing, I suspect that Azure will be just one of the places that developers deploy .NET web applications to.
• I really enjoyed each of the “4 Questions” interviews in 2011 and had a blast talking to Rick Garibay, Steef-Jan Wiggers, Jon Fancey,  Buck Woody, Sam Vanhoutte, Pablo Cibraro, Allan Mitchell, Ryan CrawCour, Scott Seely, and Clemens Vasters.

It was hard to keep up my regular pace of reading a book or two a month, but I still carved out time to read some memorable ones. I admittedly read fewer deep technical books and focused more on growing as a strategist and learning to manage my time effectively.  Here are a few of my favorites from this year:

• Your Brain at Work. Great description of what tasks tax the brain most, how to decompose complex ideas, strategies for staying focused and how to be more mindful. Useful stuff.
• Blink. Gladwell is known for writing provocative books, and this is no exception.  Instead of thinking that the quality of our decisions are based on the time/effort we put into it, we should trust our judgment more often.
• The Bullpen Gospels. I’m a sucker for baseball books, and this one was immensely satisfying.  It’s a great story of a pitcher’s journey through minor league baseball.
• Do the Work. Nice little book that encourages us to jump into a task, not fear success, and to remember that “finishing” is the most critical part of a project.
• The Naked Presenter: Delivering Powerful Presentations With or Without Slides. I’ve worked at becoming a better presenter over the past few years, and books like this help keep me focused on telling a compelling story without using slides as a crutch.
• fruITion: Creating the Ultimate Corporate Strategy for Information Technology. Good read about articulating the real role of IT in an organization and the value of better alignment with business partners.
• recrEAtion: Realizing the Extraordinary Contribution of Your Enterprise Architects. If you’re an architect, or even pretend to be one, this is a must-read.  Fundamentally changed my thinking on what it means to be an (enterprise) architect. Continues the fictitious story from the previous book, fruITion.
• Little Bets. Food for thought about the value of experimentation as most new brilliant ideas don’t form out of thin air, but are discovered.
• Game of Thrones; A Clash of Kings; A Storm of Swords. I’m not a fantasy book guy, but after watching Game of Thrones on HBO, I thought I’d try the books. I read the first three and loved the characters and “did they really do that?” plot twists.
• The Two Second Advantage: How We Succeed by Anticipating the Future–Just Enough. Excellent book on the real-time data revolution. Although written by the CEO of TIBCO, the book isn’t very technical but rather shows the reader the significant impact of real-time intelligence.
• A. Lincoln: A Biography. Fascinating, well-paced story of one of America’s most compelling historical figures. Lincoln was such a deep thinker and this book does an excellent job following his thoughts from early life through his successful navigation of the US Civil War.

As for 2012, hopefully you’ll see more blog posts, more training courses, and more interviews containing stupid questions.

The Azure AppFabric v1.5 release contains useful Service Bus capabilities for durable messaging communication through the use of Queues and Topics. The Service Bus still has the Relay Service which is great for invoking services through a cloud relay, but the asynchronous communication through the Relay Service isn’t durable.  Queues and Topics now let you send messages to one or many subscribers and have stronger guarantees of delivery.

An Iron Foundry application is just a standard .NET web application.  So, I’ll start with a blank ASP.NET web application and use old-school Web Forms instead of MVC. We need a reference to the Microsoft.ServiceBus.dll that comes with Azure AppFabric v1.5.  With that reference added, I added a new Web Form and included the necessary “using” statements.

I then built a very simple UI on the Web Form that takes in a handful of values that will be sent to the on-premises subscriber(s) through the Service Bus. Before creating the code that sends a message to a Topic, I defined an “Order” object that represents the data being sent to the topic. This object sits in a shared assembly used by this application that sends the message, and another application that receives a message.

[DataContract]
    public class Order
    {
        [DataMember]
        public string Id { get; set; }
        [DataMember]
        public string ProdId { get; set; }
        [DataMember]
        public string Quantity { get; set; }
        [DataMember]
        public string Category { get; set; }
        [DataMember]
        public string CustomerId { get; set; }
    }

The “submit” button on the Web Form triggers a click event that contains a flurry of activities.  At the beginning of that click handler, I defined some variables that will be used throughout.

//define my personal namespace
string sbNamespace = "richardseroter";
//issuer name and key
string issuer = "MY ISSUER";
string key = "MY PRIVATE KEY";

//set the name of the Topic to post to
string topicName = "OrderTopic";
//define a variable that holds messages for the user
string outputMessage = "result: ";

Next I defined a TokenProvider (to authenticate to my Topic) and a NamespaceManager (which drives most of the activities with the Service Bus).

//create namespace manager
TokenProvider tp = TokenProvider.CreateSharedSecretTokenProvider(issuer, key);
Uri sbUri = ServiceBusEnvironment.CreateServiceUri("sb", sbNamespace, string.Empty);
NamespaceManager nsm = new NamespaceManager(sbUri, tp);

Now we’re ready to either create a Topic or reference an existing one. If the Topic does NOT exist, then I went ahead and created it, along with two subscriptions.

//create or retrieve topic
bool doesExist = nsm.TopicExists(topicName);

if (doesExist == false)
   {
      //topic doesn't exist yet, so create it
      nsm.CreateTopic(topicName);

      //create two subscriptions

      //create subscription for just messages for Electronics
      SqlFilter eFilter = new SqlFilter("ProductCategory = 'Electronics'");
      nsm.CreateSubscription(topicName, "ElecFilter", eFilter);

      //create subscription for just messages for Clothing
      SqlFilter eFilter2 = new SqlFilter("ProductCategory = 'Clothing'");
      nsm.CreateSubscription(topicName, "ClothingFilter", eFilter2);

      outputMessage += "Topic/subscription does not exist and was created; ";
    }

At this point we either know that a topic exists, or we created one.  Next, I created a MessageSender which will actually send a message to the Topic.

//create objects needed to send message to topic
 MessagingFactory factory = MessagingFactory.Create(sbUri, tp);
 MessageSender orderSender = factory.CreateMessageSender(topicName);

We’re now ready to create the actual data object that we send to the Topic.  Here I referenced the Order object we created earlier.  Then I wrapped that Order in the BrokeredMessage object.  This object has a property bag that is used for routing.  I’ve added a property called “ProductCategory” that our Topic subscription uses to make decisions on whether to deliver the message to the subscriber or not.

//create order
Order o = new Order();
o.Id = txtOrderId.Text;
o.ProdId = txtProdId.Text;
o.CustomerId = txtCustomerId.Text;
o.Category = txtCategory.Text;
o.Quantity = txtQuantity.Text;

//create brokered message object
BrokeredMessage msg = new BrokeredMessage(o);
//add properties used for routing
msg.Properties["ProductCategory"] = o.Category;

Finally, I send the message and write out the data to the screen for the user.

//send it
orderSender.Send(msg);

outputMessage += "Message sent; ";
lblOutput.Text = outputMessage;

I decided to use the command line (Ruby-based) vmc tool to deploy this app to Iron Foundry.  So, I first published my website to a directory on the file system.  Then, I manually copied the Microsoft.ServiceBus.dll to the bin directory of the published site.  Let’s deploy! After logging into my production Iron Foundry account by targeting the api.gofoundry.net management endpoint, I executed a push command and instantly saw my web application move up to the cloud. It takes like 8 seconds from start to finish.

My site is now online and I can visit it and submit a new order [note that this site isn’t online now, so don’t try and flood my machine with messages!].  When I click the submit button, I can see that a new Topic was created by this application and a message was sent.

Let’s confirm that we really have a new Topic with subscriptions. I can first confirm this through the Windows Azure Management Console.

To see more details, I can use the Service Bus Explorer tool which allows us to browse our Service Bus configuration.  When I launch it, I can see that I have a Topic with a pair of subscriptions and even what Filter I applied.

I previously built a WinForm application that pulls data from an Azure AppFabric Service Bus Topic. When I click the “Receive Message” button, I pull a message from the Topic and we can see that it has the same Order ID as the message submitted from the website.

If I submit another message from the website, I see a different message because my Topic already exists and I’m simply reusing it.

Summary

So what did we see here?  First, I proved that an ASP.NET web application that you want to deploy to the Iron Foundry (onsite or offsite) cloud looks just like any other ASP.NET web application.  I didn’t have to build it differently or do anything special. Secondly, we saw that I can easily use the Windows Azure AppFabric Service Bus to reliably share data between a cloud-hosted application and an on-premises application.

The first seven modules of the course are now online at Pluralsight under the title StreamInsight Fundamentals.  The final (yet to be finished) module will be on building resilient applications and leveraging the new checkpointing feature.  This is a complex topic, and I am building a full end to end demo from scratch, and didn’t want that holding up the primary modules of the course.

So what did I build? Seven modules totaling about 4 1/2 hours of content.  Each module is very demo-heavy with a focus on realistic scenarios and none of the “let’s assume you have an object of type A with a property called Foo” stuff.

• Module 1 – Introducing StreamInsight. This module is a brief introduction into event driven architecture, complex event processing and the basics of the StreamInsight product.
• Module 2 – Developing StreamInsight Queries. Lots of content here cover filtering, projection, event windows, grouping, aggregation, TopK, join, union and a host of timestamp modification examples. This is the longest module because it’s arguably the most important topic (but still watch the other ones!).
• Module 3 – Extending StreamInsight LINQ Queries. When out-of-the-box operators won’t do, build your own!  This module looks at all the supported ways to add extensions to StreamInsight LINQ.
• Module 4 – StreamInsight Event Sources: IObservable and IEnumerable. Here I walk through how to use both IObservable and IEnumerable objects as either the source or sink in a StreamInsight application.  The IObservable stuff was fun to build, but also took the longest for me to fully understand.
• Module 5 – StreamInsight Event Sources: Developing Adapters. This module covers the strategies and techniques for building both input and output adapters. Together we’ll build a typed MSMQ input adapter and an untyped MSMQ output adapter.  Good times will be had by all.
• Module 6 – Hosting StreamInsight Applications.  In this module, I show how to host StreamInsight within an application or by leveraging the standalone service.  I also go through a series of examples on how you chain queries (and streams) together and leverage their reusable nature.
• Module 7 – Monitoring and Troubleshooting StreamInsight Applications. Here I show all the ways to collect diagnostic data on StreamInsight applications and then go through event flow analysis using the Event Flow Debugger.

All in all, it was fun thinking up the structure, preparing the material, building the demos, and producing the training videos.  There hasn’t been a whole lot of StreamInsight material out there, so hopefully this helps developers and architects who are trying to get up to speed on this very cool and powerful technology.

I’ve written a bit about Cloud Foundry in the past, and have watched it become one of the most popular platforms for cloud developers.  While Cloud Foundry supports a diverse set of platforms like Java, Ruby and Node.js, .NET has been conspicuous absent from that list.  That’s where Tier 3 jumped in.  They’ve forked the Cloud Foundry offering and made a .NET version (called Iron Foundry) that can run by an online hosted provider, or, in your own data center. Your own private, open source .NET PaaS.  That’s a big deal.

I’ve been working a bit with their team for the past few weeks, and if you’d like to read more from their technical team, check out the article that I wrote for InfoQ.com today.  Let’s jump in and try and deploy a very simple RESTful WCF service to Iron Foundry using the tools they’ve made available.

Demo

First off, I pulled the source code from their GitHub library.  After building that, I made sure that I could open up their standalone Cloud Foundry Explorer tool and log into my account. This tool also plugs into Visual Studio 2010, and I’ll show that soon [12/22 update: note that Iron Foundry's production URL has changed from the value used in the screenshot below].

It’s a nice little tool that shows me any apps I have running, and lets me interact with them.  But, I have no apps deployed here, so let’s change that!  How about we go with a very simple WCF contract that returns a customer object when the caller hits a specific URI.  Here’s the WCF contract:

[ServiceContract]
    public interface ICustomer
    {
        [OperationContract]
        [WebGet(UriTemplate = "/{id}")]
        Customer GetCustomer(string id);
    }

    [DataContract]
    public class Customer
    {
        [DataMember]
        public string Id { get; set; }
        [DataMember]
        public string FullName { get; set; }
        [DataMember]
        public string Country { get; set; }
        [DataMember]
        public DateTime DateRegistered { get; set; }
    }

The implementation of this service is extremely simple.  Based on the input ID, I return one of a few different customer records.

public class CustomerService : ICustomer
    {
        public Customer GetCustomer(string id)
        {
            Customer c = new Customer();
            c.Id = id;

            switch (id)
            {
                case "100":
                    c.FullName = "Richard Seroter";
                    c.Country = "USA";
                    c.DateRegistered = DateTime.Parse("2011-08-24");
                    break;
                case "200":
                    c.FullName = "Jared Wray";
                    c.Country = "USA";
                    c.DateRegistered = DateTime.Parse("2011-06-05");
                    break;
                default:
                    c.FullName = "Shantu Roy";
                    c.Country = "USA";
                    c.DateRegistered = DateTime.Parse("2011-05-11");
                    break;
            }

            return c;
        }

My WCF service configuration is also pretty straightforward.  However, note that I do NOT specify a full service address. When I asked one of the Iron Foundry developers about this he said:

When an application is deployed, the cloud controller picks a server out of our farm of servers to which to deploy the application. On that server, a random high port number is chosen and a dedicated web site and app pool is configured to use that port. The router service then uses that URL (http://server:49367) when requests come in to http://<application>.gofoundry.net

<configuration>
<system.web>
<compilation debug="true" targetFramework="4.0" />
</system.web>
<system.serviceModel>
<bindings>
<webHttpBinding>
<binding name="WebBinding" />
</webHttpBinding>
</bindings>
<services>
<service name="Seroter.IronFoundry.WcfRestServiceDemo.CustomerService">
<endpoint address="CustomerService" behaviorConfiguration="RestBehavior"
binding="webHttpBinding" bindingConfiguration="WebBinding" contract="Seroter.IronFoundry.WcfRestServiceDemo.ICustomer" />
</service>
</services>
<behaviors>
<endpointBehaviors>
<behavior name="RestBehavior">
<webHttp helpEnabled="true" />
</behavior>
</endpointBehaviors>
<serviceBehaviors>
<behavior name="">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="true" />
</behavior>
</serviceBehaviors>
</behaviors>
<serviceHostingEnvironment multipleSiteBindingsEnabled="true" />
</system.serviceModel>
<system.webServer>
<modules runAllManagedModulesForAllRequests="true"/>
</system.webServer>
<connectionStrings></connectionStrings>
</configuration>

I’m not ready to deploy this application. While  I could use the standalone Cloud Foundry Explorer that I showed you before, or even the vmc command line, the easiest one is the Visual Studio plug in.  By right-clicking my project, I can choose Push Cloud Foundry Application which launches the Cloud Foundry Explorer.

Now I can select my existing Iron Foundry configuration named Sample Server (which points to the Iron Foundry endpoint and includes my account credentials), select a name for my application, choose a URL, and pick both the memory size (64MB up to 2048MB) and application instance count [12/22 update: note that Iron Foundry's production URL has changed from the value used in the screenshot below].

The application is then pushed to the cloud. What’s awesome is that the application is instantly available after publishing.  No waits, no delays.  Want to see the app in action?  Based on the values I entered during deployment, you can hit the URL at http://serotersample6.gofoundry.net/CustomerService.svc/CustomerService/100. [12/22 update: note that Iron Foundry's production URL has changed, so the working URL above doesn't match the values I showed in the screenshots]

Sweet.  Now let’s check out some diagnostic info, shall we?  I can fire up the standalone Cloud Foundry Explorer and see my application running.

What can I do now?  On the right side of the screen, I have options to change/add URLs that map to my service, increase my allocated memory, or modify the number of application instances.

On the bottom left of the this screen, I can find out details of the instances that I’m running on.  Here, I’m on a single instance and my app has been running for 5 minutes.

Finally,  I can provision application services associated with my web application.

Let’s change my instance count.  I was blown away when I simply “upticked” the Instances value and instantly I saw another instance provisioned.  I don’t think Azure is anywhere near as fast.

What if I like using the vmc command line tool to administer my Iron Foundry application?  Let’s try that out. I went to the .NET version of the vmc tool that came with the Iron Foundry code download, and targeted the API just like you would in “regular” Cloud Foundry.[12/22 update: note that Iron Foundry's production URL has changed from the value used in the screenshot below].

It’s awesome (and I guess, expected) that all the vmc commands work the same and I can prove that by issuing the “vmc apps” command which should show me my running applications.

Not everything was supported yet on my build, so if I want to increase the instance count or memory, I’d jump back to the Cloud Foundry Explorer tool.

Summary

What a great offering. Imagine deploying this within your company as a way to have a private PaaS. Or using it as a public PaaS and have the same deployment experience for .NET, Java, Ruby and Node applications.  I’m definitely going to troll through the source code since I know what a smart bunch build the “original” Cloud Foundry and I want to see how the cool underpinnings of that (internal pub/sub, cloud controller, router, etc) translated to .NET.

I encourage you to take a look.  I like Windows Azure, but more choice is a good thing and I congratulate the Tier 3 team on open sourcing their offering and doing such a cool service for the community.

Let’s see how he stands up to the rigor of Four Questions.

Thanks Clemens for participating! Great perspectives.

What Is It?

While the remote procedure pattern provides looser coupling than the shared database pattern, it is still a blocking call and not particularly scalable.  Architects and developers use an asynchronous messaging pattern when they want to share data in the most scalable and responsive way possible.  Think of sending an email.  Your email client doesn’t sit and wait until the recipient has received and read the email message.  That would be atrocious. Instead, our email server does a multicast to recipients allows our email client to carry on. This is somewhat similar to publish/subscribe where the publisher does not dictate which specific receiver will get the message.

So in theory, the sender of the message doesn’t need to know where the message will end up.  They also don’t need to know *when* a message is received or processed by another party.  This supports disconnected client scenarios where the subscriber is not online at the same time as the publisher.  It also supports the principle of replicable units where one receiver could be swapped out with no direct impact to the source of the message.  We see this pattern realized in Enterprise Service Bus or Integration Bus products (like BizTalk Server) which promote extreme loose coupling between systems.

Challenges

There are a few challenges when dealing with this pattern.

• There is no real-time consistency. Because the message source asynchronously shares data that will be processed at the convenience of the receiver, there is a low likelihood that the systems involved are simultaneously consistent.  Instead, you end up with eventual consistency between the players in the messaging solution.
• Reliability / durability is required in some cases. Without a persistence layer, it is possible to lose data.  Unlike the remote procedure invocation pattern (where exceptions are thrown by the target and both caught and handled by the caller), problems in transmission or target processing do not flow back to the publisher.  What happens if the recipient of a message is offline?  What if the recipient is under heavy load and rejecting new messages? A durable component in the messaging tier can protect against such cases by doing store-and-forward type implementation that doesn’t remove the message from the durable store until it has been successfully consumed.
• A router may be useful when transmitting messages. Instead of, or in addition to a durable store, a routing component can help manage the central subscriptions for pub/sub transmissions, help with protocol bridging, data transformation and workflow (e.g. something like BizTalk Server). This may not be needed in distributed ESB solutions where the receiver is responsible for most of that.
• There is limited support for this pattern in packaged software products.  I’ve seen few commercial products that expose asynchronous inbound channels, and even fewer that have easy-to-configure ways to publish outbound events asynchronously.  It’s not that difficult to put adapters in front of these systems, or mimic asynchronous publication by polling a data tier, but it’s not the same.

Cloud Considerations

What are things to consider when doing this pattern in a cloud scenario?

• To do this between cloud and on-premises solutions, this requires creativity. I showed in the previous post how one can use Windows Azure AppFabric to expose on-premises endpoints to cloud applications. If we need to push data on-premises, and Azure AppFabric isn’t an option, then you’re looking at doing a VPN or internet-facing proxy service. Or, you could rely on aggressive polling of a shared queue (as I’ll show below).
• Cloud provider limits and architecture will influence solution design. Some vendors, such as Salesforce.com, limit the frequency and amount of polling that it will do. This impacts the ability to poll a durable store used between cloud applications. The distributed nature of cloud services. and embrace of the eventual consistency model, can change how one retrieves data.  For example, Amazon’s Simple Queue Service may not be first-in-first out, and uses a sampling algorithm that COULD result in a query not returning all the messages in the logical queue.

Solution Demonstration

Let’s say that the fictitious Seroter Corporation has a series of public websites and wants a consistent way to push customer inquiries from the websites to back end systems that process these inquiries.  Instead of pushing these inquiries directly into one or many CRM systems, or doing the low-tech email option, we’d rather put all the messages into a queue and let each interested party pull the ones they want.  Since these websites are cloud-hosted, we don’t want to explicitly push these messages into the internal network, but rather, asynchronously publish and poll messages from a shared queue hosted by Amazon Simple Queue Service (SQS). The polling applications could either be another cloud system (CRM system Salesforce.com) or an on-premises system, as shown below.

So I’ll have a web page built using Ruby and hosted in Cloud Foundry, a SQS queue that holds inquiries submitted from that site, and both an on-premises .NET application and a SaaS Salesforce.com application that can poll that queue for messages.

Setting up a queue in SQS is so easy now, that I won’t even make it a sub-section in this post.  The AWS team recently added SQS operations to their Management Console, and they’ve made it very simple to create, delete, secure and monitor queues. I created a new queue named Seroter_CustomerInquiries.

Sending Messages from Cloud Foundry to Amazon Simple Queue Service

In my Ruby (Sinatra) application, I have a page where a user can ask a question.  When they click the submit button, I go into the following routine which builds up the SQS message (similar to the SimpleDB message from my previous post) and posts a message to the queue.

post '/submitted/:uid' do	# method call, on submit of the request path, do the following

   #--get user details from the URL string
	@userid = params[:uid]
	@message = CGI.escape(params[:message])
    #-- build message that will be sent to the queue
	@fmessage = @userid + "-" + @message.gsub("+", "%20")

	#-- define timestamp variable and format
	@timestamp = Time.now
	@timestamp = @timestamp.strftime("%Y-%m-%dT%H:%M:%SZ")
	@ftimestamp = CGI.escape(@timestamp)

	#-- create signing string
	@stringtosign = "GET\n" + "queue.amazonaws.com\n" + "/084598340988/Seroter_CustomerInquiries\n" + "AWSAccessKeyId=ACCESS_KEY" + "&Action=SendMessage" + "&MessageBody=" + @fmessage + "&SignatureMethod=HmacSHA1" + "&SignatureVersion=2" + "&Timestamp=" + @ftimestamp + "&Version=2009-02-01"

	#-- create hashed signature
	@esignature = CGI.escape(Base64.encode64(OpenSSL::HMAC.digest('sha1',@@awskey, @stringtosign)).chomp)

	#-- create AWS SQS query URL
	@sqsurl = "https://queue.amazonaws.com/084598340988/Seroter_CustomerInquiries?Action=SendMessage" + "&MessageBody=" + @fmessage + "&Version=2009-02-01" + "&Timestamp=" + @ftimestamp + "&Signature=" + @esignature + "&SignatureVersion=2" + "&SignatureMethod=HmacSHA1" + "&AWSAccessKeyId=ACCESS_KEY"

	#-- load XML returned from query
	@doc = Nokogiri::XML(open(@sqsurl))

   #-- build result message which is formatted string of the inquiry text
	@resultmsg = @fmessage.gsub("%20", " ")

	haml :SubmitResult
end

The hard part when building these demos was getting my signature string and hashing exactly right, so hopefully this helps someone out.

After building and deploying the Ruby site to Cloud Foundry, I could see my page for inquiry submission.

When the user hits the “Send Inquiry” button, the function above is called and assuming that I published successfully to the queue, I see the acknowledgement page.  Since this is an asynchronous communication, my web app only has to wait for publication to the queue, not invoking a function in a CRM system.

To confirm that everything worked, I viewed my SQS queue and can clearly see that I have a single message waiting in the queue.

.NET Application Pulling Messages from an SQS Queue

With our message sitting safely in the queue, now we can go grab it.  The first consuming application is an on-premises .NET app.  In this very feature-rich application, I poll the queue and pull down any messages found.  When working with queues, you often have two distinct operations: read and delete (“peek” is also nice to have). I can read messages from a queue, but unless I delete them, they become available (after a timeout) to another consumer.  For this scenario, we’d realistically want to read all the messages, and ONLY process and delete the ones targeted for our CRM app.  Any others, we simply don’t delete, and they go back to waiting in the queue. I haven’t done that, for simplicity sake, but keep this in mind for actual implementations.

In the example code below, I’m being a bit lame by only expecting a single message. In reality, when polling, you’d loop through each returned message, save its Handle value (which is required when calling the Delete operation) and do something with the message.  In my case, I only have one message, so I explicitly grab the “Body” and “Handle” values.  The code shows the “retrieve messages” button click operation which in turn calls “receive” operation and “delete” operation.

private void RetrieveButton_Click(object sender, EventArgs e)
        {
            lbQueueMsgs.Items.Clear();
            lblStatus.Text = "Status:";

            string handle = ReceiveFromQueue();
            if(handle!=null)
                DeleteFromQueue(handle);

        }

private string ReceiveFromQueue()
        {
            //timestamp formatting for AWS
            string timestamp = Uri.EscapeUriString(string.Format("{0:s}", DateTime.UtcNow));
            timestamp = DateTime.Now.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ss.fffZ");
            timestamp = HttpUtility.UrlEncode(timestamp).Replace("%3a", "%3A");

            //string for signing
            string stringToConvert = "GET\n" +
            "queue.amazonaws.com\n" +
            "/084598340988/Seroter_CustomerInquiries\n" +
            "AWSAccessKeyId=ACCESS_KEY" +
            "&Action=ReceiveMessage" +
            "&AttributeName=All" +
            "&MaxNumberOfMessages=5" +
            "&SignatureMethod=HmacSHA1" +
            "&SignatureVersion=2" +
            "&Timestamp=" + timestamp +
            "&Version=2009-02-01" +
            "&VisibilityTimeout=15";

            //hash the signature string
			  string awsPrivateKey = "PRIVATE KEY";
            Encoding ae = new UTF8Encoding();
            HMACSHA1 signature = new HMACSHA1();
            signature.Key = ae.GetBytes(awsPrivateKey);
            byte[] bytes = ae.GetBytes(stringToConvert);
            byte[] moreBytes = signature.ComputeHash(bytes);
            string encodedCanonical = Convert.ToBase64String(moreBytes);
            string urlEncodedCanonical = HttpUtility.UrlEncode(encodedCanonical).Replace("%3d", "%3D");

             //build up request string (URL)
            string sqsUrl = "https://queue.amazonaws.com/084598340988/Seroter_CustomerInquiries?Action=ReceiveMessage" +
            "&Version=2009-02-01" +
            "&AttributeName=All" +
            "&MaxNumberOfMessages=5" +
            "&VisibilityTimeout=15" +
            "&Timestamp=" + timestamp +
            "&Signature=" + urlEncodedCanonical +
            "&SignatureVersion=2" +
            "&SignatureMethod=HmacSHA1" +
            "&AWSAccessKeyId=ACCESS_KEY";

            //make web request to SQS using the URL we just built
            HttpWebRequest req = WebRequest.Create(sqsUrl) as HttpWebRequest;
            XmlDocument doc = new XmlDocument();
            using (HttpWebResponse resp = req.GetResponse() as HttpWebResponse)
            {
                StreamReader reader = new StreamReader(resp.GetResponseStream());
                string responseXml = reader.ReadToEnd();
                doc.LoadXml(responseXml);
            }

			 //do bad xpath and grab the body and handle
            XmlNode handle = doc.SelectSingleNode("//*[local-name()='ReceiptHandle']");
            XmlNode body = doc.SelectSingleNode("//*[local-name()='Body']");

            //if empty then nothing there; if not, then add to listbox on screen
            if (body != null)
            {
                //write result
                lbQueueMsgs.Items.Add(body.InnerText);
                lblStatus.Text = "Status: Message read from queue";
                //return handle to calling function so that we can pass it to "Delete" operation
                return handle.InnerText;
            }
            else
            {
                MessageBox.Show("Queue empty");
                return null;
            }
        }

private void DeleteItem(string itemId)
        {
            //timestamp formatting for AWS
            string timestamp = Uri.EscapeUriString(string.Format("{0:s}", DateTime.UtcNow));
            timestamp = DateTime.Now.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ss.fffZ");
            timestamp = HttpUtility.UrlEncode(timestamp).Replace("%3a", "%3A");

            string stringToConvert = "GET\n" +
            "sdb.amazonaws.com\n" +
            "/\n" +
            "AWSAccessKeyId=ACCESS_KEY" +
            "&Action=DeleteAttributes" +
            "&DomainName=SeroterInteractions" +
            "&ItemName=" + itemId +
            "&SignatureMethod=HmacSHA1" +
            "&SignatureVersion=2" +
            "&Timestamp=" + timestamp +
            "&Version=2009-04-15";

            string awsPrivateKey = "PRIVATE KEY";
            Encoding ae = new UTF8Encoding();
            HMACSHA1 signature = new HMACSHA1();
            signature.Key = ae.GetBytes(awsPrivateKey);
            byte[] bytes = ae.GetBytes(stringToConvert);
            byte[] moreBytes = signature.ComputeHash(bytes);
            string encodedCanonical = Convert.ToBase64String(moreBytes);
            string urlEncodedCanonical = HttpUtility.UrlEncode(encodedCanonical).Replace("%3d", "%3D");

            //build up request string (URL)
            string simpleDbUrl = "https://sdb.amazonaws.com/?Action=DeleteAttributes" +
            "&DomainName=SeroterInteractions" +
            "&ItemName=" + itemId +
            "&Version=2009-04-15" +
            "&Timestamp=" + timestamp +
            "&Signature=" + urlEncodedCanonical +
            "&SignatureVersion=2" +
            "&SignatureMethod=HmacSHA1" +
            "&AWSAccessKeyId=ACCESS_KEY";

            HttpWebRequest req = WebRequest.Create(simpleDbUrl) as HttpWebRequest;

            using (HttpWebResponse resp = req.GetResponse() as HttpWebResponse)
            {
                StreamReader reader = new StreamReader(resp.GetResponseStream());

                string responseXml = reader.ReadToEnd();
            }
        }

When the application runs and pulls the message that I sent to the queue earlier, it looks like this.

Nothing too exciting on the user interface, but we’ve just seen the magic that’s happening underneath. After running this (which included reading and deleting the message), the SQS queue is predictably empty.

Force.com Application Pulling from an SQS Queue

I went ahead and sent another message from my Cloud Foundry app into the queue.

This time, I want my cloud CRM users on Salesforce.com to pull these new inquiries and process them.  I’d like to automatically convert the inquiries to CRM Cases in the system.  A custom class in a Force.com application can be scheduled to execute every interval. To account for that (as the solution below supports both on-demand and scheduled retrieval from the queue), I’ve added a couple things to the code.  Specifically, notice that my “case lookup” class implements the Schedulable interface (which allows it be scheduled through the Force.com administrative tooling) and my “queue lookup” function uses the @future annotation (which allows asynchronous invocation).

Much like the .NET application above, you’ll find operations below that retrieve content from the queue and then delete the messages it finds.  The solution differs from the one above in that it DOES handle multiple messages (not that it loops through retrieved results and calls “delete” for each) and also creates a Salesforce.com “case” for each result.

//implement Schedulable to support scheduling
global class doCaseLookup implements Schedulable
{
	//required operation for Schedulable interfaces
    global void execute(SchedulableContext ctx)
    {
        QueueLookup();
    }

    @future(callout=true)
    public static void QueueLookup()
    {
	  //create HTTP objects and queue namespace
     Http httpProxy = new Http();
     HttpRequest sqsReq = new HttpRequest();
     String qns = 'http://queue.amazonaws.com/doc/2009-02-01/';

     //monkey with date format for SQS query
     Datetime currentTime = System.now();
     String formattedTime = currentTime.formatGmt('yyyy-MM-dd')+'T'+ currentTime.formatGmt('HH:mm:ss')+'.'+ currentTime.formatGmt('SSS')+'Z';
     formattedTime = EncodingUtil.urlEncode(formattedTime, 'UTF-8');

	  //build signing string
     String stringToSign = 'GET\nqueue.amazonaws.com\n/084598340988/Seroter_CustomerInquiries\nAWSAccessKeyId=ACCESS_KEY&' +
			'Action=ReceiveMessage&AttributeName=All&MaxNumberOfMessages=5&SignatureMethod=HmacSHA1&SignatureVersion=2&Timestamp=' +
			formattedTime + '&Version=2009-02-01&VisibilityTimeout=15';
     String algorithmName = 'HMacSHA1';
     Blob mac = Crypto.generateMac(algorithmName, Blob.valueOf(stringToSign),Blob.valueOf(PRIVATE_KEY));
     String macUrl = EncodingUtil.urlEncode(EncodingUtil.base64Encode(mac), 'UTF-8');

	  //build SQS URL that retrieves our messages
     String queueUrl = 'https://queue.amazonaws.com/084598340988/Seroter_CustomerInquiries?Action=ReceiveMessage&' +
			'Version=2009-02-01&AttributeName=All&MaxNumberOfMessages=5&VisibilityTimeout=15&Timestamp=' +
			formattedTime + '&Signature=' + macUrl + '&SignatureVersion=2&SignatureMethod=HmacSHA1&AWSAccessKeyId=ACCESS_KEY';

     sqsReq.setEndpoint(queueUrl);
     sqsReq.setMethod('GET');

     //invoke endpoint
     HttpResponse sqsResponse = httpProxy.send(sqsReq);

     Dom.Document responseDoc = sqsResponse.getBodyDocument();
     Dom.XMLNode receiveResponse = responseDoc.getRootElement();
     //receivemessageresult node which holds the responses
     Dom.XMLNode receiveResult = receiveResponse.getChildElements()[0];

     //for each Message node
     for(Dom.XMLNode itemNode: receiveResult.getChildElements())
     {
        String handle= itemNode.getChildElement('ReceiptHandle', qns).getText();
        String body = itemNode.getChildElement('Body', qns).getText();

        //pull out customer ID
        Integer indexSpot = body.indexOf('-');
        String customerId = '';
        if(indexSpot > 0)
        {
           customerId = body.substring(0, indexSpot);
        }

        //delete this message
        DeleteQueueMessage(handle);

	     //create a new case
        Case c = new Case();
        c.Status = 'New';
        c.Origin = 'Web';
        c.Subject = 'Web request: ' + body;
        c.Description = body;

		 //insert the case record into the system
        insert c;
     }
  }

  static void DeleteQueueMessage(string handle)
  {
	 //create HTTP objects
     Http httpProxy = new Http();
     HttpRequest sqsReq = new HttpRequest();

     //encode handle value associated with queue message
     String encodedHandle = EncodingUtil.urlEncode(handle, 'UTF-8');

	 //format the date
     Datetime currentTime = System.now();
     String formattedTime = currentTime.formatGmt('yyyy-MM-dd')+'T'+ currentTime.formatGmt('HH:mm:ss')+'.'+ currentTime.formatGmt('SSS')+'Z';
     formattedTime = EncodingUtil.urlEncode(formattedTime, 'UTF-8');

		//create signing string
     String stringToSign = 'GET\nqueue.amazonaws.com\n/084598340988/Seroter_CustomerInquiries\nAWSAccessKeyId=ACCESS_KEY&' +
					'Action=DeleteMessage&ReceiptHandle=' + encodedHandle + '&SignatureMethod=HmacSHA1&SignatureVersion=2&Timestamp=' +
					formattedTime + '&Version=2009-02-01';
     String algorithmName = 'HMacSHA1';
     Blob mac = Crypto.generateMac(algorithmName, Blob.valueOf(stringToSign),Blob.valueOf(PRIVATE_KEY));
     String macUrl = EncodingUtil.urlEncode(EncodingUtil.base64Encode(mac), 'UTF-8');

	  //create URL string for deleting a mesage
     String queueUrl = 'https://queue.amazonaws.com/084598340988/Seroter_CustomerInquiries?Action=DeleteMessage&' +
					'Version=2009-02-01&ReceiptHandle=' + encodedHandle + '&Timestamp=' + formattedTime + '&Signature=' +
					macUrl + '&SignatureVersion=2&SignatureMethod=HmacSHA1&AWSAccessKeyId=ACCESS_KEY';

     sqsReq.setEndpoint(queueUrl);
     sqsReq.setMethod('GET');

	  //invoke endpoint
     HttpResponse sqsResponse = httpProxy.send(sqsReq);

     Dom.Document responseDoc = sqsResponse.getBodyDocument();
  }
}

When I view my custom APEX page which calls this function, I can see the button to query this queue.

When I click the button, our function retrieves the message from the queue, deletes that message, and creates a Salesforce.com case.

Cool!  This still required me to actively click a button, but we can also make this function run every hour.  In the Salesforce.com configuration screens, we have the option to view Scheduled Jobs.

To actually create the job itself, I had created an Apex class which schedules the job.

global class CaseLookupJobScheduler
{
    global void CaseLookupJobScheduler() {}

    public static void start()
    {
 		// takes in seconds, minutes, hours, day of month, month and day of week
		//the statement below tries to schedule every 5 min, but SFDC only allows hourly
        System.schedule('Case Queue Lookup', '0 5 1-23 * * ?', new doCaseLookup());
    }
}

Note that I use the System.schedule operation. While my statement above says to schedules the doCaseLookup function to run every 5 minutes, in reality, it won’t.  Salesforce.com restricts these jobs from running too frequently and keeps jobs from running more than once per hour. One could technically game the system by using some of the ten allowable polling jobs to set of a series of jobs that start at different times of the hour. I’m not worrying about that here. To invoke this function and schedule the job, I first went to the System Log menu.

From here, I can execute Apex code.  So, I can call my start() function, which should schedule the job.

Now, if I view the Scheduled Jobs view from the Setup screens, I can see that my job is scheduled.

This job is now scheduled to run every hour.  This means that each hour, the queue is polled and any found messages are added to Salesforce.com as cases.  You could use a mix of both solutions and manually poll if you want to (through a button) but allow true asynchronous processing on all ends.

Summary

Asynchronous messaging is a great way to build scalable, loosely coupled systems. A durable intermediary helps provide assurances of message delivery, but this patterns works without it as well.  The demonstrations in this post shows how two cloud solutions can asynchronously exchange data through the use of a shared queue that sits between them.  The publisher to the queue has no idea who will retrieve the message and the retrievers have no direct connection to those who publish messages.  This makes for a very maintainable solution.

My goal with these posts was to demonstrate that classic Integration patterns work fine in cloudy environments. I think it’s important to not throw out existing patterns just because new technologies are introduced. I hope you enjoyed this series.

What Is It?

One uses this remote procedure call (RPC) pattern when they have multiple, independent applications and want to share data or orchestrate cross-application processes. Unlike ETL scenarios where you move data between applications at defined intervals, or the shared database pattern where everyone accesses the same source data, the RPC pattern accesses data/process where it resides. Data typically stays with the source, and the consumer interacts with the other system through defined (service) contracts.

You often see Service Oriented Architecture (SOA) solutions built around the pattern.  That is, exposing reusable, interoperable, abstract interfaces for encapsulated services that interact with one or many systems.  This is a very familiar pattern for developers and good for mashup pages/services or any application that needs to know something (or do something) before it can proceed. You often do not need guaranteed delivery for these services since the caller is notified of any exceptions from the service and can simply retry the invocation.

Challenges

There are a few challenges when leveraging this pattern.

• There is still some coupling involved. While a well-built service exposes an abstract interface that decouples the caller from the service’s underlying implementation, the caller is still bound the service exposed by the system. Changes to that system or unavailability of that system will affect the caller.
• Distinct service and capability offerings by each service. Unlike the shared database pattern where everyone agrees on a data schema and central repository, a RPC model leverages many services that reside all across the organization (or internet). One service may want certificate authentication, another uses Kerberos, and another does some weird token-based security. One service may support WS-Attachment and another may not.  Transactions may or may not be supported between services. In an RPC world, you are at the mercy of each service provider’s capabilities and design.
• RPC is a blocking call. When you call a service that sends a response, you pretty much have to sit around and wait until the response comes back. A caller can design around this a bit using AJAX on a web front end, or using a callback pattern in the middleware tier, but at root, you have a synchronous operation that holds a thread while waiting for a response.
• Queried data may be transient. If an application calls a service, gets some data, and shows it to a user, that data MAY not be persisted in the calling application. It’s cleaner that way, but, this prevents you from using the data in reports or workflows.  So, you simply have to decide early on if your calls to external services should result in persisted data (that must then either by synchronized or checked on future calls) or transient data.
• Package software platforms have mixed support. To be sure, most modern software platforms expose their data via web services. Some will let you query the database directly for information. But, there’s very little consistently. Some platforms expose every tiny function as a service (not very abstract) and some expose giant “DoSomething()” functions that take in a generic “object” (too abstract).

Cloud Considerations

As far as I can tell, you have three scenarios to support when introducing the cloud to this pattern:

• Cloud to cloud. I have one SaaS or custom PaaS application and want to consume data from another SaaS or PaaS application. This should be relatively straightforward, but we’ll talk more in a moment about things to consider.
• On-premises to cloud. There is an on-premises application or messaging engine that wants data from a cloud application. I’d suspect that this is the one that most architects and developers have already played with or built.
• Cloud to on-premises. A cloud application wants to leverage data or processes that sit within an organization’s internal network. For me, this is the killer scenario. The integration strategy for many cloud vendors consists of “give us your data and move/duplicate your processes here.” But until an organization moves entire off-site (if that ever really happens for large enterprises), there is significant investment in the on-premises assets and we want to unlock those and avoid duplication where possible.

So what are the  things to think about when doing RPC in a cloud scenario?

• Security between clouds or to on-premises systems. If integrating two clouds, you need some sort of identity federation, or, you’ll use per-service credentials. That can get tough to manage over time, so it would be nice to leverage cloud providers that can share identity providers. When consuming on premises services from cloud-based applications, you have two clear choices:
  • Use a VPN. This works if you are doing integration with an IaaS-based application where you control the cloud environment a bit (e.g. Amazon Virtual Private Cloud). You can also pull this off a bit with things like the Google Secure Data Connector (for Google Apps for GAE) or Windows Azure Connect.
  • Leverage a reverse proxy and expose data/services to public internet. We can define a intermediary that sits in an internet-facing zone and forwards traffic behind the firewall to the actual services to invoke. Even if this is secured well, some organizations may be wary to expose key business functions or data to the internet.
• There may be additional latency. For some application, especially based on location, there could be a longer delay when doing these blocking remote procedure calls.  But more likely, you’ll have additional latency due to security.  That is, many providers have a two step process where the first service call against the cloud platform is for getting a security token, and the second call is the actual function call (with the token in the payload).  You may be able to cache the token to avoid the double-hop each time, but this is still something to factor in.
• Expect to only use HTTP. Few (if any) SaaS applications expose their underlying database. You may be used to doing quick calls against another system by querying it’s data store, but that’s likely a non-starter when working with cloud applications.

The one option for cloud-to-on-premises that I left out here, and one that I’m convinced is a differentiating piece of Microsoft software, is the Azure AppFabric Service Bus.  Using this technology, I can securely expose on-premises services to the public internet WITHOUT the use of a VPN or reverse proxy. And, these services can be consumed by a wide variety of platforms.  In fact, that’s the basis for the upcoming demonstration.

Solution Demonstration

So what if I have a cloud-based SaaS/PaaS application, say Salesforce.com, and I want to leverage a business service that sits on site.  Specifically, the fictitious Seroter Corporation, a leader in fictitious manufacturing, has an algorithm that they’ve built to calculate the best discount that they can give a vendor. When they moved their CRM platform to Salesforce.com, their sales team still needed access to this calculation. Instead of duplicating the algorithm in their Force.com application, they wanted to access the existing service. Enter the Azure AppFabric Service Bus.

Instead of exposing the business service via VPN or reverse proxy, they used the AppFabric Service Bus and the Force.com application simply invokes the service and shows the results.  Note that this pattern (and example) is very similar to the one that I demonstrated in my new book. The only difference is that I’m going directly at the service here instead of going through a BizTalk Server (as I did in the book).

WCF Service Exposed Via Azure AppFabric Service Bus

I built a simple Windows Console application to host my RESTful web service. Note that I did this with the 1.0 version of the AppFabric Service Bus SDK.  The contract for the “Discount Service” looks like this:

[ServiceContract]
    public interface IDiscountService
    {
        [WebGet(UriTemplate = "/{accountId}/Discount")]
        [OperationContract]
        Discount GetDiscountDetails(string accountId);
    }

    [DataContract(Namespace = "http://CloudRealTime")]
    public class Discount
    {
        [DataMember]
        public string AccountId { get; set; }
        [DataMember]
        public string DateDelivered { get; set; }
        [DataMember]
        public float DiscountPercentage { get; set; }
        [DataMember]
        public bool IsBestRate { get; set; }
    }

My implementation of this contract is shockingly robust.  If the customer’s ID is equal to 200, they get 10% off.  Otherwise, 5%.

public class DiscountService: IDiscountService
    {
        public Discount GetDiscountDetails(string accountId)
        {
            Discount d = new Discount();
            d.DateDelivered = DateTime.Now.ToShortDateString();
            d.AccountId = accountId;

            if (accountId == "200")
            {
                d.DiscountPercentage = .10F;
                d.IsBestRate = true;
            }
            else
            {
                d.DiscountPercentage = .05F;
                d.IsBestRate = false;
            }

            return d;

        }
    }

The secret sauce to any Azure AppFabric Service Bus connection lies in the configuration.  This is where we can tell the service to bind to the Microsoft cloud and provide the address and credentials to do so. My full configuration file looks like this:

<configuration>
<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup><system.serviceModel>
        <behaviors>
            <endpointBehaviors>
                <behavior name="CloudEndpointBehavior">
                    <webHttp />
                    <transportClientEndpointBehavior>
                        <clientCredentials>
                          <sharedSecret issuerName="ISSUER" issuerSecret="SECRET" />
                        </clientCredentials>
                    </transportClientEndpointBehavior>
                    <serviceRegistrySettings discoveryMode="Public" />
                </behavior>
            </endpointBehaviors>
        </behaviors>
        <bindings>
            <webHttpRelayBinding>
              <binding name="CloudBinding">
                <security relayClientAuthenticationType="None" />
              </binding>
            </webHttpRelayBinding>
        </bindings>
        <services>
            <service name="QCon.Demos.CloudRealTime.DiscountSvc.DiscountService">
                <endpoint address="https://richardseroter.servicebus.windows.net/DiscountService"
                    behaviorConfiguration="CloudEndpointBehavior" binding="webHttpRelayBinding"
                    bindingConfiguration="CloudBinding" name="WebHttpRelayEndpoint"
                    contract="IDiscountService" />
            </service>
        </services>
    </system.serviceModel>
</configuration>

I built this demo both with and without client security turned on.  As you see above, my last version of the demonstration turned off client security.

In the example above, if I send a request from my Force.com application to https://richardseroter.servicebus.windows.net/DiscountService, my request is relayed from the Microsoft cloud to my live on-premises service. When I test this out from the browser (which is why I earlier turned off client security), I can see that passing in a customer ID of 200 in the URL results in a discount of 10%.

Calling the AppFabric Service Bus from Salesforce.com

With an internet-accessible service ready to go, all that’s left is to invoke it from my custom Force.com page. My page has a button where the user can invoke the service and review the results.  The results may, or may not, get saved to the customer record.  It’s up to the user. The Force.com page uses a custom controller that has the operation which calls the Azure AppFabric endpoint. Note that I’ve had some freakiness lately with this where I get back certificate errors from Azure.  I don’t know what that’s about and am not sure if it’s an Azure problem or Force.com problem.  But, if I call it a few times, it works.  Hence, I had to add exception handling logic to my code!

public class accountDiscountExtension{

    //account variable
    private final Account myAcct;

    //constructor which sets the reference to the account being viewed
    public accountDiscountExtension(ApexPages.StandardController controller) {
        this.myAcct = (Account)controller.getRecord();
    }

    public void GetDiscountDetails()
    {
        //define HTTP variables
        Http httpProxy = new Http();
        HttpRequest acReq = new HttpRequest();
        HttpRequest sbReq = new HttpRequest();

        // ** Getting Security Token from STS
       String acUrl = 'https://richardseroter-sb.accesscontrol.windows.net/WRAPV0.9/';
       String encodedPW = EncodingUtil.urlEncode(acsKey, 'UTF-8');

       acReq.setEndpoint(acUrl);
       acReq.setMethod('POST');
       acReq.setBody('wrap_name=ISSUER&wrap_password=' + encodedPW + '&wrap_scope=http://richardseroter.servicebus.windows.net/');
       acReq.setHeader('Content-Type','application/x-www-form-urlencoded');

       //** commented out since we turned off client security
       //HttpResponse acRes = httpProxy.send(acReq);
       //String acResult = acRes.getBody();

       // clean up result
       //String suffixRemoved = acResult.split('&')[0];
       //String prefixRemoved = suffixRemoved.split('=')[1];
       //String decodedToken = EncodingUtil.urlDecode(prefixRemoved, 'UTF-8');
       //String finalToken = 'WRAP access_token=\"' + decodedToken + '\"';

       // setup service bus call
       String sbUrl = 'https://richardseroter.servicebus.windows.net/DiscountService/' + myAcct.AccountNumber + '/Discount';
        sbReq.setEndpoint(sbUrl);
       sbReq.setMethod('GET');
       sbReq.setHeader('Content-Type', 'text/xml');

       //** commented out the piece that adds the security token to the header
       //sbReq.setHeader('Authorization', finalToken);

       try
       {
       // invoke Service Bus URL
       HttpResponse sbRes = httpProxy.send(sbReq);
       Dom.Document responseDoc = sbRes.getBodyDocument();
       Dom.XMLNode root = responseDoc.getRootElement();

       //grab response values
       Dom.XMLNode perNode = root.getChildElement('DiscountPercentage', 'http://CloudRealTime');
       Dom.XMLNode lastUpdatedNode = root.getChildElement('DateDelivered', 'http://CloudRealTime');
       Dom.XMLNode isBestPriceNode = root.getChildElement('IsBestRate', 'http://CloudRealTime');

       Decimal perValue;
       String lastUpdatedValue;
       Boolean isBestPriceValue;

       if(perNode == null)
       {
           perValue = 0;
       }
       else
       {
           perValue = Decimal.valueOf(perNode.getText());
       }

       if(lastUpdatedNode == null)
       {
           lastUpdatedValue = '';
       }
       else
       {
           lastUpdatedValue = lastUpdatedNode.getText();
       }

       if(isBestPriceNode == null)
       {
           isBestPriceValue = false;
       }
       else
       {
           isBestPriceValue = Boolean.valueOf(isBestPriceNode.getText());
       }

       //set account object values to service result values
       myAcct.DiscountPercentage__c = perValue;
       myAcct.DiscountLastUpdated__c = lastUpdatedValue;
       myAcct.DiscountBestPrice__c = isBestPriceValue;

       myAcct.Description = 'Successful query.';
       }
       catch(System.CalloutException e)
       {
          myAcct.Description = 'Oops.  Try again';
       }
    }

Got all that? Just a pair of calls.  The first gets the token from the Access Control Service (and this code likely changes when I upgrade this to use ACS v2) and the second invokes the service.  Then there’s just a bit of housekeeping to handle empty values before finally setting the values that will show up on screen.

When I invoke my service (using the “Get Discount” button, the controller is invoked and I make a remote call to my AppFabric Service Bus endpoint. The customer below has an account number equal to 200, and thus the returned discount percentage is 10%.

Summary

Using a remote procedure invocation is great when you need to request data or when you send data somewhere and absolutely have to wait for a response. Cloud applications introduce some wrinkles here as you try to architect secure, high performing queries that span clouds or bridge clouds to on-premises applications. In this example, I showed how one can quickly and easily expose internal services to public cloud applications by using the Windows Azure AppFabric Service Bus.  Regardless of the technology or implementation pattern, we all will be spending a lot of time in the foreseeable future building hybrid architectures so the more familiar we get with the options, the better!

In the final post in this series, I’ll take a look at using asynchronous messaging between (cloud) systems.