Evaluation Criteria for SaaS/Cloud Platform Vendors

My company has been evaluating (and in some cases, selecting) SaaS offerings and one of the projects that I’m currently on has us considering such an option as well.  So, I started considering the technology-specific evaluation criteria (e.g. not hosting provider’s financial viability) that I would use to determine our organizational fit to a particular cloud/SaaS/ASP vendor.  I’m lumping cloud/SaaS/ASP into a bucket of anyone who offers me an off-premises application.  When I finished a first pass of the evaluation, my list looked a whole lot like my criteria for standard on-premises apps, with a few obvious omissions and modifications.

First off, what are the things that I should have an understanding of, but am assuming that I have little control over and that  the service provider will simply “do for me” (take responsibility for)?

Category	Considerations / Questions
Scalability	How does the platform accommodate more users and increasingly intensive calculations? Is there intelligent application-specific delivery capabilities available or just basic server load balancing?
Availability	How do you maintain high uptime for both domestic and international users?
Manageability	What (user and programmatic) interfaces do I have to manage the application? How can on-premises administrators mash up your client-facing management tools with their own?
Hardware/Software	What is the underlying technology of the cloud platform or specific instance details for the ASP provider?
Storage	What are the storage limits and how do I scale up to more space?
Network configuration and modeling	How is the network optimized with regards to connectivity, capacity, load balancing, encryption and quality of service? What is the firewall landscape and how does that impact how we interact with you?
Disaster recovery	What is the DR procedure and what is the expected RPO and RTO?
Data retention	Are there specific retention policies for data or does it stay in the active transaction repository forever?
Concurrency	How are transactions managed and resource contention handled?
Patch management	What is the policy for updating the underlying platform and how are release notes shared?
Security	How do you handle data protection and compliance with international data privacy laws and regulations? How is data securely captured, stored, and accessed in a restricted fashion? Are there local data centers where country/region specific content can reside?
User Interfaces	Are there mobile interfaces available?

So far, I’m not a believer that the cloud is simply a place to stash an application/capability and that I need not worry about interacting with anything in that provider’s sandbox.  I still see a number of integration points between the cloud app and the infrastructure residing on premises.  Until EVERYTHING is in the cloud (and I have to deal with cloud-to-cloud integration), I still need to deal with on-premises applications. This next list addresses the key aspects that will determine if the provider can fit into our organization and its existing on-site investments (in people and systems).

Category	Considerations / Questions
Security	How do I federate our existing identity store with yours? What is the process for notifying you of a change in employment status (hire/fire)? Are we able to share entitlements in a central way so that we can own the full provisioning of users?
Backwards compatibility of changes	What is the typical impact of a back end change on your public API? Do you allow direct access to application databases and if so, how are your environment updates made backwards compatible? Which “dimensions of change” (i.e. functional changes, platform changes, environment changes) will impact any on-premises processes, mashups, or systems that we have depending on your application?
Information sharing patterns	What is your standard information sharing interface?  FTP?  HTTP? How is master data shared in each direction? How is reference data shared in each direction? Do you have both batch (bulk) and real-time (messaging) interfaces? How is initial data load handled? How would you propose handling enterprise data definitions that we use within our organizations?  Adapters with transformation on your side or our side? How is information shared between our organizations securely?  What are your standard techniques? For real-time data sharing, do you guarantee once-only, reliable delivery?
Access to analytics and reporting	How do we access your reporting interface? How is ad-hoc reporting achieved? Do we get access to the raw data in order extract a subset and pull it in house for analysis?
User interface customization	What are the options for customizing the user interface? Does this require code or configuration?  By whom?
Globalization /  localization	How do you handle the wide range of character sets, languages, text orientations and units of measure prevalent in an international organization?
Exploiting on-premises capabilities	Can this application make use of any existing on-premises infrastructure capabilities such as email, identity, web conferencing, analytics, telephony, etc?
Exception management	What are the options for application/security/process exception notification and procedures?
Metadata ownership	While I don’t expect cloud portability, who owns the metadata that defines the cloud application configuration?
Locked in components	What aspects of your solution are proprietary and “locked in” and can only be part of an application in your cloud platform?
Developer toolkit	What is the developer experience for our team when interfacing with your cloud platform and services?  Are there SDKs, libraries and code samples?
Enhancement cost	What types of changes to the application incur a cost to me (e.g. changing a UI through configuration, building new reports, establishing new API interfaces)?

This is a work in progress.  There are colleagues of mine doing more thorough investigations into our overall cloud strategy, but I figured that I’d take this list out of OneNote and expose it to the light of day.  Feel free to point out glaring mistakes or omissions.

As an aside, the two links I included in the lists above point to the Dev Central blog from F5.  I’ll tell you what, this has really become one of my “must read” blogs for technology concepts and infrastructure thoughts.  Highly recommended regardless of whether or not you use their products.  It’s thoughtfully written and well reasoned.

Technorati Tags: Architecture, Cloud Computing