Posts tagged "selecting a cms"

Selecting a CMS: Managing Product Demos

If you followed the advice from the first two articles in this series (How to build a short listand Developing scenarios), you should have a good idea of what you are looking for and with what products you might find some content management system bliss. This next article provides guidance on how you can start evaluating actual products against your defined requirements.


This next phase of the selection process is where you evaluate the products against your requirements. Successful completion of this phase will mean that you have selected a product/vendor that is compatible with your content and your way of working. The product satisfies both your objective and subjective criteria.

Failure in this phase means that you will either be swayed by the most charismatic salesperson or that you will be stuck in a never ending sales cycle that doesn't drive you towards an informed decision. Neither case is very appealing — so lets avoid both.

Take Product Demos Seriously

The vendor presentation and product demonstration is one of the most critical components in a CMS selection process. You will learn more from seeing a product in action than reading an analyst report or a RFP response from a vendor.

But to be effective, a product demonstration needs considerable investment from both sides. You won't learn anything by occasionally peaking up from your email to glance at a canned demo about a fictional business that has nothing to do with your company.

Instead, you should partner with the vendor to develop a prototype that supports the scenarios you have given in the RFP. In this exercise, you get to experience what it feels like to be a customer working with the vendor to achieve success. If you run a demonstration properly you will be able to answer the following questions:

  • Does the vendor understand your business and the way you work?
  • Will you be treated like an important customer?
  • Does your company and the vendor have good chemistry?
  • How naturally does the product fit your vision?
  • What customizations or compromises would have to be made to use this product?

Construct a Thoughtful Invitation

The first thing you need to do is connect with the vendors on your short list to tell them that you are evaluating their product and could use their help. This is where most companies go wrong. By pulling out their standard RFP template and loading it up with demands one shoots oneself in the foot, and early.

The worst of such offenses have a glob of irrelevant boilerplate text and then a long feature checklist. One CMS vendor I know even received an RFP that asked if the product contained any radioactive materials — clearly this was language designed for another type of procurement and the customer was too lazy to even read what he sent out.

Trust me, when you do work like this, you are sending a signal to the vendors that you don't care. Some vendors will not engage at all. Others will play along but invest as little as possible in the opportunity — they know that the sales cycle will be long and unpredictable.

You get what you give: an RFP with a lot of boilerplate text will get responses with a lot of boilerplate text. Plus, you will get stuck in "qualification" queue until you show some signs of intelligent life.

The good news is that, if you have done the work of developing scenarios, you have a lot of information that shows (a) you are serious about this initiative and, (b) you know what you want.

Vendors love scenarios because they efficiently tell the story of content in your organization and help them understand what you need. In addition to your 10 most important scenarios, your RFP should contain the following information:

  • Background about your company and division
  • Sample content types and perhaps some screen captures of how they appear on your site
  • A roadmap of your selection process with a timetable
  • A point of contact
  • The response format you you expect.

Be Concise in Your Communications

Attributed to an impressive number of famous folk including Mark Twain, Pascal and a host of others, there's a quote that I find to be a useful and humorous reminder. It goes something like this: "

If i'd had more time, i'd have writen a much shorter letter."

The most important part to keep in mind is that assembling your RFP is not a contest for who can write the longest, most elaborate proposal. The RFP response will not help you manage your content and the quality of an RFP response says nothing about the product. In fact, the prettiest proposals are usually written by a dedicated proposal writer and re-used across lots of sales opportunities — they have very little to do with your RFP.

Some RFPs demand that participants spend weeks of time filling out a response. You don't want the vendors to spend all of their effort on the RFP and then coast through the rest of the sales process. Vendor resources are much better spent building a prototype that shows you how the product would work in your organization. This will give the vendors a chance to show how they approach problems and how their products work. Keep things as informal as you can.  The more leeway you give, the easier it will be easier to identify differences between the vendors and products. When differences are more visible, decisions are easier.

If you really want a vendor to put in the effort to get to know you and translate their features in terms of your requirements, you should let it be known that you are only evaluating 2-3 products. This indicates that you are in the home-stretch of your decision and it is time to pull out the stops. This will help the vendors justify putting their best people on the opportunity rather than pacing themselves for a long slog. Time-boxing a decision helps people work more efficiently on both sides.


A successful demo is all about preparation. You need to prepare the vendor — or systems integrator or in house staff if you are evaluating non-commercial software — with the information they need so they can do their best. You also need to prepare the audience on what they should be looking for.

  • Verify that the vendor understands your requirements
    Have the vendor prepare a written response describing how their product can support your scenarios. Review it and give them feedback with ample time to adjust their demo in case they misunderstood what you need. I typically encourage vendors to do a pre-demo walk-through of the scenario in front of their customer contact person. If you are a vendor, always take advantage of this offer. In my own selection consulting work, when one of the three candidates does a pre-demo walk-through, the demos are so much better that they win 100% of the time.
  • Prepare the audience 
    Prepare your audience for the demo by telling them what they should be looking for. A scorecard that lists the scenarios is useful for keeping people's attention on their needs, not gimicky features. Vendors tend to up their game when the realize they are dealing with a sophisticated audience.

    If the audience does not understand basic content management theory (separation of content and layout, re-usability, content life-cycle, etc.) address that before the first demo. Vendors are actually pretty good at explaining that stuff but there are more effective uses of their time.

  • Call the vendor's references
    Don't wait to the end of the process to call references. If you talk to references before the demonstration, you will be more educated for the demo. Maybe a complaint from a reference was addressed in a newer version of the software. Maybe a feature that demos really nicely isn't practical for everyday use.


The demo should use everyone's time as effectively as possible and should be organized to ensure that vital information is communicated to the right people. I usually allocate 4 hours but the agenda is broken up so that not all stakeholders have to sit through the whole thing.

  • Limit company background information 
    The vendor should be able to introduce their company and make the case that it is a stable company, it gets content management, and knows your business. However, you need to contain the amount of time that they take to do it. They should be able to build a level of credibility and comfort with the audience but not infringe on the time they have to talk about their product within your context. Your short-listing exercise should have already pre-qualified the vendors along these lines.
  • Mind your manners 
    Even if your corporate culture thinks it is OK for staff to attend meetings in-body only, keep distractions to a minimum. Ask your audience to put aside their email, blackberries, and cell phones and pay attention. Give the vendors every opportunity to engage with the audience. If the vendor is missing the mark, don't tune out. Instead, help steer them back on course. If you can't do that, politely end the meeting as quickly as possible and be happy that you were able to eliminate an option in a very hard decision.
  • Mark your scorecards
    Without making it feel like a Bingo hall, have the audience take notes in their scorecards so that they remember what they saw and their impressions. By the time they have gotten back to their desks and answered their first of fifty waiting emails, they will have forgotten half of what they saw. The most important thing for your audience to capture is their doubts. These are aspects of the product or service that raise questions and concerns. The follow up phase will focus on these doubts.

Follow up

Don't let wait long to get feedback from the audience. It doesn't take long for people to forget. Follow up and plan the next steps as soon as possible.

  • The post mortem
    As soon as possible, get everyone in a room and have them express their observations and impressions and, most importantly, doubts. For each doubt, you need to first validate (was this a misunderstanding or an oversight?), mitigate (what compromises or customizations could compensate for this issue?), and rate (what risk remains after mitigation? Is the customization expensive or does it risk future upgrades? Is the compromise sustainable?).
  • Schedule follow ups
    Review the doubts that came up and have the vendor invalidate or suggest mitigation strategies. For the vendors that didn't make the cut, explain why. If the demo was a disaster but you think the product still has potential, you could give them another chance or you could take it as a sign that they are not prepared to support you. Remember, after the contract is signed, things are only going to get worse.
  • Prototype
    Some doubts will be best addressed with a prototype that the vendor can leave behind for you to use. Different vendors will have different policies around this. Some create hosted sand boxes and allow business users to experiment. Others provide trial versions of the software so that a customer can attend training and try to build the prototype themselves.
Selecting a CMS: Developing Usage Scenarios

In my last article, I described how to avoid the analysis-paralysis trap and quickly make your way to a short list of content management software options. If you missed that article, check out Selecting a CMS: How to Build a Short List.  If you followed the recommended approach, you should have a good idea of your high-level vision (what type of website you need to manage), your financial and technical constraints and few promising products to look at.

In this article I describe how to define some practical usage scenarios which you will use to shape the product evaluation process.

Up to this point you have been thinking very objectively — asking direct yes/no questions and eliminating anything with a "no" response.  But you haven't addressed any of the subjective factors like efficiency, usability and manageability.  All of the products on your short list could work  — given enough compromises and customization — but that isn't good enough. You want something better than nearly adequate.

Writing Effective Scenarios

To find the product that will be the best fit for your organization, you need to dig into your requirements; but put those spreadsheets away.  Spreadsheets are great for naming features but they won't guide you to understand how these products would work with your users and the content that your organization needs to manage.

This is where scenarios come in

A scenario is a short story — written in a language that regular people understand; not business analyst-speak —  that describes a user's interaction with the system to achieve a business objective. A scenario encapsulates lots of specific requirements and gives them greater meaning and context.

These are the four attributes of an effective scenario:

  1. It is written with specific users in mind
  2. It addresses an important and commonly executed task
  3. It references the content that you intend to manage
  4. It is open-ended enough to expose the difference in product design and approach

That list was dense so let me unpack it a bit.

1. Understanding Specific Users

Understanding users is the most important element of a CMS project because in the end, a content management system is a tool for users. The perceived success or failure of your project will hinge on how effective those users feel when they interact with the system. 

I typically write scenarios for four different user groups:

  • content contributors
  • content consumers or visitors
  • system administrators
  • software developers

Each of these groups have different responsibilities that can either be eased or frustrated by the technology: 

  • Scenarios for the content contributor will be about adding, editing, organizing, finding and approving content. 
  • Visitor scenarios will be about front end functionality like searching, reading content on different devices and potentially interactive functionality like commenting and user profile management.
  • Administrators will be concerned with managing contributor accounts, system upgrades and backups. 
  • Developers will need to define content types, develop presentation templates and potentially integrate with other technologies.  

Some people like to build "personas" that give personality and character to these abstract user types but I don't think that adds much value to this exercise. You might as well think of the real person (your actual co-worker or customer) than some made up symbol of him. Personae are more useful for branding and design exercises. But please don't let that stop you from creating an imaginary friend if you want to — we all have our special needs.

2. Prioritizing the Tasks

For each of the user groups outlined above, brainstorm their most important activities with regards to the system. 

What do they spend their time on? What do their "clients" hound them for?  What are they most afraid of messing up? All these are good candidates for scenarios. But don't forget the tasks that they take for granted with their current systems and processes. A new CMS means the old tools go away and even the worst systems do somethings well.

While the types of scenarios will be similar across CMS buyers, the details of how the tasks need to be executed can vary widely. This is why simply naming tasks is not enough. 

For example, if the scenario involves finding a piece of content, we need to think about what information the user starts with. Does he just have some key words that he thinks should be mentioned in the title or the body of the asset? Does he know the URL or the path? What additional information will help him filter down to the asset? The date it was published?  Where it occurs on the site?

This is why scenarios are so different from features. All CMS products will claim to support finding content; the differences will be in these critical details.

While how people work is important, be careful not to get mired in your dysfunctional processes.  Some tasks will go away or be redefined when your broken system gets retired. With that said, don't wander down that slippery slope of business process re-engineering either. Stumble onto that path and before you know it your project will get too big and political. Focus on not re-creating your big, obvious problems

3. Deconstructing Content

When thinking about your content, there are two dimensions to consider: structure and organization.

From a structural perspective, content is often more than simple pages with a title and body. Some content is hierarchical and inter-related. For example, I recently completed a project for a museum that had exhibitions with start and end dates, and ordered collections of works of art. Each work of art has a reference to an artist. Works of art could be re-used across exhibitions and could also be in permanent collections.

This example also presents a question of organization. An individual work of art can appear in different contexts but one is considered the primary. By making the structure and organization of your content part of your scenarios, you will get to see different approaches to meeting these requirements. You will see how the user interface handles input validation, content relationship building and content placement.

It is a good idea to include a model of one of the more complex types with the scenarios. It doesn't have to be fancy. A simple outline with field names and data types will do. If there are rules that restrict privileges to different areas of the site, it is a good idea to include a site map with those rules as well. Some CMS restrict access by content type; others restrict by placement in the content tree; others do a hybrid of both. Know the implications of the vendor's approach in regards to how it will interact with your business rules.

4. Using, not Abusing the Level of Detail

When you are writing these scenarios, be careful not to be overly prescriptive or rigid about how the system works. Some details will not be relevant to certain systems. Focus on the details that are important to your business rather than arbitrary implementation decisions like how you might navigate to a piece of content or what the save button is called.

Save time by glossing over functionality that is more or less homogeneous and ubiquitous across the marketplace. For example, nearly all of the Web CMS platforms on the market use a handful of third party rich text editors. Rather than describing mundane features like bolding and underlining, concentrate on areas where they differ like browsing and searching through content repository to find links to other assets.

If you have a requirement that different types of users should have more control over rich text than others  — like the ability to embed  objects and JavaScript — describe those types of rules in your scenarios.

The scenarios that I write tend to be between one quarter and three quarters of a page. If a scenario is really long, maybe it needs to be broken up into multiple scenarios. Or maybe it just has too much detail.

Using Your Scenarios

Once you have written you interaction scenarios, it is time to put them to work.

First, you will include your scenarios in your RFP. This will help educate the supplier about how you work and what you are trying to do with their software. Keep in mind that the scenarios are also an excellent means to beginning a dialog that will map your specific needs to the features and configurations of a vendor's software. They speak in ways that simple requirements matrices never can.

Next, you should plan and construct your evaluation processes around these scenarios. The scenarios — and your content model — will dictate at least part of your product demonstrations. This is key. If vendors fail to address your scenarios, then work with them to reformat the demonstrations, or recognize that the vendor is side-stepping your demands.

Next Up: Custom Product Demonstrations

Stay tuned. In my next article, I will walk you through preparing for and conducting customized product demos, including how to best process what you have seen. In the mean time, the following articles provide valuable context and advice for the software selection process.