Chuck the Checklist
Although I understand the temptation to line up competing products side-by-side so that you can compare features, as most business analysts will tell you that you should compare “apples-to-apples” and “oranges-to-oranges” and follow the common practice of the trade rags which publish page after page of tinted paper that “goes deep” on comparing product A to B to C, I also understand that you should resist this temptation and not do this. There are too many problems with this approach, including:

• It’s impossible to sum up a feature in one sentence, or even a short paragraph.
  Product differences are never as simple as a “yes/no” in a 400 line grid. For example, when comparing two ERP systems, it’s much more intrinsically important to understand that one product gives you direct access to its internal schema while the other does not. Without direct access, you’ll have to hire an outside integrator to “fix” your application interfaces on every upgrade, as you won’t have the information you need to do it yourself. And at $1,500 to $2,500 per consulting day, this will add up very quickly.
• No piece of software can do everything.
  A vendor might say that “we can do XYZ” and be correct in principle, but leave out that it will take days of effort, that it’s a non-repeatable process, that the results are only available on a static, non-downloadable, HTML page, or that it will cost you $2,000 a day in consulting services every time you need it. Will these subtleties show up in the matrix? Of course not!
• Trade Rags and Analysts Give you Bad Lists.
  If you use a matrix generated by a trade rag, or analyst firm, which items will show up on the list out of the multitude that could be selected? The items promoted by the big vendors with the most marketing people (who get the most face time with the analysts and trade rag editors), of course. But are those items really the important items? Maybe to their legacy customers, but not necessarily to you! You need to remember that the majority of innovation comes from new start-ups and small firms that the analysts and trade-rags don’t even cover yet!
• RFP Templates are Poison Pills
  I’ve said it many times before, and I’ll say it again. If you use, or modify, an RFP template that a vendor makes “freely available” to you, you’ve dug your own grave. Those templates are filled with “must have” irrelevant features designed to make the vendor look good and their primary competition look bad. That’s why the doctor gave you questions you need to ask in his X-emplication and X-asperation series and not useless matrices. Although there are a few common elements that every customer will need in a solution for X, the specific feature lists are different for every customer and depend on their current processes and platforms.

Instead of a checklist, hire a consultant (like the doctor) who can help you understand what your true needs really are, what the relevant differences are between the software solutions you’re considering, and which features are most important to your company. This endeavor, which will likely only cost you a few thousand, could save you hundreds of thousands of dollars.

Wait for the Blush to Leave the Rose
Although testimonials and references from customers who have recently implemented the software product will usually be sincere, they will always be useless. Why?

1. New customers are highly motivated to say the software is great.
   They just spent a bundle on the product and they are under great pressure to justify the purchase to their CFO. They’ll blindly see it as a success even if it’s not for quite some time.
2. Almost all software solutions are better than no software solution at all.
   Software automates tasks, simplifies processes, and ultimately solves some problems. There will almost be some initial euphoria over the fact that at least part of a task has been simplified. Wait for the euphoria to die down and for people to start complaining about the annoyances. Only then do you find out if it’s worth the money or not. If they worst complaint is “the UI is ugly”, then it’s probably worth the dough, but if many users complain “I can’t even requisition a stapler through this gawd-awful piece of cr@p”, then you know it’s definitely not.

For a testimonial to be of value to you, it should be from a customer who has used the software for at least a year, and preferably two. Then you get a fuller, more balanced story about what’s good and what’s not so good. (But that’s why software vendors ask for reviews and testimonials while the blush is still on the rose. They don’t want you to see the thorns.)

Of course, the best testimonial or reference comes from a customer who has not only been using the solution for at least two years, but also left a company, moved to a new company, and bought the exact same software again. This person has been through the wringer with the product, warts and all, understands exactly what she’s getting, and wants more. That’s a testimonial. Or from a user who wasn’t the original purchaser of the software but inherited it, such as a new CPO taking over, and who has no history with the company. If the new user is excited about the product, that’s also a testimonial. But you won’t get either if you don’t let the blush leave the rose.