Chuck the Checklist
Although I understand the temptation to line up competing products side-by-side so that you can compare features, as most business analysts will tell you that you should compare “apples-to-apples” and “oranges-to-oranges” and follow the common practice of the trade rags which publish page after page of tinted paper that “goes deep” on comparing product A to B to C, I also understand that you should resist this temptation and not do this. There are too many problems with this approach, including:

• It’s impossible to sum up a feature in one sentence, or even a short paragraph.
  Product differences are never as simple as a “yes/no” in a 400 line grid. For example, when comparing two ERP systems, it’s much more intrinsically important to understand that one product gives you direct access to its internal schema while the other does not. Without direct access, you’ll have to hire an outside integrator to “fix” your application interfaces on every upgrade, as you won’t have the information you need to do it yourself. And at $1,500 to $2,500 per consulting day, this will add up very quickly.
• No piece of software can do everything.
  A vendor might say that “we can do XYZ” and be correct in principle, but leave out that it will take days of effort, that it’s a non-repeatable process, that the results are only available on a static, non-downloadable, HTML page, or that it will cost you $2,000 a day in consulting services every time you need it. Will these subtleties show up in the matrix? Of course not!
• Trade Rags and Analysts Give you Bad Lists.
  If you use a matrix generated by a trade rag, or analyst firm, which items will show up on the list out of the multitude that could be selected? The items promoted by the big vendors with the most marketing people (who get the most face time with the analysts and trade rag editors), of course. But are those items really the important items? Maybe to their legacy customers, but not necessarily to you! You need to remember that the majority of innovation comes from new start-ups and small firms that the analysts and trade-rags don’t even cover yet!
• RFP Templates are Poison Pills
  I’ve said it many times before, and I’ll say it again. If you use, or modify, an RFP template that a vendor makes “freely available” to you, you’ve dug your own grave. Those templates are filled with “must have” irrelevant features designed to make the vendor look good and their primary competition look bad. That’s why the doctor gave you questions you need to ask in his X-emplication and X-asperation series and not useless matrices. Although there are a few common elements that every customer will need in a solution for X, the specific feature lists are different for every customer and depend on their current processes and platforms.

Instead of a checklist, hire a consultant (like the doctor) who can help you understand what your true needs really are, what the relevant differences are between the software solutions you’re considering, and which features are most important to your company. This endeavor, which will likely only cost you a few thousand, could save you hundreds of thousands of dollars.

Wait for the Blush to Leave the Rose
Although testimonials and references from customers who have recently implemented the software product will usually be sincere, they will always be useless. Why?

1. New customers are highly motivated to say the software is great.
   They just spent a bundle on the product and they are under great pressure to justify the purchase to their CFO. They’ll blindly see it as a success even if it’s not for quite some time.
2. Almost all software solutions are better than no software solution at all.
   Software automates tasks, simplifies processes, and ultimately solves some problems. There will almost be some initial euphoria over the fact that at least part of a task has been simplified. Wait for the euphoria to die down and for people to start complaining about the annoyances. Only then do you find out if it’s worth the money or not. If they worst complaint is “the UI is ugly”, then it’s probably worth the dough, but if many users complain “I can’t even requisition a stapler through this gawd-awful piece of cr@p”, then you know it’s definitely not.

For a testimonial to be of value to you, it should be from a customer who has used the software for at least a year, and preferably two. Then you get a fuller, more balanced story about what’s good and what’s not so good. (But that’s why software vendors ask for reviews and testimonials while the blush is still on the rose. They don’t want you to see the thorns.)

Of course, the best testimonial or reference comes from a customer who has not only been using the solution for at least two years, but also left a company, moved to a new company, and bought the exact same software again. This person has been through the wringer with the product, warts and all, understands exactly what she’s getting, and wants more. That’s a testimonial. Or from a user who wasn’t the original purchaser of the software but inherited it, such as a new CPO taking over, and who has no history with the company. If the new user is excited about the product, that’s also a testimonial. But you won’t get either if you don’t let the blush leave the rose.

Don’t Get Blind-Sided by IT
How many times has this happened to you. You identify a new software program that will make your life easier and deliver ROI to your department, but just before you sign the deal, IT steps in, in a very public way, and says “we already have a product that can do that and we have lots of extra licenses“, even though the product doesn’t do what you need it to, doesn’t deliver ROI, and, to top it off, makes your life harder than it would be if you had no solution at all!

This doesn’t have to happen. Especially since you’re all on the same team and the only reason you want to buy the product is because it solves a problem that no other product in your arsenal does. To avoid this scenario, be sure to make your ROI case with the IT department first and get them on board. That way you can make sure that none of the solutions available solves your problem and present an even stronger case to management.

This will also address the slightly less common but still problematic “we can develop that for you and it will cost less and take less time than implementing yet another solution” when you know, based on past performance, that they damn well can’t because their skills are implementation, integration, maintenance, and support — not customized new supply chain management product development, which is an area they don’t have the requisite expertise in, to start. As any good software architect knows, many applications looks easy from the outside, but turn out to be much tougher to implement once you fully understand everything they must do in order to be useful. Then there’s all the effort it takes to “optimize” the code-base so that it’s as quick, efficient, and robust as possible. And let’s not forget QA, Beta Testing, and “tweaking” time. It all adds up. There’s a reason why the majority of internal development efforts fail, having wasted months of effort, accomplishing nothing, and wasting budget that should have been applied directly to an existing solution in the first place. (And if you can’t convince them, then you have time to bring in an outside consultant [like the doctor] who can evaluate the situation objectively and determine what the best solution is for your particular situation.)

Watch Out for the Big Lie
Some software vendors will lie and say “yes, we have that capability” even if they don’t have it at all and there are no plans to incorporate it in a future release, and many software vendors will give you a resounding “yes, we do” if they only have part of the capability, or have a similar capability that they believe is “close enough” to land the sale. And then, as per the urban legend, if you insist on it, they’ll price the missing feature really high in hopes that you will decide that you can’t afford it and / or don’t need it anyway. And they’re usually right (although I have heard of a few cases where the customer actually “bought it”; needless to say, those relationships didn’t go very well).

That’s not to say that you should not trust a vendor who claims they can produce a new feature they currently don’t have. Some can, and you have to give credit to any vendor who owns up to missing functionality and treats you honestly and fairly. But you should check them out before signing on the dotted line, as there is an easy way to assess the credibility of the claim. Ask them for a history of all new features added to the product over time, by release number and release date. If releases occur regularly, with significant new features being added every three to twelve months, they probably deserve the benefit of the doubt. However, if the change history looks suspiciously void of new functionality, especially over the past year or so, or if there haven’t been many changes or bug fixes, then the product is probably “walking dead”, and being phased out, and you should be skeptical of any claims of significant enhancement.

To understand why, you have to understand software developers and the nature of software. Software developers live to develop software. In fact, given the choice between a high paying, legacy support job and a low paying, exciting new development effort, the vast majority would choose the latter. As a result, when software isn’t under active development, key developers quickly move on to other projects, and often to other companies.

Furthermore, the nature of software is that the IP exists not in the code, but in the head of the developers who wrote it. Even though C++ (C, or C#) is a standardized language, and many programmers know it, the nature of a programming language is that there are many ways, often dozens, of accomplishing the same task and many acceptable programming styles. As a result, code is nowhere as easy to read as it is to write. And since your average enterprise software product will have hundreds of thousands, or millions, of lines of code, you can imagine how difficult it is for a new developer to lean an established code-base. It’s almost impossible for your average developer to learn the software well enough to add a significant new feature without breaking some key piece of functionality.

As a result, once software developers move on, so does most of the IP, and changes inevitably slow to a crawl. “New features” are reduced to minor bug fixes, eye candy, and cosmetic “enhancements” that add no new capability and mean nothing from an ROI perspective. A slick new user interface is just another form of “big lie”, and reviewers that avoid the hard-work of understanding the true value proposition of the product simply propagate the lie. (Unfortunately, some companies will get taken in by these cosmetic changes. I know of more than one situation where the vendor simply re-did all the application pages with a new color scheme and layout, and convinced the customer that the “new” application was “much better”, even though no new functionality was added. This happens in B2C software too. Take the Microsoft Ribbon, for example. There’s no value — all it does is eat your limited screen real estate and irritate those of us who know how to use a word processor. [There’s a reason the doctor moved to Mac and decided to stick with Office 2004.])