Sheesh is Right! The average analyst doesn’t have a clue how to properly evaluate a technology offering. But what should you expect considering most of these individuals come from a liberal arts background and think “C” is for Cookie? And if you don’t believe me, maybe you should read those tragic quadrants, graves, and benchmarking futiles more carefully … because the reality is that an evaluation scheme based on chicken-with-its-head-cut-off bingo would likely be just as accurate as the rankings in the average analyst report these days.
But back to the point. Recently, over on Spend Matters, Jason Busch decided to tackle the issue of product demonstrations (Part I and Part II). In his first post he said that you should be prescriptive where the demo is concerned and that, except in a few rare cases, you should focus on ease-of-use. I disagree.
In the words of commenter “Sheesh” [italics in the rest of this post], Jason’s points are true if the analyst knows what to ask. If all you do is focus on “usability”, you’ll end up choosing the solution with the most flash. Problem is, there’s always a trade-off between flash and substance. A software company only has so many development hours before it runs out of cash and has to sell something. This means that the more hours it spends on creating a razzle-dazzle UI, the fewer hours it has to spend on actually building useful functionality. Remember, we’re talking about enterprise software, not end user entertainment.
Jason’s followed with a second post where he proposes a series of demonstrations based on specifically prescribed scenarios.
The problem is that, as Sheesh says, you’re not a tech expert and you shouldn’t [even] attend the demo unless you bring one with you. The “business scenario” approach assumes that the ability to address “business scenarios” is a useful measure of anything. Typically “business scenarios” are crafted from melted-down aggregations of vendor marketing messages and feature lists, combined with some business problem that the vendor has already got a pat answer for unless he’s a complete idiot. If he’s on the ball at all, you won’t discover anything interesting; or if you do, it will be something on the order of Debbie Wilson’s mouse clicks. ‘OMG this vendor takes seven mouse clicks and that one takes twelve!!!!!’ Not at all useful!
Sheesh continues, You have to go deeper than chin-stroking about a suite of standard reports and baked in transaction processing flows for “business scenarios.” What’s behind the curtain? What if you want to write your own report, or modify one of theirs? What if you want to change the way the transaction processing works, fundamentally? If you find a solution that’s extensible in those dimensions, then it’s tons better than any canned solution, by definition. Because you can make it do whatever you want it to do.
Furthermore, the software procurement process shouldn’t try to find a “best fit” in a shoe store that doesn’t carry your size. Rather, the software procurement process should focus on finding a solution that can be easily extended and adapted to your needs, on as many dimensions as possible. If the solution can be extended and adapted, it really doesn’t matter what it does out-of-the-box, in terms of canned reports, canned transaction flows, canned sourcing and optimization templates, or canned demos — or what its relative performance might be on business scenarios that encompass what you think you need today, but tomorrow might fail miserably at characterizing your needs. If the solution is extensible, then it can solve a universe of problems rather than a small subset of them.
As the commenter stated, there are lots of such questions that need to be asked, but they need to be asked with the aid of a technical resource.And there’s really no excuse not to have an expert technical resource on hand. Expert advice is cheap, especially compared to what you’d pay to some analyst firm, or waste on some solution that compares well on an analyst checklist, but is actually hopelessly inflexible. In the long run, it’s probably a few tenths of a percentage point of what you’ll spend on the “enterprise” solution (which won’t be a solution at all if it sits there, on the “shelf”, unused).