It’s been 17 years since SI published its first major series on generic insider tips back in 2009 where we gave you a lot of advice that more-or-less still stands today if you want to safely acquire software. In our preamble, we overviewed what those 11 pieces of advice were then, and summarized the 6 major difference that affect how you apply that advice today so you can continue to make the right decisions when acquiring software in the age of AI Hype and exaggerated I2O claims. In Part II we addressed the first two pieces of advice and how they have evolved over the years. Today, we continue.

Chuck the checklist

Better yet, go all Chucky on it. The problem with every checklist every created is that its predominantly features, not functions, and leads you down the wrong path. Checklists should be what to look for in a vendor, what to ensure in a contract, what steps you can’t bypass or forget. Not for a list of features, which never compare one to one, when what you really need is functions.

Furthermore, no piece of software can do everything, and checklists always mix must have with should have with nice to have with that would be cool (but totally useless when you think about it), which is rarely done by anyone when you ask them what features they need in a solution). Plus, we know you’re going to jump start your list using BS Gen AI which is going to regurgitate features common to multiple Free RFPs, Trade Rags and Big Analyst Firm lists, which aren’t very good (and that’s why SI has warned you about all of these before).

What you need is a description of the functions that are required to satisfy your process requirements (you mapped those first, right?), user case descriptions that describe how the software will be used and/or needs to be employed, and what the end results need to be. Not a checklist.

Wait for the blush to leave the rose

Although the testimonials and references, from clients who have recently implemented the product, that the vendor brings you, will be sincere, they will also usually be useless. Why?

1. New customers are highly motivated to say the software is great.
   Saying otherwise is paramount to saying they screwed up, and that wouldn’t go over vey well with their management. So even if the software wasn’t working as they expected, they wouldn’t admit to it.
2. Almost all solutions are better than no solution at all.
   So everything looks great when you’ve never had, or even seen, anything else.
3. New customers haven’t experienced what happens when something major goes wrong.
   Something will. It might be their fault, the vendor’s fault, or the fault of a third party like the implementer, the cloud provider, or the AI provider. It’s how the vendor steps up and (helps to) fix(es) it that counts. A reference that will tell you something went horribly wrong but the vendor immediately jumped in, worked evenings and weekends, and made it right is worth way more than a reference who just says its great. Because such a provider would have also done a post mortem, figured out what went wrong, and what they can do to minimize the chance that it happens again (be it better customer training, specific checks in their vendor oversight, real-time monitoring of third party integrations, etc.).

You might have to go out and track down those customers (and references) yourself, but do that if you really want to evaluate the vendor properly.