A recent rant on LinkedIn noted that, at DPW (and other events), there was “Tons of talk about using AI, not so much about how to procure it, scope it, contract it, measure it, commercial agreements, which suppliers are best for which usage.”

Well, there’s a simple reason for that. And it goes as follows.

It’s the tech-du-jour. More specifically, the hype-du-jour. As a result you can forget any advice on how to:

–> Procure It

No one really knows who has what, or what it actually does, what it’s really worth, how to properly cost it, how to compare offerings and offers. So how can they tell you how to procure it?

–> Scope It

According to the hype it’s your new employee that does everything for every one, despite the fact that it hallucinates more times per second than an LSD junkie does in a lifetime. Without knowing what it does or where it goes, they can’t tell you how to scope it!

–> Contract It / Commercial Agreements

Even the vendors wrapping someone else’s model don’t really know what it does, what they can guarantee, what they can’t, or who is really liable (although courts are starting to make those decisions). So no one knows how to contract it.

–> Measure It

Whereas we have tried-and-true mathematically sound measures for traditional deep neural networks where you can get accuracy ranges with confidence ranges, when it comes to LLMs, no one has a f*ck1ng clue how to measure them. Random tests by random humans judged by random people with random definitions of accuracy is not a measure. And I’d have more confidence in a decision made by a Koala. At least it’s cute!

–> Assign It

If we can’t even measure what it does, do you think we can measure how well the vendor pushing it really understands it? Definitely NOT!

And that, in a nutshell, is why there’s a lot of hot air and nothing of actual substance in all the AI discussions. Which you should avoid anyway. Because you don’t want tech with a 6% success rate [McKinsey, MIT], which is half the general success rate of new tech installations (now that tech failure rates have reached an all time of 88% [Bain]).