Category Archives: rants

If You Have Two “AI” “Agents” Talking to Each Other …

… then, as Stephen Klein of Curioser.AI points out, you have a puppet show, “except instead of sock puppets, we’re using large language models and API loops”!

Just because it happens autonomously, looks social, appears to have an identify, and fakes a dialogue, it doesn’t mean there is anything more to it than the modern equivalent of a puppet show.

Gen-AI is the ultimate show and if P.T. Barnum were alive today, it would be his ultimate circus. But unlike the scarecrow, it doesn’t have a brain. It may have the ability to harness more compute power and data than any algorithm we have developed to date, but it is still dumber than a pond snail.

It has very few valid uses. I’ve discussed some of them before, but let’s make it perfectly clear what little it can actually do:

  • natural language processing — and, properly trained, it can not only equal, but even exceed the best last generation tech in semantic and sentiment processing
  • large corpus search — while it will never be 100% accurate, it can find just a few potentially relevant documents among millions with few false positives and negatives
  • large corpus summarization — again, while it will never be 100% accurate, and most good summaries won’t be top tier, it can summarize large amounts of data, and usually extract just the relevant data in response to your query
  • idea retrieval — not generation, retrieval of ideas based on a review and summarization of petabytes of data; very relevant for users dependent on LLMs who are suffering minor to severe cognitive atrophy; with proper prompting this can take the form of
    • strategy / workflow suggestion
    • devil’s advocate
  • usage and workflow prediction during application development
  • rapid PROTOTYPE generation for usability and efficacy analysis
    (not enterprise application development)

The reality is that Gen-AI

  • cannot reason,
  • is not deterministic, and
  • is essentially nothing more than a meta-prediction engine;
  • is providing ideas based on meta-pattern identification,
  • is predicting based on a layered statistical model beyond ANY human understanding, and
  • generates code riddled with security issues and possibly even boundary errors;
  • and let’s not ignore the fact that hallucinations are a core function that CANNOT be trained out .

This means that often the only way to succeed with Gen-AI is to more-or-less abandon Gen-AI LLMs in production applications except as Natural Language Parsers (as they are easier to train to accuracy levels beyond last generation semantic parsers which could take months to train to high effectiveness — and I know this from personal experience), and revert back to the AI tech that was just reaching maturity and industrialization readiness that I was writing about in the late 2010s. The reality is that, if you are willing to use some elbow grease and put the hours in, you can create spectacular applications with last-generation tech, and then use Gen-AI as a natural language interface layer to simplify utilization, integration, and complex workflows. If you are willing to create the right guardrails, where the Gen-AI LLM can only trigger specific application services with specific data in specific contexts, with HUMAN approval, then you can use it responsibly. Otherwise, it’s a crapshoot as to the results you’ll get.

For example, you should never use it for negotiation, which can be as much as reading the other person, as this is a very risky application as the number of soft-based data points you need for a decent prediction typically far outnumbers what you have available … even for public figures where you believe you have lots and lots of data on them available to judge their reactions. But hey, if you want to lose your lunch money, and possibly your entire bank account, go ahead and let it act as your buyer (but if it can lose hundreds powering a vending machine, imagine how much it can lose on a seven to nine figure category).

Even though plenty of vendors will provide some very convincing demos that seem to indicate Gen-AI LLMs can do otherwise, don’t fall for the tricks. During the demo, The Wizard of Oz is hiding behind the curtain. The not-so-great thing about LLMs is that, for a very specific set of tasks/situations, they can be overtrained on a very specific corpus to over-perform against those tasks and greatly increase the chances that any demo they deliver to you works fantastically well.

However, what this also means, is that you definitely do not want to use the Gen-AI LLM for tasks that are quite distinct and significantly different than the tasks/situations the Gen-AI LLM was over-trained for as the Gen-AI LLM is going to perform quite poorly at best, and possibly quite disastrously at worst. The reality is that once the puppeteer is no longer pulling the strings, all bets as to efficiency and effectiveness are off.

The Gen-AI ringmasters are employing the same philosophy and same techniques that made some of the early spend auto-classification providers “leaders” with unheard of success rates compared to when the average organization employed similar auto-classification tech and got dismal results. (Because they just didn’t know what “AI” actually stood for!)

Don’t be fooled by the ringmasters. If you want results, lie its AI and buy solutions that work.

It’s Not Outcomes. It’s Capability.

And that’s why outcomes is a dirty word! (Part I and Part II)

More specifically, it’s about capability, knowledge, the ability to be self-sufficient, and continual improvement.

Our rant focussed on the fact that the entire point of “outcome”-based pricing was to not only lure you away from more affordable products and services (especially if you were willing to do just a little bit more yourself), but take away your self-sufficiency, capability, and even knowledge and ensure your entire existence slowly became 100% dependent on the vendor for key processes. That you’d have no choice but to keep using them because you lost the capability to take the function back in-house. That you’d be the next mark in the grift that keeps on taking.

A big problem with “outcomes”, and another reason that it is a dirty word, is that it’s always focussed on “metrics” that have an impact on “the bottom line” today in a manner that the C-Suite can see on the balance sheet. Since the point of a business is to make profit, all of the “outcome”-pricing vendors argue that it’s the right approach.

While you should get “results”, that’s not the only thing you should be measuring, and it should not be the focus of your measurements. Because when you focus only on “results”, the focus is whatever gets you the best results, and, more exactly, what gets you the best results TODAY. That means you will make decisions that will jeopardize the potential for mid, and definitely long, term results in exchange for better results today that will please the client, your boss, the C-Suite, and/or the shareholders.

A great example of the danger of “outcome”-focus is classic sourcing — and the introduction of e-auctions (which are surging again because people forget the long-term impacts of auction over-use) that kicked our space off!

When awards are reduced to lowest price, and the volumes are large enough that a few contracts can sustain a struggling supplier, especially in tough economic times, suppliers will often sacrifice almost all of their margin just to get an award. This results in a great, immediate, win for the buyer, who can show a huge savings on the balance sheet, but it’s actually a huge risk. If the supplier sacrifices too much margin and costs rise too quickly, their viability is at risk. If they unexpectedly go out of business, the buyer has to find new supply quickly, and if the market becomes tight, this could skyrocket costs or even result in costly stock-outs or, even worse, production line shutdowns. The savings not only disappear over night, but costs increase. And even if the supplier doesn’t go bankrupt, when you go back to market, after a few years, if inflation was low, you might save 1% to 2%, but typically the best case scenario is you find someone who can match the price. However, what typically happens is that the price increases, sometimes by a lot! Why? Because the focus was on getting the best price now, versus coming up with a plan to ensure prices, or at least production costs, continued to decrease over time. Instead of looking for a supplier who would continually invest in better technology, renewable materials and energy, process improvement, etc. to keep costs down, you look for a supplier who’ll cut every corner they can to get a good price now. If you do a strategic engagement and find the first type of supplier, and enter into a long term contract where they know they can continue to invest in improvement, they’ll likely come back with a solution, and a contract, that guarantees a continual cost decrease year-over-year. This would actually benefit you more because not only you would you be able to claim an “outcome” every single year, but you know you have a supplier you can count on to deliver! (And you won’t have to explain the cost increase next time you go to market.)

In order to be a successful business, you don’t have to just profit this year, but you have to profit next year, and the year after that, and the year after that, and so on.

What this really means is that you need to be:

  • instituting processes that will allow you to not only be more efficient, but get more efficient (with experience) over time,
  • implementing supporting technologies that help you continually increase efficiency, including automation solutions that requires less and less exception management
  • increasing your knowledge and capability, so you can always make the best decisions, use the best solutions, and know when a third party can be more efficient or more cost effective (because it’s either a part-time position that’s not worth the hire internally or a function that’s not core to your business and you’d rather it be managed externally until such time as it makes sense to reclaim the function)
  • identifying metrics that focus on capturing process improvement, increasing capabilities, capturing knowledge (for future generations of HUMAN employees), and that result in improvement year-over-year

and NOT focussing on destructive one-time outcomes (that will hurt you later, and possibly a lot more than you realize).

“Outcomes” is Just Code For …

You’re Getting Ripped Off.

But let’s back up.

THE REVELATOR recently explained Why He’s Done Tracking Gartner (spoiler: they are simply not designed to solve the problem of implementation success in the AI era), and in the post he made two key observations:

1) “Subscription revenue continues regardless of outcome. Predictions expire and are replaced. There are no consequences for failure.”

In other words, they are dangling outcomes, but not doing anything to ensure you get them, and because they are never to blame, the subscription revenue continues. However, it’s only fair to point out that this is NOT unique to Gartner! It’s the Big Analyst Firm Model. It’s why the doctor doesn’t work for Big Analyst Firms (because they refuse to update their methodologies which are decades out of date and hinder more than they help), and why I worked for Spend Matters for years until the buyout (by a PE firm that, frankly, almost destroyed it as they completely stopped all innovation) and why the doctor deeply respects boutique firms like HFS Research because they keep trying to modernize their offerings to provide real value and guide clients to real results.

2) “What scales in this industry is engagement—not outcomes.”

And this is dead-on. The rhetoric is being thrown around by way too many services(-adjacent) firms (who want to charge based on it) and software firms (who won’t charge based on it). And all of it is usually to mislead you on what you should be getting and what you should be paying!

Here’s the reality you’re not being told when they want to, or refuse to, price on outcomes.

1) If a services firm wants to charge based only outcomes, it’s because it expects to make way more money that way. It’s common in audit recovery, contract re-negotiation, and SaaS consolidation because these expert firms know just how much you are overpaying if you’ve never done these efforts before. They also know that they can use cheap software and benchmarks and experience to quickly find the savings you can’t, and make big bucks off of you by charging on outcomes (and not effort and/or software).

2) If a services-as-software firm wants to charge based on outcomes, especially an “Agentic AI” powered one, it’s because their true costs are higher than they let on (due to hefty Gen-AI compute costs) and they aren’t viable offering a classic subscription-based service model.

3) If a software firm refuses to price in (a hybrid cost model based on) outcomes, it’s because they know you won’t get those outcomes unless they (as a firm) put in a lot of work training, guiding, and helping you achieve those outcomes. When their model is “install and backhaul” (out of there) until renewal time (because if they don’t hit their unattainable PE-defined sales numbers, they will be told to hit the road, so they have to spend all their time on sales).

The reality is “outcome-based pricing” only encourages success when done right — and done right is done in a manner that encourages, as THE REVELATOR notes, engagement-focussed.

That’s why, in our post on why you should STOP PAYING PROCURETECH/FINTECH ADVISORIES A DOLLAR JUST TO LOSE THREE DOLLARS!, we told you this is how you should negotiate, and pay for, software and services when “outcomes” are involved:

1. For software, you will pay a base annual fee for the platform that will cover 150% of their base hosting costs, so they won’t lose, and then a percentage of transactions, identified savings through sourcing events, contract value, etc. where the percentage is calculated such that if you save 100% of their promised savings, they will make 50% more than what you would pay on a fixed cost after negotiation -— if they are so confident in their claims, this should be a no-brainer for them. (But if they won’t agree to this, it should tell you what ROI you can actually expect!)

2. For GPO agreements, you will pay a fixed amount on each transaction, calculated based upon the expected savings before you sign the contract, and if they can deliver the savings, you will definitely be using them regularly —- and, as with the Tech Provider — you will calculate this so that they win bigger than if you pay them a fixed cost IF they generate a return for you!

3. For services (outsourcing), you will pay a fixed rate per hour that is enough to cover the assigned personnel cost (their salary plus 30% overhead), and any compensation beyond that will be dependent on the department delivering an ROI beyond a certain amount (which is the amount required to cover the basic fee you are paying them); and again, you’ll fix the compensation such that if they deliver 100% or more of what they promise, they will win big too. (And if they deliver less, while their costs will be covered, their profit will be next to 0.)

There are MANY reasons you are NOT ready for AI!

A few weeks ago, we told you that if you think you’re ready for AI, you’re not ready for AI because, even though the vast majority of you are chasing AI, only a minority of you are ready to even investigate it. And we mean investigate, not use. That depends on whether or not there are any relevant AI solutions for you needs — and despite the repeated BS claims by the big AI vendors, there may not yet be any!

And it’s not just because you haven’t

  • admitted you’re only chasing AI because of FOMO and FUD
  • assessed where you are
  • realized you are generations of tech behind
  • determined you just don’t have the right resources

But it goes beyond that.

In order to have any hope of succeeding with AI:

You need great data and great Master Data Management
… but you don’t even know where your data is! You have no governance policies, no management processes to ensure data is kept up to date (or even backed up unless you have already suffered a data loss and determined losing that specific data would be disastrous), and no clue about what that entails. And even if you realize that you need (master) data management, you won’t get the C-Suite to sign off on it, even if you call it E-MDMA and tell them they’re getting free samples!
You need a good IT infrastructure, with context-based integration and workflow capability
… but you have no central strategy for data integration, system orchestration, or enterprise workflows, and your IT infrastructure is whatever cloud your ERP runs on. AI, especially Gen-AI, requires massive data and massive compute and, guess what, that requires massively powerful, solid, infrastructure — and yours is probably held together with spit, glue, and duct tape!
You need an in-depth understanding of not only the problem you want to solve, but what AI algorithm will actually work reliably and with measurable confidence
… but guess what? In order to properly evaluate AI, you need an advanced understanding of the technology, which usually requires an advanced, graduate level, understanding of the underlying mathematics as well as deep understanding of the problem and how to mathematically model it.
You need a strong technical quotient (TQ) to implement, train, and verify those AI algorithms
… and that’s more than just a single expert who can evaluate, but a strong bench of architects and developers to make it work — you can’t rely solely on the vendor as they can go away, their bench can leave, or they can get pressured by their investors to just sell, sell, sell (and pretend you don’t exist once they get the cheque) and that leaves you to your own skillsets.
You need domain experts on hand to verify the results
… and this goes double for critical results. If you are using an augmented intelligence to help with sourcing, market analysis, strategy recommendations, etc. you can’t let an agentic system execute on a computation without verifying it. No system ever has all the data, no system ever knows all of the options, and no system has the soft information (and how you might be able to work a sales rep to your advantage). And if someone messed up the data, considering just one wrong number can entirely throw off a hundred thousand variable model, you’re in deep doo-doo if the system executes an order without your verification.
You need to redesign your processes to optimally take advantage of AI
… because your processes come from the time before office machines existed, so obviously they weren’t designed for modern technology. And while traditional workflow / RPA can easily automate what you have (even though it shouldn’t), since AI requires good data, good structure, properly designed models, etc. — it’s not going to work with whatever Guilded Age process you’re using now.

And so on. The reality is, despite what all the big vendors, big consultancies, and big analyst firms tell you — you’re just not ready for AI. (And definitely NOT ready for big bang projects that will end in big busts!) It’s just the latest silicon snake oil panecea — like all purpose predictive analytics, the fluffy magic cloud, SaaS, and the World Wide Web and every other panacea that has come before. (Just remember the last time silicon snake oil was hyped this much, it resulted in the dot com bust!)