Category Archives: AI

You CAN NOT Safely Use LLMs for Contracts or Legal Work!

Darlene Newman recently wrote a great article that makes it abundantly clear why you CAN NOT Safely use LLMs for Contracts or any other document with any Legal implications whatsoever!

Not only can you not train out hallucinations, because they are a fundamental function of the technology, but every time the LLM touches the document, it can (and likely will) corrupt something that was already correct (and reviewed) before.

In other words, you collect all your reference documents, ask it to generate a contract that contains all of your mandatory clauses, addresses all the risks, incorporates the schedule, specifies the requirements, etc. etc. etc. and get back a 50 page document where the section, paragraph, and sentence quality ranges from masterpiece to monkey on crack. You then spend hours (to days) fixing everything and ask the LLM to simply correct spelling, grammar, and ensure key requirements are met in the new/changed sections only (giving it the original document for comparison). The LLM spits out a cleaned up copy, you review all the sections you updated, it looks good, and you send it out.

Little do you know that because you added an article in one section, shortened a sentence in another section, and improved the grammar in a third section that it decided to rewrite half those sections for you, because it decided the specific requirements you called out for the new sections weren’t addressed enough. In the process, other key requirements are dropped, risk mitigations have been written out, and the contract now heavily favours the other side when something goes wrong. Not at all what you intended, but that’s what you got because you didn’t review all 50 pages with care.

Maybe not too bad if nothing goes wrong, and maybe devastating if it does.

But nothing goes wrong in the short term, so your Legal team decides to use it to try and defend a claim against your company. This is where it goes from bad to much, much, worse. You upload the brief, you outline your counterpoints, you upload your supporting documents — including the relevant law and cases you know of, you ask it to find more law and cases relevant to your defense, and ask it to create your first response. You let it chug, go to lunch, and come back to a 60 page, 220 point response with half a dozen statues and two dozen cited cases.

You go through all the law, realize that only 8 of the statutes are (somewhat) relevant, remove the 3 that aren’t and the fake one the LLM found on the internet. Then you go through all the cases, realize only 14 are actually supporting, 7 are not relevant, and 3 were completely hallucinated and make the corrections. Mark all the paragraphs that are okay, the ones that need updates, and what updates are needed. Get sign off on what’s good, what needs updates, and push it through again. It comes back with a couple of new potential statutes, another 8 potential cases, updates to multiple paragraphs, and you review again. You find one of the statutes potentially relevant, 4 of the cases real and usable, and half of the paragraphs look good. You mark all this, make the updated correction lists, get sign-off, and send it back to the LLM. You don’t notice it also changed 5 of the paragraphs you were completely happy with, changed some quotes to non-existent quotes, and replaced an approved reference with a hallucinated one. This goes on for a few more iterations, where key clauses/references are not rechecked, and you still end up with a 70 page document with a dozen hallucinations, 3 non-existent cases, and faulty logic despite review by multiple senior partners, because no one checked what they were happy with last iteration because they expected the LLM would not change it because they explicitly told the LLM not to.

Unlike an intern, who is naturally lazy and tired of working 84 to 112 weeks for peanuts and will happily ignore anything you tell him to ignore, as well as intelligent (when he chooses to be), the dumber-than-a-doornail LLM recomputes the meaning of inputs on every request, has the same chance of messing up on every request, has the same chance of understanding the request but predicting you were being facetious and actually want it to rewrite the paragraphs chock full of hallucinations, and so on. You don’t notice, submit the brief with $1,000/hour senior partner sign off, and make a mockery of your firm with all the AI slop (as well as securing it a massive fine from a p!ssed off judge tired of AI slop).

And there’s no way to stop it. It doesn’t matter how detailed your instructions are. It doesn’t matter how much effort you go through to lock parts of the document down with automated input and output checks and re-dos when the LLM screws up. Every time the LLM touches the document, something will corrupt. The only thing that is unknown is whether or not is how detrimental the corruption is.

As per Darlene’s post,

Microsoft Research tested 19 AI models across 310 professional documents. They gave each model a document editing task, then another, then another … for 20 interactions in total. Frontier models corrupted 25% of document content by the end.

25%! That’s a lot of corruption of good content. And enough to ensure you get AI slop every time!

Like Any Tool, AI Won’t Solve Leadership Problems!

Paul Martyn is right to cringe a little every time he hears a solution provider say:

AI and automation won’t replace employees. It will free them up for more strategic work
Because there are two fundamental problems with this statement.

1. As Paul points out in his recent article, if strategic work is not already happening, that’s not a technology problem. That’s a leadership problem!

2A. You can’t drop tech in and suddenly become more efficient unless you have all the data and processes in place to support it — and it’s a money back guarantee you don’t have all of the data and processes in place to support it.

2B. Unless AI stands for Augmented Intelligence, AI will actually consume MORE of your time as you deal with the hallucinations and errors it will create on a regular basis. (Remember, only 1 in 20 organizations are seeing a return on their AI investments, and I guarantee those are the ones that either got tricked into, or simply bought, old fashioned RPA (robotic process automation) that actually works.

Don’t fall for the spin. If you want strategy

1. Make sure it’s already happening.

Maybe it’s only 10% of categories going through strategic sourcing, but you have to start somewhere. Then you can increase that percentage as you automate more tactical work.

2. Allocate time to (old-school) automation.

One at a time, pick a very time consuming process ripe for automation. Map it end to end. Redesign it for automation. Automate it. As time frees up, more time for strategy and automating more processes.

3. When the automation effort in time-consuming / painful processes that remain exceeds the expected time return over the next 12 months, look for outside help.

Not before. And that’s how you don’t fall for the spin!

AI is NOT Failing Because of a Lack of Forward Positioned Data

Lack of forward positioned data is NOT the problem.

(It is a problem, but not the biggest one!)

An AI agent making 1000X the decisions IS!

Right now, while the big AI players have achieved 80% to 90% “accuracy” on their carefully designed synthetic benchmarks, when applied to real world problems, accuracy in many domains drops to 25% (or worse, as at most 20% of code generated by an AI survives into a production application once it gets reviewed by a senior developer who finds a plethora of security issues, boundary condition errors, and code that, frankly, just doesn’t solve the problem at all).

THIS MEANS THAT THE AI IS MAKING 750X MORE WRONG DECISIONS THAN THE HUMAN!

That’s a LOT of mistakes.

Meanwhile, give an expert human

a) always available forward positioned data and Augmented Intelligence applications to process it (so all the data the expert human needs to make the decision is at her fingertips)

b) A-RPA (Automation) software that is best-of-breed and capable of immediately executing any decision the human makes (possibly using the forward positioned data and appropriate augmented intelligence outputs)

And that human will make 100X the decisions she’s making now, and get 95% of them correct. So if you hire 10 humans, you will have 25X less errors (5% vs 75%).

When you consider ten humans will cost considerably less than AI when you consider the rapidly rising token costs and the costs of dealing with the 25X increase in errors the AI will bring, Augmented Intelligence powered by Forward Deployed Data and a small team of humans will be a LOT more productive than you ever thought possible.

If You’re Spending 250K Annually Per Engineer On AI …

Then not only are you contributing to planetary destruction (through the generation of between 1.32 tons (high end models, 1 joule per token) and 84 tons (low end models, 2 joules per token) of CO2 to power those data centres, which is about 0.2 to 12.7 times the average individual carbon footprint, with an expectation of 7 to 11 tons (Source), and the utilization of 300,000 gallons to 5,000,000 gallons of water a day to keep those servers cool, or a town’s worth of water every day!

BUT YOU ARE NEEDLESSLY WASTING 400K+ A YEAR

1. Less than 20% of AI generated code survives unscathed in a commercial enterprise software product once senior developers weed out all the security errors, boundary condition errors, and generated code that doesn’t even solve the problem. So, that’s 200K of 250K down the drain as only 20% of output is usable.

2. Having to fix AI generated slop will consume 80% of a good senior developer’s time — a developer you should also be paying 250K a year.

End result, you’ll losing 200K + 200K per developer you force AI coding tools upon!

But hey, it’s your money. If you want to p!ss it away so NVIDEA’s CEO can get richer selling more CPUs we don’t need, that’ up to you!

The linked article contains some metrics, but here are a few others.

  • token prices vary widely, from an average of around 50c/M tokens on the smallest, cheaper models to $75/M tokens (or higher) for higher end “workhorse” models
  • energy processing requirements per token are estimated to be between 1 joule and 2 joules
  • you can buy 14.3 Trillion tokens at the median of around $17.5/M tokens (and 35 times that at the lower end)
  • processing 14.3 T tokens will take about 4000 kwH @ 1 joule/token
  • on an average NA grid, expect to produce 500 to 600 g of Co2 per kWh (since most of our grids are still dirty)

The Bullshit Filter for Enterprise AI Startups consists of 12 Questions!

Not 11!

Backing up, earlier this year Jason Busch published his 11-Question Bullshit Filter for Enterprise AI startups. This was, and is, needed because the vast majority of Enterprise AI startups are bullshit (especially in FinTech and Procurement) and the sooner you figure that out, the better.

I was hoping that, by now, the AI startup scene would start crashing due to over investment, lack of returns (only 6% of AI implementations have generated an ROI), and, generally, lack of usefulness. (AI can serve up your data, show you complexity and even help with automating some tasks, but it can’t make decisions and, due to lack of anything close to intelligence, can’t even do basic tasks without your oversight.) But, even worse, these solutions are still multiplying like Fibonacci’s rabbits and their claims are getting more outlandish by the day. (How many times do we have to tell you AI Employees Aren’t Real, you should NOT engage any vendor selling “AI Employees”, because you definitely do NOT want AI Employees.)

So, since they are flooding our space with BS marketing and making ridiculous claims about what their useless apps can do, it’s more critical than ever that you be able to suss out the BS claims from the non-BS claims. (Hint: 95% are BS claims, so it wont’ be easy!)

We’ll start with Jason’s 11 filters, which we’ll number 12 down to 2, because he left out the most important filter, and the one that, if it fails, allows you to skip the next 11.

Filter 12: Founder DNA
Can they build and sell? Likely not. Chances are, if they’ve cut through the noise and reached you, they can only sell. And if you did find a builder, they won’t survive long enough to support you if they can’t sell.

Filter 11: Motivation
Is failure unacceptable? (Every startup team will say it is, but unless every founder has a reason they simply cannot accept failure, when the going gets tough … the tough get going … and quit.)

Filter 10: Interface
Is it designed for those who will ACTUALLY be using it?

Filter 09: Categorization
Does the product actually do something new? Is there a strong reason for the market to adopt it?

Filter 08: “Found Money”
Are there instant benefits that sell themselves on the first demo.

Filter 07: Displacement
Does the product workaround or replace a solution that buyers hate?

Filter 06: Functional Bonds
Does the solution cross boundaries that increase value beyond peers?

Filter 05: Data Delta
Is there a “data” strategy to exploit the delta between what humans can easily consume and what AI can leverage (and summarize into something useful for human data ingestion)?

Filter 04: “Messy Middle”
Can the solution ingest external “dark data” and turn it into actionable insights without requiring a(n extensive) manual data-cleansing project? (Quick review and correction is okay.)

Filter 03: Connect the Dots
Does the app bridge the gap between “Watercooler Data” and “System of Record Data” (ERP/PO) to explain the why behind an analysis or recommendation?

Filter 02: “Show Your Work” Audit
Can the user drill into any output, see each and every step the AI took, drill down to the source data, and verify that everything is correct, accurate, and no data was changed?

These are all great filters, but there’s no point going through them if you don’t check the most important filter first:

Filter 01: Is it LLM-based?
If yes, move along. Don’t waste any time.

Most of the failures in the age of AI come from Gen-AI LLMs that promise the world and don’t even deliver a pile of dirt. That hallucinate on every other query. That burn up thousands of dollars of tokens to deliver less than fresh MBA interns with no real world experience and no clue to share on their first day no less.

Even worse, the majority of these players are simply wrapping third party LLMS in the creation of their “solution”. That’s not a solution at all. That’s an unmitigated disaster waiting to happen!

In the rare case an LLM actually offers a partial solution, it is best to go straight to one of the major providers. That way, you know who’s responsible when something goes wrong and don’t have to worry about providers playing the blame game and pointing fingers at each other.