the doctor has to ask because when Jon The Revelator asked if you would eat a piece of fish that has been in your freezer for 10 years? 5 years? 1 year? not many of you spoke up and it seems you are quite okay with old, smelly fish, which, in this case was a metaphor for provider case studies, as this was a follow up to The Revelator‘s post that asked Should Solution Provider Case Studies Have a Best Before Date.

A question, which was in turn sparked from a comment by Duncan Jones to his preceding inquiry on what can 2005 tell us on why most AI initiatives fail in 2024, which is a question that was partially sparked off of a post the doctor himself made on how we need to hasten onshoring and nearshoring — the drivers will pound those who don’t into the ground! (Part 2).

While this sounds like a long, meandering, pointless introduction, it’s exactly the opposite. The purpose is to demonstrate that not only are many parts of Procurement and Supply Chain connected, but they are connected in complex ways that require sufficiently broad, as well as sufficiently deep, solutions that address the complexities being experienced by the organizations a vendor is trying to sell to.

Furthermore, this means that for an organization, or a consulting partner, to select the right solution, they need deep information on what the solution does, where it’s been used, and what it has been proven to do. Traditionally, this would mean that they would require product sheets and demos, customer references, and case studies to make a good decision.

However, centering in on this last requirement, not all case studies are created equal, and not all are even “case studies” at all. What once was the domain of third party analysts, consultants, and professors (who would do proper due diligence, data collection, and impartial write-ups for educational and investment purposes) has now become the domain of marketers who get happy customers, often still wearing the rose-coloured glasses that came free with the install, to tell a story that they write-up and promote using very little, and often unverified, data. Those are not useful at all. Furthermore, if you don’t know what version of the software, what stack the customer ran on, and/or, and sometimes most importantly, when the study was done (and the time period it was done over), is it even still relevant at all?

This prompted the critical question from The Revelator about whether or not studies should have a best before date. the doctor leans towards no on best before date, because just like different types of fish have a different shelf life, different case studies will have a different shelf life, but votes a most definite yes on a packaging date.

To elaborate on the comment he made when asked, the following is absolutely critical to be included in the case study:

• when the case study was written (packaging date)
• the time period it was over (processing dates)
• the precise metrics that were tracked and how they were computed (labelling compliance)
• the extent of organizational data that was used (ingredients)
  [as well as the full extent of data available (may contain)]
• the products, and versions, that were used (processing)

In other words, a feel-good story with a few random numbers is not case study! (the doctor would say any marketer trying to pass such off as one should be ashamed, but any marketer who did would obviously be without shame, so there’s really no point in saying it.) A case study has rigour in definition, methodology, data collection, and exposition and contains all the information that would be needed if a third party wanted to repeat it. (The same way a scientific study provides enough detail for an independent team to verify it.) Anything less should be considered unacceptable.

And, most importantly, since business processes, products, systems, and stacks continually change, a study (processing) date and a publication (packaging) date MUST be included so that a buyer can make an informed decision as to whether that study is still relevant to them (as they decide just how much stink they are willing to tolerate).