Breaking Down The Barriers: Data Integration/Management/Analytics

We’re continuing our foray into the top barriers to success that we outlined in our top barriers post that chronicles the barriers that keep coming up over and over again in every Procurement survey in our effort to ensure that you don’t have to read another state of procurement study for the next 5 years. Today we turn to one of the biggest barriers in any organization, and especially if they don’t realize it. Data^*

A Brief History …

Unlike all of the other barriers on this list, this is the one barrier that is relatively new. Up until the introduction of computers in the average business, the only real data that was maintained was the accounting data. Orders, and costs, sales, and revenues. That’s it. Other than that, the
“data” of the business was its contracts (on paper), its product designs (on paper), and its processes (on paper).

But with the introduction of modern computing into the average business in the 1980s, a lot of this data became computerized. Plus, the amount of data collected, and maintained, started to increase significantly over time. In addition to costs you could maintain quotes. In addition to revenues, you could maintain sales volume by product and location. In addition to current designs, you could maintain historical designs and alternate designs being considered.

You could start to collect and maintain market data on commodity costs and availability. You could collect and maintain data on currencies and exchanges and markets. You could build your own database of global logistics options instead of relying on suppliers and trading partners to select local carriers for you. And so on.

As time went on the average business unit went from having essentially no data when computers were introduced in the 80s to having lots of data as the internet took over in the late 90s to having a combinatorial explosion of data by the 10s when there was a SaaS app for everything!

The Problem

As data exploded, a number of problems arose.

How do you manage and maintain the data?
How do you analyze the data?
How do you integrate data from other departments? partners? the internet?
What data do you need for a meaningful analysis to make meaningful decisions?
What data do you need to send to other departments? partners?

And the reality is that as the data exploded, the need to understand the data exploded, and the need to integrate the data exploded

training and technical competence fell behind with each advancement
data formats and models exploded (as SaaS apps exploded)
internal and external data needs exploded, but the ways to easily get and send the right data at the right time shrank

The Necessary Realization

The data explosion that was supposed to be the blessing has instead become the curse.

Every system uses its own storage formats behind it’s own data models — so you need to obtain custom middleware / iPaaS to integrate data between systems, and often services to link in all the systems not supported out of the box
Back office analytics software has not kept up — most of the big name software is ROLAP, Relational Online Analytical Processing — where you are limited to drilling down in pre-defined cubes (and it’s not easy to create new cubes to power new reports)
Analytics capability has not kept up — the average employee doesn’t know what can be done (and what techniques to use to do it)
AI is more of a curse than a blessing — sure it can uncover interesting trends, outliers, deviations, etc. — but it doesn’t really understand the data, whether its prediction on what should be done is accurate (as AI is NOT intelligent), or how to guide you on what additional analysis to do to figure out what to make of its “discoveries”

This means that to make progress you need to understand:

what modern analytics is (and what AI is not)
what systems support it
what systems you need for integration and transformation of data
(even though most analytics can do all the necessary data transformations, some systems still require proprietary integrations to get that data)

And there is very little education out there on all of this. (A lot of marketing, but not a lot of real education.)

The Technological Requirements

The technological requirements are considerable and require supply chain aware sourcing and sourcing aware supply chain and expertise from source to sink and back again on both sides.

A continuing reminder that if you want guidance in the short term, hope that your favourite provider reaches out to Bob Ferrari of Supply Chain Matters or the doctor and enables us to focus on writing the series (or in-depth e-book) explaining what modern Procurement and Supply Chain Tech needs to look like (and how it needs to be implemented) to address the challenges, reduce the risks, and address the priorities versus just dripping out tidbits as free time permits.

^* Remember every office worker’s favourite song!

Well …
I have a little data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

Oh, data, data, data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

It has nonstandard fields
The records short and lank
When I try to read it
The blocks all come back blank

I have a little data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

My data is so ancient
Drive sectors start to rot
I try to read my data
The effort comes to naught