Category Archives: MDM

Just like there was no Æther, there’s no data fabric either!

In a recent LinkedIn posting just before the holidays, THE REVELATOR asked a very important question. A question that may have gone overlooked given that many people are busy trying to get their work done before the holidays so they can get a few days off. And a question that must NOT be forgotten.

1. How does the old technology phrase “garbage-in, garbage-out” apply to Gartner’s Data Fabric post?

Data files. Databases. Data stores. Data warehouses. Data lakes. Data Lakehouses. And now … the data fabric … which is, when all is said and done, just another bullsh!t organizational data scheme designed to distract you from the fact that your data is dirty, that data storage providers don’t know what to do about it, but these data storage providers still need to sell you on something new to maintain their revenue streams.

You see, the great thing about today’s SaaS middleware enabled apps is that they don’t care where the data is, what organizational structure the data is stored in, etc. As long as the data has a descriptor that says “this field, which is in this format, in this db stores X” (where X describes the data) and an access key, the SaaS middleware can suck the data in, convert that data into the format it needs, and work with that data.

However, now that we are in the age of “AI”, the most important thing has become good, clean, data. However, just “weaving” your bad data together doesn’t solve anything. In fact, with today’s technology, it just makes things MANY times worse. We are now at garbage in, hazardous waste out!

Unfortunately there’s nothing we can do if the AI zealots are now adding hallucinogenics to their kool-aid, because it sounds like they are trying to bring back the magical medeival Æther … *groan*

THE REVELATOR then went on to ask …

2. Why does Gartner confuse more than inform and enlighten?

At the end of the day, you have a better chance of appearing as an enlightened Guru to someone who is lost and confused than to someone who is clear headed and confident in one’s direction!

Like the other big analyst firms, they profit off of being the Gurus the executives turn to when they can’t make sense of the hogwash filled marketing madness they are inundated with every day!

More specifically, their sales people need to say: “Our senior analyst has all of the answers … and they can be yours at the low, low introductory price of only 9,999,99 USD a day*.” So they don’t really care about whether or not they are confusing more than enlightening, as long as the sales are coming in. (In fact, they aren’t even looking to see how they are doing as long as the money keeps rolling in …

* one day only, after that, full rate of 29,999.99 a day applies …

But the questions didn’t stop there. The next question was:

3. Why are Data Problems Solved Downstream?

The answer to this is not as easy or straightforward, but when you consider that:

  1. it’s hardwork to solve the problems at the source and
  2. most of these analyst firms are staffed with analysts with little fundamental understanding of technology or the domains they are analyzing the technology for, don’t want to admit it, and are happy to take guidance from the vendors cutting them the biggest cheques and spending the most time “educating” them on the paradigm the vendor wants to see …

What should one expect.

Case in point. Did IDC just happen to come up with a “Worldwide SaaS and Cloud-Enabled Spend Orchestration Map” on its own at the same time a whole bunch of these solutions hit mainstream? (Especially when it takes person years of research and development to design a new map and analyze vendors, at least if you want to try and get it right.) Especially when they don’t have enough senior analyst talent to adequately cover core S2P?

Another case in point. Did Gartner merge it’s P2P into a S2P map because it honestly believes the entire market is heading there (FYI it’s not, look at the Mega Map), or because it doesn’t have enough analyst talent left to attempt to cover the market fragmented?

At the end of the day, it takes many years and many degrees to get a fundamental understanding of modern technology (which all runs on math, by the way) and many more years to get expertise in a business domain … so what can you honestly expect of kids straight out of school who make up significant portions of analyst teams???

Which led to the next question.

4. Can innovation co-exist with exclusivity?

Innovation happens, but then big stalwarts in the space scoop it up to try and remain competitive enough to keep their current customers locked in, a vacuum is created, and the cycle starts anew.

Until Trump dismantles them entirely, the US, like most of the pseudo-free first world, has enough anti-monopoly laws to ensure the cycle continues.

So yes, innovation can coexist with exclusivity, it just takes decades to realize what could happen in less than one decade as a result of having to start over so many times.

Finally, this led to the final question:

5. Does the VC investment model of: for every ten investments, seven fail, two are mediocre, and one “hits pay dirt” have anything to do with the 80%+ technology project failure rate?

It most certainly does! The fact that VCs are happy for seven investments to fail entirely (and then just move the good people to other investments if those people want to keep working) doesn’t help the project failure rate … especially since so many companies don’t survive long enough to master models that will lead to success, instead of failure, 80%+ of the time or to take the time to gauge, plan, and do implementations properly (because, if they don’t sell the next deal within a quarter, the investors will drop them faster than a hot potato).

Enterprises have a Data Problem. And they will until they accept they need to do E-MDM, and it will cost them!

This originally published on April (29) 2024.  It is being reposted because MDM is becoming more essential by the day, especially since AI doesn’t work without good, clean, data.

insideBIGDATA recently published an article on The Impact of Data Analytics Integration Mismatch on Business Technology Advancements which did a rather good job on highlighting all of the problems with bad integrations (which happen every day [and just result in you contributing to the half a TRILLION dollars that will be wasted on SaaS Spend this year and the one TRILLION that will be wasted on IT Services]), and an okay job of advising you how to prevent them. But the problem is much larger than the article lets on, and we need to discuss that.

But first, let’s summarize the major impacts outlined in the article (which you should click to and read before continuing on in this article):

  • Higher Operational Expenses
  • Poor Business Outcomes
  • Delayed Decision Making
  • Competitive Disadvantages
  • Missed Business Opportunities

And then add the following critical impacts (which is not a complete list by any stretch of the imagination) when your supplier, product, and supply chain data isn’t up to snuff:

  • Fines for failing to comply with filings and appropriate trade restrictions
  • Product seizures when products violate certain regulations (like ROHS, WEEE, etc.)
  • Lost Funds and Liabilities when incomplete/compromised data results in payments to the wrong/fraudulent entities
  • Massive disruption risks when you don’t get notifications of major supply chain incidents when the right locations and suppliers are not being monitored (multiple tiers down in your supply chain)
  • Massive lawsuits when data isn’t properly encrypted and secured and personal data gets compromised in a cyberattack

You need good data. You need secure data. You need actionable data. And you won’t have any of that without the right integration.

The article says to ensure good integration you should:

  • mitigate low-quality data before integration (since cleansing and enrichment might not even be possible)
  • adopt uniformity and standardized data formats and structures across systems
  • phase out outdated technology

which is all fine and dandy, but misses the core of the problem:

Data is bad (often very, very bad), because the organizations don’t have an enterprise data management strategy. That’s the first step. Furthermore this E-MDM strategy needs to define:

  1. the master schema with all of the core data objects (records) that need to be shared organizational wide
  2. the common data format (for ids, names, keys, etc.) (that every system will need to map to)
  3. the master data encoding standard

With a properly defined schema, there is less of a need to adopt uniformity across data formats and structures across the enterprise systems (which will not always be possible if an organization needs to maintain outdated technology either because a former manager entered into a 10 year agreement just to be rid of the problem or it would be too expensive to migrate to another system at the present time) or to phase out outdated technology (which, if it’s the ERP or AP, will likely not be possible) since the organization just needs to ensure that all data exchanges are in the common data format and use the master data encoding standard.

Moreover, once you have the E-MDM strategy, it’s easy to flush out the HR-MDM, Supplier/SupplyChain-MDM, and Finance-MDM strategies and get them right.

As THE PROPHET has said, data will be your best friend in procurement and supply chain in 2024 if you give it a chance.

Or, you can cover your eyes and ears and sing the same old tune that you’ve been singing since your organization acquired its first computer and built it’s first “database”:

Well …
I have a little data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

Oh, data, data, data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

It has nonstandard fields
The records short and lank
When I try to read it
The blocks all come back blank

I have a little data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

My data is so ancient
Drive sectors start to rot
I try to read my data
The effort comes to naught

Oh, data, data, data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

Enterprises have a Data Problem. And they will until they accept they need to do E-MDM, and it will cost them!

insideBIGDATA recently published an article on The Impact of Data Analytics Integration Mismatch on Business Technology Advancements which did a rather good job on highlighting all of the problems with bad integrations (which happen every day [and just result in you contributing to the half a TRILLION dollars that will be wasted on SaaS Spend this year and the one TRILLION that will be wasted on IT Services]), and an okay job of advising you how to prevent them. But the problem is much larger than the article lets on, and we need to discuss that.

But first, let’s summarize the major impacts outlined in the article (which you should click to and read before continuing on in this article):

  • Higher Operational Expenses
  • Poor Business Outcomes
  • Delayed Decision Making
  • Competitive Disadvantages
  • Missed Business Opportunities

And then add the following critical impacts (which is not a complete list by any stretch of the imagination) when your supplier, product, and supply chain data isn’t up to snuff:

  • Fines for failing to comply with filings and appropriate trade restrictions
  • Product seizures when products violate certain regulations (like ROHS, WEEE, etc.)
  • Lost Funds and Liabilities when incomplete/compromised data results in payments to the wrong/fraudulent entities
  • Massive disruption risks when you don’t get notifications of major supply chain incidents when the right locations and suppliers are not being monitored (multiple tiers down in your supply chain)
  • Massive lawsuits when data isn’t properly encrypted and secured and personal data gets compromised in a cyberattack

You need good data. You need secure data. You need actionable data. And you won’t have any of that without the right integration.

The article says to ensure good integration you should:

  • mitigate low-quality data before integration (since cleansing and enrichment might not even be possible)
  • adopt uniformity and standardized data formats and structures across systems
  • phase out outdated technology

which is all fine and dandy, but misses the core of the problem:

Data is bad (often very, very bad), because the organizations don’t have an enterprise data management strategy. That’s the first step. Furthermore this E-MDM strategy needs to define:

  1. the master schema with all of the core data objects (records) that need to be shared organizational wide
  2. the common data format (for ids, names, keys, etc.) (that every system will need to map to)
  3. the master data encoding standard

With a properly defined schema, there is less of a need to adopt uniformity across data formats and structures across the enterprise systems (which will not always be possible if an organization needs to maintain outdated technology either because a former manager entered into a 10 year agreement just to be rid of the problem or it would be too expensive to migrate to another system at the present time) or to phase out outdated technology (which, if it’s the ERP or AP, will likely not be possible) since the organization just needs to ensure that all data exchanges are in the common data format and use the master data encoding standard.

Moreover, once you have the E-MDM strategy, it’s easy to flush out the HR-MDM, Supplier/SupplyChain-MDM, and Finance-MDM strategies and get them right.

As THE PROPHET has said, data will be your best friend in procurement and supply chain in 2024 if you give it a chance.

Or, you can cover your eyes and ears and sing the same old tune that you’ve been singing since your organization acquired its first computer and built it’s first “database”:

Well …
I have a little data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

Oh, data, data, data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

It has nonstandard fields
The records short and lank
When I try to read it
The blocks all come back blank

I have a little data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

My data is so ancient
Drive sectors start to rot
I try to read my data
The effort comes to naught

Oh, data, data, data
I store it on my drive
And when it’s old and flawed
The data I’ll archive

Do You Have a Data Quartet that Can Carry a Tune?

Originally posted on the Synertrade blog in April, 2018.

Right now you’re probably thoroughly confused, but that’s okay. Because if I just entitled this article with the core topic of “data harmonization”, you’d be equally confused. And at least this title asks a relevant question.

So what is data harmonization? Technically, it refers to the effort to combine data from different sources into one consistent view that can be provided to a user in a comprehensible, and sensible, fashion. And that sounds simple, until you try to dissect what that means and what it entails.

Let’s start with the “combining”. This is easier said than done. Data can be fully structured (as it is in a well-defined relational record), semi-structured (as it is in a relational record with meta-data fields and blob fields or object records with ids and unstructured data), or fully unstructured (such as full text articles, audio and video files, etc.). So if one source is structured, and one is unstructured, how do you combine it?

Also, data can be stored in different types of systems — old school flat file databases and mega Excel sheets, slightly more refined column-oriented databases, traditional relational database systems, newer object oriented databases, and modern NoSQL databases are just a few of the more common examples.

And even if you can figure out how to mesh all of these, you have to remember that each database can be stored in a different character set (ANSI, ISO, Unicode, etc.) and use different base formats for its integer and real numbers (even if both databases call the data type float, there can be different bit counts AND different uses for the leading or trailing bit, one of which will usually signify sign).

And even once your data jockeys figure all of this out and can take all of your structured, semi-structured, and un-structured data from a variety of record (row) oriented, column oriented, relational, object, and NoSQL systems and put them all into one massive (virtual) data warehouse (through a super view) with appropriately mapped data elements that all use the same, commonly defined, data types in the same character set, that’s just the entrance exam. They still haven’t passed the course, and you still can’t do anything. Why?

Just because you can get the data into one data store doesn’t mean you can actually do anything useful with the data. Why? When you have data across ten systems, you tend to have duplicate data across a dozen systems and until you can merge all copies of the data into one, de-duplicated, correct data record, any report you run will be incorrect and useless.

Why would you have ten (10) copies? Think about it. You will be keeping, and working with, data on a supplier in the MRP (manufactured / custom product details), ERP (locations and some master data), AP (for payments), Catalog (for finished products), e-Sourcing application (where data is collected), SRM (where performance data is tracked and relationships are managed), SCAR (to solve problems), CLM (where the contracts go), e-Invoicing (which processes all the invoices), and Risk Management (where you collect internal metrics and third party data), and maybe more.

And it’s not just as simple as identifying the 10 supplier records into one mega record, because all will have a supplier id, supplier name, basic contact information, etc. This means that you will have up to 10 copies of a piece of information when you only want one, and, moreover, not all will be the same. Why? Some system will use different ids, and some systems will have incorrect, misspelled, or abbreviated data. You will need to identify all copies of the data, correct and incorrect, and replace them with one, up to date, correct copy and do this for every single data record, which could be tens of thousands for suppliers, hundreds of thousands in your employee database, and millions in your product database. Not an easy task.

But even if you manage to do this, you’ll find that you still have critical gaps in the data, such as risk and sustainability scores, product and service catalogs beyond what you currently buy, deep location data (all facility locations, warehouse size, global lane options, etc.), and so on. You will need to enrich your data with data from third party sources and feeds in order to make it useable. And then you have to consider all of the issues that go hand in hand with mapping yet another source into the central (virtual) data warehouse (since the EDI feed could have its own schema), for dealing with duplicate, incorrect, and incomplete data (as certain values will be repeated, contact information could be out of date if the feed was only validated three months ago but you updated a new contact manually one month ago), and no single feed will have all the data you’re missing. So enrichment can be a challenge as well.

However, even when you manage to

  • (virtually) centralize,
  • de-duplicate and cleanse, and
  • enrich

the data, you’ll still find that your data is not harmonized. In order for data to be harmonized, it has to be structured for use. While you will be mapping each piece of data you import to a well defined record format, just having good record formats is not that useful. To do detailed analysis and reporting, you need well defined cubes and views on those cubes. Those cubes need to be well defined and well structured.

And even though analytics is the first application you think of that needs structure, it’s not the only one. The internal catalog also needs appropriately structured data to allow for fast searching and retrieval. The tax analysis system to review payments by country, vendor, and agreement to determine if there are reclamation options. And so on.

In other words, data harmonization is not a simple effort, and requires a quartet of capabilities to get it right:

  • schema and data format normalization for (virtual) centralization,
  • de-duplication and cleansing,
  • enrichment, and
  • structuring

And unless all of these capabilities are sound and in harmony, an organization will never achieve data harmonization, which is critical for Supply Management success.

And this is why an understanding not only of why you need harmonization, but what’s involved and why you might need a provider that understands the intricacies, is so important. In this article, we’ve tried to give you a sense of what’s involved so you can work with your IT department to ask potential vendors the right questions to separate who talks the talk from who actually walks the walk. The reality is that while every vendor understands the importance, not every vendor can do it well, especially in an efficient and timely manner.