As per our last post, if you want to design and run a relatively smooth running supply chain, you need a lot of data. As per our post, at a minimum:
- product compositions
- supplier locations
- route details
- alternate products, suppliers, and routes
- sanctions
- denied parties
- tariffs, export and import
- taxes and recoverables
- carbon/GHG
- commodity market and product cost data
- currency conversions and trends
- natural disaster risks
- man-made disaster data
- consumer and market sentiment data
Which leaves you with two major problems.
- where do you get it?
- how do you manage it?
There are two choices on where to get it:
- data consolidators and brokers
- government / public sector sources
As much as possible, you want to rely on option b), because, in this AI-HYPE filled world, data is now the most valuable commodity, the data brokers know it, and even when they are getting it from a government / public source (for free) and then processing it for your consumption, they are charging a premium for it. A subscription to even a fraction of the above data could cost you more than the annual SaaS subscriptions you are feeding it into (not counting your AI token costs which are going to continue to increase without bound if you are unnecessarily relying on AI for tasks you do not, and should not, be using Gen-AI LLMs for).
So you want to use the cheap/free government and public sources as much as possible. While this sounds easy enough, every single source will be in a different format, with different access requirements, different update frequencies, and different levels of completeness. We’re talking about everything from Excel files to real-time json requests, with multiple types of authorizations, access protocols, and transport protocols.
This bring us to the second issue — how do you manage it?
You basically need a DIY data orchestration platform. And the reality is, the majority of today’s orchestration providers, despite their grand claims, don’t really do that! Out-of-the-box, you can only integrate solutions they have already integrated with, and only import any data they have previously mapped. Plus, they are limited in intake to what they have designed for. Most of them aren’t even as powerful as last generation data mapping frameworks that allowed a data analyst to integrate all of the data from various sources into one common, central, database. (Now, it typically involved creating yet another data warehouse / lake / lakehouse.)
What you need is a next-gen supply chain orchestration platform that was built from the ground up to allow you to plug, play, and orchestrate data sources as well as workflows and applications using pre-defined mappings, Web 3.0 Markup, standard terminology, and statistical AI (with known confidence) that will auto-map as much as possible, minimize what can’t be auto-mapped, and generate additional data objects and tables for storage where your systems aren’t already handling that type of data.
A next generation system that also incorporates auto-mapping, auto-schema-extension, and auto-data-orchestration alongside workflow construction and third party system integration using APIs, MCP, and other integration technologies along with all of the common authorization protocols. A fully dynamic data platform that can serve as the core of a next generation SCP platform. One that is a level beyond the majority of platforms calling them orchestration platforms today,
And, finally, one that understands how to organize, federate, and classify data for true analysis.
But that’s another post!
