Real analysis requires more than just classifying data into a simple hierarchy, rolling up some numbers, and generating a report or two. It requires more than traditional BI tools which, when all is said and done, don’t do much more than that.

Going back to the beginning when we first defined the New Horizons (Part 1 and Part 2), we find that modern AI still doesn’t always support even the foundations that the Spend Master Eric Strovink defined two decades ago.

Two decades ago, in the original series, we defined:

Meta Aggregation

You don’t just need roll-ups and derived dimensions, you also need “cluster dimensions”. The ability to consolidate spend / vendors in groupings in pre-set ranges (such as the number of consulting vendors under 100K, the number between 100K and 500K, and the number getting the big bucks at more than 500K a year). In other words, classic f(x) doesn’t cut it. You need g(f(x)). You need recursive functions. Which can dynamically update in real time not just when the ranges are changed, but when the hierarchy of the source dimensions is altered (or the source data altered).

Visual Crosstabs

The utility of Shneiderman diagrams (or “treemaps”) to display hierarchical information is well known; the treemap is useful because it is visually intuitive. The relative sizes of the rectangles represent the relative magnitude of spending; the colors indicate relative change in spending; and inner rectangles show the breakdown at the next level of the hierarchy.

Now, suppose that rather than the inner rectangles showing a lower level of the same hierarchy, instead they showed the breakdown of spending within another dimension entirely — i.e., a “visual crosstab.” The visual crosstab would not only show magnitudes, but trends as well.

Federation

One of the most serious limitations of OLAP analysis is the schema structure itself — typically a “star” schema, where a voluminous “fact” or “transaction” file is surrounded by supporting files, or “dimensions.” In the case of spend analysis, dimensions are Supplier, Cost Center, Commodity, and so on; transactions are typically AP records.

There are only certain ways that a dimension file can be linked to transaction files, and it isn’t always clear which file ought to be the transaction file and which files ought to be dimensions. For example, suppose that the transaction file consists of AP transactions, and a dimension file consists of invoice line items. The problem is that the invoice line item file is “moving faster” than the AP file; i.e. for every invoice number that appears in AP, there are multiple invoice lines that match. Which invoice line item should we link to?

There are numerous other examples. But the essential problem is that we have two separate datasets, and we’re trying to join them at the hip. There is an AP dataset, and there is an invoice line item dataset, and never the twain shall meet, except artificially. Even when there is no granularity issue at all, and when one dataset can be normalized or snowflaked such that every matching line item can be joined through from the other, the amount of effort required to set up the index->index->index relationships can be daunting.

Instead, why not create two separate datasets, efficiently and quickly; and then as a final step, federate them together on a common dimension? … Federation represents a key productivity enhancer for dataset creation, as well as a simplification to the dataset building process in general.

But that was just the beginning.