Daily Archives: November 3, 2015

Technological Damnation 87: OLAP

OLAP, short for on-line analytical processing, is a great thing, right? It is at the foundation of reporting tools like Business Objects and Cognos, that, when they were released, gave users unparalleled insights into raw data compared to the rather static reports they were used to. It was revolutionary. And that’s the kicker. It WAS.

Now, it’s old technology and, to make matters worse, there is still the widespread misconception that it is the right tool for Spend Analysis. As SI has addressed many times, often with the help of the Spend Master (and there is only one), nothing could be further from the truth. This post will summarize just a few of the reasons this damnation continues to savage us, and will likely continue to do so for some time.

OLAP has no real-time capability

On-line analytical processing is just a fancy term for pre-computing a large number of intermediate and final totals against a roll-up models so that, when a user logs in to a system, not only can they see a report, but drill down into each line item to pre-defined subtotals according to a fixed hierarchy. For example, they can click on total sales and then drill down into sales by region and then sales by country and then sales by state/province. However, if the model is ordered by geography, but the user wants by department and then by geography, unless there is another report where those totals have been pre-calculated, the user is out of luck.

… and ROLAP doesn’t count!

In OLAP, the roll-ups are pre-computed off-line at regular, pre-scheduled intervals. In ROLAP, the system will rebuild the hierarchical roll-up on the fly — but if the roll-up takes an hour to generate, who cares whether the user initiates or a system script initiates. It still takes too damn long.

OLAP requires a rigid data model

Not only does OLAP require a rigid definition of the roll-ups being done against a rigid hierarchy, but that rigid hierarchy requires a rigid data model to work against. One model, with one hierarchy per OLAP report. Multiple reports, multiple hierarchies — but only to the extent supported by the underlying data model. If the data is missing or not fine-grained enough for the OLAP data processor, OLAP just will not work.

And that just doesn’t work for spend analysis.

As SI has indicated dozens upon dozens of times over the years, spend analysis requires flexibility — the ability to redefine roll ups, drill-downs, and underlying data models to support the analysis the analyst needs to do — not the analysis a vendor thinks that the analyst needs to do.

OLAP requires a lot more server memory than an average organization can afford

Today it’s all about big data and big data is huge. This means that summaries are huge compared to simple static reports, especially OLAP roll up summaries. One detailed multi-level roll-up summary with one drill down report on terabytes of data can take over a hundred gigabytes and max out memory on an average server — but an organization will need dozens of such drill downs to even come anywhere in the vicinity to meeting its analysts’ needs, and a server with terabytes of (D)RAM. We are beyond server territory into mini super computer territory, and mini super computers come with price tags that start (well) over half a million.

There are alternatives to (R)OLAP that can actually do real-time analysis and reporting on tens of millions of records on an average high-end multi-core laptop, but given that these systems are still the exception, and not the norm, this damnation is going to be with us for decades.