The Aggregation Lattice Framework

A successful performance strategy for very large databases (VLDB) often involves aggregations. This is why major DBMS vendors offer mature system support for aggregation maintenance and optimization. DB2 offers Materialized Query Tables (MQT), Teradata offers Aggregate Join Indexes (AJI), Oracle offers Materialized Views (MV). BI tool vendors support aggregate awareness and dynamic sourcing, while OLAP technologies offer cost-effective, in-memory retention alternatives. All too often, however, these powerful capabilities are used redundantly, without too much forethought, and although they often improve query performance, they sometimes also disappoint, by impacting ETL windows, availability, stability, sustainability and sometimes quality. This is an overview of InfoKarta’s approach to implementing an Aggregation Strategy. It proposes a model and a framework to reason about aggregates, qualify work-loads that benefit from them, and pick most effective levels from a total performance perspective. It also recommends design and implementation approaches for exploiting the DBMS aggregation technology at hand in a way that is resilient, sustainable, scalable and maintainable.