Privacy Policy Disclaimer
  Advanced SearchBrowse




Journal Article

Building a Hybrid Warehouse: Efficient Joins between Data Stored in HDFS and Enterprise Warehouse


Tian,  Yuanyuan
External Organizations;

Özcan,  Fatma
External Organizations;

Zou,  Tao
External Organizations;


Goncalves,  Romulo
0 Pre-GFZ, Departments, GFZ Publication Database, Deutsches GeoForschungsZentrum;

Pirahesh,  Hamid
External Organizations;

External Ressource
No external resources are shared
Fulltext (public)
There are no public fulltexts stored in GFZpublic
Supplementary Material (public)
There is no public supplementary material available

Tian, Y., Özcan, F., Zou, T., Goncalves, R., Pirahesh, H. (2016): Building a Hybrid Warehouse: Efficient Joins between Data Stored in HDFS and Enterprise Warehouse. - ACM Transactions on Database Systems, 41, 4, 1-38.

Cite as: https://gfzpublic.gfz-potsdam.de/pubman/item/item_5025244
The Hadoop Distributed File System (HDFS) has become an important data repository in the enterprise as the center for all business analytics, from SQL queries and machine learning to reporting. At the same time, enterprise data warehouses (EDWs) continue to support critical business analytics. This has created the need for a new generation of a special federation between Hadoop-like big data platforms and EDWs, which we call the hybrid warehouse. There are many applications that require correlating data stored in HDFS with EDW data, such as the analysis that associates click logs stored in HDFS with the sales data stored in the database. All existing solutions reach out to HDFS and read the data into the EDW to perform the joins, assuming that the Hadoop side does not have efficient SQL support. In this article, we show that it is actually better to do most data processing on the HDFS side, provided that we can leverage a sophisticated execution engine for joins on the Hadoop side. We identify the best hybrid warehouse architecture by studying various algorithms to join database and HDFS tables. We utilize Bloom filters to minimize the data movement and exploit the massive parallelism in both systems to the fullest extent possible. We describe a new zigzag join algorithm and show that it is a robust join algorithm for hybrid warehouses that performs well in almost all cases. We further develop a sophisticated cost model for the various join algorithms and show that it can facilitate query optimization in the hybrid warehouse to correctly choose the right algorithm under different predicate and join selectivities.