Architectural Foundations for Migrating Legacy Warehouses to Hadoop-Based Hybrid Data Platforms: A Review
DOI:
https://doi.org/10.63282/3050-922X.IJERET-V1I4P109Keywords:
Data Migration, Hadoop Ecosystem, SQL-On-Hadoop, Columnar File Formats, Apache SparkAbstract
Organizations that move analytical workloads from relational warehouses to Hadoop-based platforms must decide how data is stored, queried, replicated, and verified in the new environment. This review examines ten peer-reviewed publications that describe the building blocks of such platforms. These cover distributed computation and structured storage, SQL access to distributed data, columnar file organization, warehouse data pipelines, massively parallel query engines, and unified in-memory processing. The review is organized in five sections, each drawing on two sources. The sources show that SQL-like interfaces lower the barrier for analysts, that block replication and checkpointing protect stored data, and that columnar formats and native execution engines reduce scan cost. They also show that declarative optimizers let one engine serve many workloads. The synthesis offers practical guidance for planning migrations. It stresses matching file formats to access patterns, choosing engines by latency need, and preserving metadata and lineage so that source-to-target parity can be demonstrated.
References
[1] Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1), 107–113.
[2] Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., & Gruber, R. E. (2008). Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems, 26(2), Article 4.
[3] Thusoo, A., Sen Sarma, J., Jain, N., Shao, Z., Chakka, P., Anthony, S., Liu, H., Wyckoff, P., & Murthy, R. (2009). Hive: A warehousing solution over a map-reduce framework. Proceedings of the VLDB Endowment, 2(2), 1626–1629.
[4] Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010). The Hadoop Distributed File System. In Proceedings of the 26th IEEE Symposium on Mass Storage Systems and Technologies (pp. 1–10). IEEE.
[5] Thusoo, A., Shao, Z., Anthony, S., Borthakur, D., Jain, N., Sen Sarma, J., Murthy, R., & Liu, H. (2010). Data warehousing and analytics infrastructure at Facebook. In Proceedings of the ACM SIGMOD International Conference on Management of Data (pp. 1013–1020). ACM.
[6] Melnik, S., Gubarev, A., Long, J. J., Romer, G., Shivakumar, S., Tolton, M., & Vassilakis, T. (2010). Dremel: Interactive analysis of web-scale datasets. Proceedings of the VLDB Endowment, 3(1–2), 330–339.
[7] Huai, Y., Chauhan, A., Gates, A., Hagleitner, G., Hanson, E. N., O'Malley, O., Pandey, J., Yuan, Y., Lee, R., & Zhang, X. (2014). Major technical advancements in Apache Hive. In Proceedings of the ACM SIGMOD International Conference on Management of Data (pp. 1235–1246). ACM.
[8] Kornacker, M., Behm, A., Bittorf, V., Bobrovytsky, T., Ching, C., Choi, A., Erickson, J., Grund, M., Hecht, D., Jacobs, M., Joshi, I., Kuff, L., Kumar, D., Leblang, A., Li, N., Pandis, I., Robinson, H., Rorke, D., Rus, S., … Yoder, M. (2015). Impala: A modern, open-source SQL engine for Hadoop. In Proceedings of the 7th Biennial Conference on Innovative Data Systems Research (CIDR).
[9] Armbrust, M., Xin, R. S., Lian, C., Huai, Y., Liu, D., Bradley, J. K., Meng, X., Kaftan, T., Franklin, M. J., Ghodsi, A., & Zaharia, M. (2015). Spark SQL: Relational data processing in Spark. In Proceedings of the ACM SIGMOD International Conference on Management of Data (pp. 1383–1394). ACM.
[10] Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M. J., Ghodsi, A., Gonzalez, J., Shenker, S., & Stoica, I. (2016). Apache Spark: A unified engine for big data processing. Communications of the ACM, 59(11), 56–65.
