A model of multilayer tiered architecture for big data
DOI:
https://doi.org/10.18046/syt.v14i37.2257Keywords:
Big data, data warehouse, multi-layered tiered architecture, repetitive structured data, non-repetitive unstructured data, hadoop, mapreduce, noSql.Abstract
Until recently, the issue of analytical data was related to Data Warehouse, but due to the necessity of analyzing new types of unstructured data, both repetitive and non-repetitive, Big Data arises. Although this subject has been widely studied, there is not available a reference architecture for Big Data systems involved with the processing of large volumes of raw data, aggregated and non-aggregated. There are not complete proposals for managing the lifecycle of data or standardized terminology, even less a methodology supporting the design and development of that architecture. There are architectures in small-scale, industrial and product-oriented, which limit their scope to solutions for a company or group of companies, focused on technology but omitting the functionality. This paper explores the requirements for the formulation of an architectural model that supports the analysis and management of data: structured, repetitive and non-repetitive unstructured; there are some architectural proposals –industrial or technological type– to propose a logical model of multi-layered tiered architecture, which aims to respond to the requirements covering both Data Warehouse and Big Data.
References
Agrawal, D. (2009). The reality of Real-Time Business Intelligence. En: M, Castellanos, U, Dayal. & T, Sellis. (Eds.), Lecture Notes in Business Information Processing. Vol. 27. Business Intelligence for the Real-Time Enterprise (pp. 75-88). Berlin Heidelberg : Germany : Springer-Verlag Berlin Heidelberg : Germany
Apache Hive TM. (n.d.). Retrieved from https://hive.apache.org/
Apache Impala. (n.d). Retrieved from: http://www.cloudera.com/products/apache-hadoop/impala.html
Apache Sqoop (2016, march 4). Retrieved from: http://sqoop.apache.org/
Apache SparkTM-Lightning-fast cluster computing. (n.d.). Retrieved from: http://spark.apache.org/
Apache Thrift - Home. (n.d.). Retrieved from https://thrift.apache.org/
Apache ZooKeeper - Home. (n.d.). Retrieved from https://zookeeper.apache.org/
Architecture - Apache Drill. (n.d.). Retrieved from http://drill.apache.org/architecture/
Bedi, P., Jindal, V., & Gautam, A. (2014). Beginning with big data simplified. In: Data Mining and Intelligent Computing (ICDMIC), 2014 International Conference on. IEEE. doi:10.1109/ICDMIC.2014.6954229
Brewer, E. (2012). CAP twelve years later: How the “rules” have changed. Computer. 45(2), 23-29.
Carter, S. (2013, Feb, 21). Social and BIG Data! #socbiz #ibmsocialbiz #bigdata #socialbusiness. Retrieved from: http://socialbusinesssandy.com/tag/big-data-2/page/14/
Chandarana, P. & Vijayalakshmi, M. (2014). Big data analytics frameworks. In Circuits, Systems, Communication and Information Technology Applications (CSCITA), 2014 international conference on (pp. 430-434. IEEE.
Cox, M. & Ellsworth, D. (1997). Application-controlled demand paging for out-of-core visualization [NASA Reports]. Retrieved from: http://www.nas.nasa.gov/assets/pdf/techreports/1997/nas-97-010.pdf
Cuzzocrea, A. (2014). Privacy and security of big data: current challenges and future research perspectives. In: Proceedings of the First International Workshop on Privacy and Security of Big Data (pp. 45-47). New York, NY: ACM. http://doi.acm.org/10.1145/2663715.2669614
Demchenko, Y., Laat, C. & Membrey, P. (2014). Defining architecture components of the big data ecosystem. In: Collaboration Technologies and Systems (CTS), 2014 International Conference on, 104-112. IEEE.
Díaz, Ma. (2011). Evaluación de la herramienta de código libre Apache Hadoop [thesis]. Universidad Carlos III de Madrid Escuela Politécnica Superior: Leganés, España.
Gudivada, V., Rao, D. & Raghavan, V. (2014). NoSQL systems for big data management. In: 2014 IEEE World Congress on Services (pp. 190-197). IEEE.
HDFS architecture guide. (2013, April 8). Retrieved from: http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html
Hewlett Packard. (2013). HP Reference Architecture for MapR M5 [technical white paper]. Retrieved from: https://www.mapr.com/sites/default/files/hp_reference_architecture_for_mapr_m5.pdf
Inmon, W. (2005). Building the data warehouse [4a ed.]. Indianapolis, IN: Wiley.
Inmon, W.,Strauss, D. & Neushloss, G. (2008). DW 2.0: The Architecture for the Next Generation of Data Warehousing. Burlington, MA: Morgan Kaufmann
Inmon. H. & Linstedt, D. (2014). Data architecture: A primer for the data scientist: big data, data warehouse and data vault. Waltham, MA: Morgan Kaufmann.
Katal, A., Wazid, M. & Goudar, R. (2013). Big data: Issues, challenges, tools and good practices. In: Contemporary Computing (IC3), 2013 Sixth International Conference on (pp. 404-409). IEEE.
Kimball, R. (2011). The evolving role of the enterprise data warehouse in the era of big data analytics [Kimball Group white paper]. Retrieved from: http://www.montage.co.nz/assets/Brochures/DataWarehouseBigDataAnalyticsKimball.pdf
Kimball, R. (2012). Newly emerging best practices for big data [Kimball Group, white paper]. Retrieved from: http://www.kimballgroup.com/wp-content/uploads/2012/09/Newly-Emerging-Best-Practices-for-Big-Data1.pdf
Kimball, R., Ross, M., Thorthwaite, W., Becker, B. & Mundy, J. (2008). The data warehouse lifecycle toolkit [2a ed.]. Indianapolis, IN: Wiley.
Lomotey, R. K., & Deters, R. (2014). Towards knowledge discovery in big data. In: Service Oriented System Engineering (SOSE), 2014 IEEE 8th International Symposium on (pp. 181-191). IEEE.
MacDonald, A. (2015). Integrating SAP HANA and hadoop. Boston, MA: SAP Press.
Maiorescu, T. (2010). General Information on Business Intelligence and OLAP systems architecture. In: Computer and Automation Engineering (ICCAE), 2010 The 2nd International Conference on (V.2, pp. 294-297). IEEE.
Manikandan, S. G., & Ravi, S. (2014). Big data analysis using Apache Hadoop. In: IT Convergence and Security (ICITCS), 2014 International Conference on. doi: 10.1109/ICITCS.2014.7021746
Manning, C. & Schütze. H. (1999). Foundations of statistical natural language processing. Cambridge, MA: The MIT.
Marz, N. (n.d). Storm, distributed and fault-tolerant realtime computation. Retrieved from: http://cloud.berkeley.edu/data/storm-berkeley.pdf
Muntean, M., & Surcel, T. (2013). Agile BI - The Future of BI. Informatica Económica, 17(3), 114–124.
Nam, T., Choi, K., Ok, C. & Yeom, K. (2014). Service composition framework for big data service. In: Future Internet of Things and Cloud (FiCloud), 2014 International Conference on (pp. 328-333). IEEE.
Nandimath, J., Banerjee, E., Patil, A., Kakade, P., & Vaidya, S. (2013). Big Data analysis using Apache Hadoop. In: 2013 IEEE 14th International Conference on Information Reuse & Integration (IRI) (pp. 700-703). IEEE.
Oracle Corp. (2015). An enterprise architect's guide to big data [Oracle enterprise architecture - white paper.]. Retrieved from: http://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdf
Pal, A. & Agrawal, S. (2014). An experimental approach towards big data for analyzing memory utilization on a hadoop cluster using HDFS and MapReduce. In: Networks & Soft Computing (ICNSC), 2014 First International Conference on (pp. 442-447). IEEE.
Schaffner, J., Bog, A., Krüger, J., & Zeier, A. (2009). A hybrid row-column OLTP database architecture for operational reporting. In: M. Castellanos, U. Dayal, & T. Sellis (Eds.), Business intelligence for the real-time enterprise (pp. 61-74). Berlin Heidelberg, Germany: Springer.
Todman, C. (2001). Designing a data warehouse: Supporting customer relationship management. Nueva Jersey, NJ: Prentice Hall.
Vaish, G. (2013). Getting started with NoSQL. Birmingham UK: Packt.
Welcome to ApacheTM Hadoop®! (n.d.). Retrieved from: https://hadoop.apache.org/
YiChuan, S. & Yao, X. (2012). Research of Real-time Data Warehouse Storage Strategy Based on Multi-level Caches. Physics Procedia, 25, 2315–2321.
Zhang, R., Hildebrand, D., & Tewari, R. (2014). In unity there is strength: Showcasing a unified Big Data platform with MapReduce Over both object and file storage. In: Big Data (Big Data), 2014 IEEE International Conference on (pp. 960-966). IEEE.
Apache Hive TM. (n.d.). Retrieved from https://hive.apache.org/
Apache Impala. (n.d). Retrieved from: http://www.cloudera.com/products/apache-hadoop/impala.html
Apache Sqoop (2016, march 4). Retrieved from: http://sqoop.apache.org/
Apache SparkTM-Lightning-fast cluster computing. (n.d.). Retrieved from: http://spark.apache.org/
Apache Thrift - Home. (n.d.). Retrieved from https://thrift.apache.org/
Apache ZooKeeper - Home. (n.d.). Retrieved from https://zookeeper.apache.org/
Architecture - Apache Drill. (n.d.). Retrieved from http://drill.apache.org/architecture/
Bedi, P., Jindal, V., & Gautam, A. (2014). Beginning with big data simplified. In: Data Mining and Intelligent Computing (ICDMIC), 2014 International Conference on. IEEE. doi:10.1109/ICDMIC.2014.6954229
Brewer, E. (2012). CAP twelve years later: How the “rules” have changed. Computer. 45(2), 23-29.
Carter, S. (2013, Feb, 21). Social and BIG Data! #socbiz #ibmsocialbiz #bigdata #socialbusiness. Retrieved from: http://socialbusinesssandy.com/tag/big-data-2/page/14/
Chandarana, P. & Vijayalakshmi, M. (2014). Big data analytics frameworks. In Circuits, Systems, Communication and Information Technology Applications (CSCITA), 2014 international conference on (pp. 430-434. IEEE.
Cox, M. & Ellsworth, D. (1997). Application-controlled demand paging for out-of-core visualization [NASA Reports]. Retrieved from: http://www.nas.nasa.gov/assets/pdf/techreports/1997/nas-97-010.pdf
Cuzzocrea, A. (2014). Privacy and security of big data: current challenges and future research perspectives. In: Proceedings of the First International Workshop on Privacy and Security of Big Data (pp. 45-47). New York, NY: ACM. http://doi.acm.org/10.1145/2663715.2669614
Demchenko, Y., Laat, C. & Membrey, P. (2014). Defining architecture components of the big data ecosystem. In: Collaboration Technologies and Systems (CTS), 2014 International Conference on, 104-112. IEEE.
Díaz, Ma. (2011). Evaluación de la herramienta de código libre Apache Hadoop [thesis]. Universidad Carlos III de Madrid Escuela Politécnica Superior: Leganés, España.
Gudivada, V., Rao, D. & Raghavan, V. (2014). NoSQL systems for big data management. In: 2014 IEEE World Congress on Services (pp. 190-197). IEEE.
HDFS architecture guide. (2013, April 8). Retrieved from: http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html
Hewlett Packard. (2013). HP Reference Architecture for MapR M5 [technical white paper]. Retrieved from: https://www.mapr.com/sites/default/files/hp_reference_architecture_for_mapr_m5.pdf
Inmon, W. (2005). Building the data warehouse [4a ed.]. Indianapolis, IN: Wiley.
Inmon, W.,Strauss, D. & Neushloss, G. (2008). DW 2.0: The Architecture for the Next Generation of Data Warehousing. Burlington, MA: Morgan Kaufmann
Inmon. H. & Linstedt, D. (2014). Data architecture: A primer for the data scientist: big data, data warehouse and data vault. Waltham, MA: Morgan Kaufmann.
Katal, A., Wazid, M. & Goudar, R. (2013). Big data: Issues, challenges, tools and good practices. In: Contemporary Computing (IC3), 2013 Sixth International Conference on (pp. 404-409). IEEE.
Kimball, R. (2011). The evolving role of the enterprise data warehouse in the era of big data analytics [Kimball Group white paper]. Retrieved from: http://www.montage.co.nz/assets/Brochures/DataWarehouseBigDataAnalyticsKimball.pdf
Kimball, R. (2012). Newly emerging best practices for big data [Kimball Group, white paper]. Retrieved from: http://www.kimballgroup.com/wp-content/uploads/2012/09/Newly-Emerging-Best-Practices-for-Big-Data1.pdf
Kimball, R., Ross, M., Thorthwaite, W., Becker, B. & Mundy, J. (2008). The data warehouse lifecycle toolkit [2a ed.]. Indianapolis, IN: Wiley.
Lomotey, R. K., & Deters, R. (2014). Towards knowledge discovery in big data. In: Service Oriented System Engineering (SOSE), 2014 IEEE 8th International Symposium on (pp. 181-191). IEEE.
MacDonald, A. (2015). Integrating SAP HANA and hadoop. Boston, MA: SAP Press.
Maiorescu, T. (2010). General Information on Business Intelligence and OLAP systems architecture. In: Computer and Automation Engineering (ICCAE), 2010 The 2nd International Conference on (V.2, pp. 294-297). IEEE.
Manikandan, S. G., & Ravi, S. (2014). Big data analysis using Apache Hadoop. In: IT Convergence and Security (ICITCS), 2014 International Conference on. doi: 10.1109/ICITCS.2014.7021746
Manning, C. & Schütze. H. (1999). Foundations of statistical natural language processing. Cambridge, MA: The MIT.
Marz, N. (n.d). Storm, distributed and fault-tolerant realtime computation. Retrieved from: http://cloud.berkeley.edu/data/storm-berkeley.pdf
Muntean, M., & Surcel, T. (2013). Agile BI - The Future of BI. Informatica Económica, 17(3), 114–124.
Nam, T., Choi, K., Ok, C. & Yeom, K. (2014). Service composition framework for big data service. In: Future Internet of Things and Cloud (FiCloud), 2014 International Conference on (pp. 328-333). IEEE.
Nandimath, J., Banerjee, E., Patil, A., Kakade, P., & Vaidya, S. (2013). Big Data analysis using Apache Hadoop. In: 2013 IEEE 14th International Conference on Information Reuse & Integration (IRI) (pp. 700-703). IEEE.
Oracle Corp. (2015). An enterprise architect's guide to big data [Oracle enterprise architecture - white paper.]. Retrieved from: http://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdf
Pal, A. & Agrawal, S. (2014). An experimental approach towards big data for analyzing memory utilization on a hadoop cluster using HDFS and MapReduce. In: Networks & Soft Computing (ICNSC), 2014 First International Conference on (pp. 442-447). IEEE.
Schaffner, J., Bog, A., Krüger, J., & Zeier, A. (2009). A hybrid row-column OLTP database architecture for operational reporting. In: M. Castellanos, U. Dayal, & T. Sellis (Eds.), Business intelligence for the real-time enterprise (pp. 61-74). Berlin Heidelberg, Germany: Springer.
Todman, C. (2001). Designing a data warehouse: Supporting customer relationship management. Nueva Jersey, NJ: Prentice Hall.
Vaish, G. (2013). Getting started with NoSQL. Birmingham UK: Packt.
Welcome to ApacheTM Hadoop®! (n.d.). Retrieved from: https://hadoop.apache.org/
YiChuan, S. & Yao, X. (2012). Research of Real-time Data Warehouse Storage Strategy Based on Multi-level Caches. Physics Procedia, 25, 2315–2321.
Zhang, R., Hildebrand, D., & Tewari, R. (2014). In unity there is strength: Showcasing a unified Big Data platform with MapReduce Over both object and file storage. In: Big Data (Big Data), 2014 IEEE International Conference on (pp. 960-966). IEEE.
Downloads
Published
2016-08-05
Issue
Section
Discussion papers
License
This journal is licensed under the terms of the CC BY 4.0 licence (https://creativecommons.org/licenses/by/4.0/legalcode).
