Problems with traditional large-scale systems in big data

11.6 Conclusions and Future Directions

Big data problems have brought many changes in the way data is processed and managed over time. Today, data is not just posing challenge in terms of volume but also in terms of its high speed generation. The data quality and validity varies from source to source, and thus are difficult to process. This issue has led to the development of several stream processing engines/platforms by different companies such as Yahoo, LinkedIn, etc. Besides better performance in terms of latency, stream processing overcomes another shortcoming of batch data processing systems, i.e., scaling with high “velocity” data. Availability of several platforms also resulted in another challenge for user organizations in terms of selecting the most appropriate stream processing platform for their needs. In this chapter, we proposed a taxonomy that facilitated the comparison of different features offered by the stream processing platforms. Based on this taxonomy we presented a survey and comparison study of five open source stream processing engines. Our comparison study provides an insight of how to select the best platform for a given use case.

From the comparison of different open source stream processing platforms based on our proposed taxonomy, we observed that each platform offers very specific special feature that makes its architecture unique. However, some features make a stream processing platform more applicable for different scenarios. For example, if the organization's data volume changes dynamically, it is better to choose a platform such as Storm which allows dynamic addition of nodes rather than Yahoo! S4. Similarly, if an organization wants to process all the data that is ingested into the system, the guaranteed data processing feature is what it should look for. In contrast to commercial offerings, organizations can save on licensing fees by using open-source platforms. However, the support given for maintenance of such platforms becomes an important factor in making decisions about adopting a particular platform. The user base and support given for each platform varies quite drastically. Storm has the largest user base and also supports services. Yahoo! S4 comes with almost no support.

Based on the survey, it also becomes clear that the performance of a stream processing system depends on multiple factors. However, the performance will always be limited by the capacity of the underlying cluster environment in which real processing is done. More or less every system that was surveyed does not allow using cloud computing resources which can scale up and down according to the volume and velocity of data that needs to be processed. Moreover, the job scheduling mechanisms used by the systems are not very sophisticated and do not take into the consideration the performance of underlying infrastructure which can be quite heterogeneous in some cases. In the future, we would like to conduct a cost and risk analysis of different streaming platforms and conduct a more extensive comparison study. The current taxonomy is derived after studying different open-source stream processing platforms, which limits the scope of our taxonomy. To overcome this limitation, we would also study some key commercially available stream processing platforms such as IBM Stream.

Traditional big data use cases

Certain big data problems and use cases are common to every organization, such as e-mail, contracts and documents, web data, and social media data. In addition, almost every industry has specific cases where they have big data management problems to solve. Many industries have always had to deal with huge volumes and varied types of data. Organizations in the telecommunications industry must keep track of the huge network of nodes through which communications can pass and the actual activity and history of connections. Finance has to process both the history of prices of financial products and the detailed history of financial transactions. Organizations in airplane manufacturing and operation must track the history of every part and screw of every airplane and vehicle planned and operated. Publishing organizations must track all the components of documents through the development versions and production process. Interestingly, pharmaceutical firms have similar strict document management requirements for their drug submissions to the FDA in the United States, and comparable requirements in other countries; thus, for pharmaceutical firms advanced document management capabilities are a core competency and a traditional “big data” problem.

4.2 Big Data Problems in Bioinformatics

Main categorizes of the big data problems in bioinformatics include seven terms as entitled by Fig. 9 [6]. They are described in the following with more details.

Microarray data analysis: Because of reducing the operational costs and widespread use of microarray experiments, the size and number of microarray datasets are increasing quickly. Beside, microarray experiments are carried out for gene-sample-time space to store the noticeable changes over time or various stages of a disease. Fast construction of the coexpression and regulatory networks needs big data technologies with the aid of voluminous microarray.

Gene–gene network analysis: Gene coexpression network analysis predicts the relation among different gene–gene networks achieved from gene expression analysis. Differential coexpression analysis looks for those of the changes which are created by the gene complexes over time or various stages of a disease. This process leads to find the correlation between gene complexes and traits of interest. Consequently, gene coexpression network analysis is defined as a complex and highly iterative problem that needs large-scale data analytics systems.

PPI data analysis: PPI complexes and their changes contain high information about various diseases. PPI networks are considered in different domains of life sciences in cooperation with the production of voluminous data. The volume, velocity, and variety of data cause PPI complex analytics to be a genuine big data problem. Hence, an efficient and scalable architecture should be provided to give the fast and accurate PPI complex generation, validation, and rank aggregation.

Sequence analysis: RNA sequencing technology is addressed as a strong successor to the microarray technology. This successor is resulted because of its more accurate and quantitative gene expression measurements. RNA sequence data involve additional information that needs considerable machine-learning models. Therefore, big data technologies can be applied to indicate mutations, allele-specific expressions, and exogenous RNA contents [e.g., viruses].

Evolutionary research: The recent advances in molecular biological technologies have led to a noticeable source for big data generation. Various projects at microbial level [e.g., genome sequencing and microarrays] have generated huge amounts of data. Bioinformatics has been considered as an important platform for the analysis and achievement of this essential information. Functional trend of the adaptation and evolution by microbial research is an important big data problem in bioinformatics.

Pathway analysis: Pathway analysis makes an association between genetic products and phenotypes of interest. This correlation is done to estimate gene function, identify biomarkers and traits, and also make a category of the patients and samples. Association analysis on huge volumes of the data can be performed by increasing genetic, genomic, metabolomics, and proteomic data as well as by offering the big data technologies.

Disease network analysis: Large disease networks are provided by formulating various species [e.g., human]. Complexity and data of these networks increase over time as well as new networks are added by using different sources in their own format. Consequently, sophisticated networks of molecular phenotypes cause to estimative genes or mechanisms for the disease-associated traits. Some of the efficient integration methods should be applied to evaluate multiple, heterogeneous omics databases.

Hadoop as an Analytical Engine

The other option for Hadoop within an analytics solution is where the entire data mining algorithm is implemented within the Hadoop programming environment and it acts as the data mining engine shown in Figure 11.4. This is used when there is no option of reducing, aggregating, or sampling the data to eliminate the V’s from Big Data characteristics. This becomes quite complex as the performance variables, the heart of innovation within analytics modeling, cannot be easily added to the data set without creating an additional storage layer. However, various problems actually require running the entire Big Data set looking for specific trends or performing fuzzy search or correlations between variables through Hadoop and its data access programs.

Outside of analytics—as defined in the first chapter—Hadoop can be used as generic data processing engine that can search and look for patterns and correlations almost like a reporting tool of sorts. It should be treated as a commodity data processing layer that can be applied to any overwhelming data size problem.

11.1 Introduction

The goal of this book is to influence rational thinking when it comes to big data and big data analytics. Not every business problem is a big data problem, and not every big data solution is appropriate for all organizations. One benefit of rational thought is that instead of blindly diving into the deep end of the big data pool, following the guidance in this book will help frame those scenarios in which your business can [as well as help determine it cannot] benefit from big data methods and technologies. If so, the guidance should also help to clarify the evaluation criteria for scoping the acquisition and integration of technology.

This final chapter is meant to bring the presented ideas together and motivate the development of a roadmap for evaluating and potentially implementing big data within the organization. Each of the sections in this chapter can be used as a springboard for refining a more detailed task plan with specific objectives and deliverables. Hopefully, this chapter can help devise a program management approach for transitioning to an environment that effectively exploits these exciting technologies.

MDM and Big Data – The N-Squared Problem

Although many traditional data processes can easily scale to take advantage of Big Data tools and techniques, MDM is not one of them. MDM has a Big Data problem with Small Data. Because MDM is based on ER, it is subject to the O[n2] problem. O[n2] denotes the effort and resources needed to complete an algorithm or data process growing in proportion to the square of the number of records being processed. In other words, the effort required to perform ER on 100 records is 4 times more than the effort to perform the same ER process on 50 records because 1002 = 10,000 and 502 = 2,500 and 10,000/2,500 = 4. More simply stated, it takes 4 times more effort to scale from 50 records to 100 records because [100/50]2 = 4.

Big Data not only brings challenging performance issues to ER and MDM, it also exacerbates all of the dimensions of MDM previously discussed. Multi-domain, multi-channel, hierarchical, and multi-cultural MDM are impacted by the growing volume of data that enterprises must deal with. Although the problems are formidable, ER and MDM can still be effective for Big Data. Chapters 9 and 10Chapter 9Chapter 10 focus on Big Data MDM.

Summary

Big Data is the hot frontier of today’s information technology development. The Internet of Things, the Internet, and the rapid development of mobile communication networks have spawned big data problems and have created problems of speed, structure, volume, cost, value, security privacy, and interoperability. Traditional IT processing methods are impotent when faced with big data problems, because of their lack of scalability and efficiency. Big Data problems need to be solved by Cloud computing technology, while big data can also promote the practical use and implementation of Cloud computing technology. There is a complementary relationship between them. We focus on infrastructure support, data acquisition, data storage, data computing, data display, and interaction to describe several types of technology developed for big data, and then describe the challenges and opportunities of big data technology from a different angle from the scholars in the related fields. Big data technology is constantly growing with the surge of data volume and processing requirements, and it is affecting our daily habits and lifestyles.

1.1 Introduction

Although the term “Big Data” has become popular, there is no general consensus about what it really means. Often, many professional data analysts would imply the process of extraction, transformation, and load [ETL] for large datasets as the connotation of Big Data. A popular description of Big Data is based on three main attributes of data: volume, velocity, and variety [or 3Vs]. Nevertheless, it does not capture all the aspects of Big Data accurately. In order to provide a comprehensive meaning of Big Data, we will investigate this term from a historical perspective and see how it has been evolving from yesterday’s meaning to today’s connotation.

Historically, the term Big Data is quite vague and ill defined. It is not a precise term and does not carry a particular meaning other than the notion of its size. The word “big” is too generic; the question how “big” is big and how “small” is small [1] is relative to time, space, and circumstance. From an evolutionary perspective, the size of “Big Data” is always evolving. If we use the current global Internet traffic capacity [2] as a measuring stick, the meaning of Big Data volume would lie between the terabyte [TB or 1012 or 240] and zettabyte [ZB or 1021 or 270] range. Based on the historical data traffic growth rate, Cisco claimed that humans have entered the ZB era in 2015 [2]. To understand the significance of the data volume’s impact, let us glance at the average size of different data files shown in Table 1.

Table 1. Typical Size of Different Data Files

The main aim of this chapter is to provide a historical view of Big Data and to argue that it is not just 3Vs, but rather 32Vs or 9Vs. These additional Big Data attributes reflect the real motivation behind Big Data analytics [BDA]. We believe that these expanded features clarify some basic questions about the essence of BDA: what problems Big Data can address, and what problems should not be confused as BDA. These issues are covered in the chapter through analysis of historical developments, along with associated technologies that support Big Data processing. The rest of the chapter is organized into eight sections as follows:

A historical review for Big Data

Interpretation of Big Data 3Vs, 4Vs, and 6Vs

Defining Big Data from 3Vs to 32Vs

Big Data and Machine Learning [ML]

Big Data and cloud computing

Hadoop, Hadoop distributed file system [HDFS], MapReduce, Spark, and Flink

ML + CC [Cloud Computing] → BDA and guidelines

Conclusion

XGBoost

XGBoost10 is an open-source library for optimized distributed gradient tree boosting [28] that gained popularity lately as it has been the algorithm of choice for many winning teams from several ML competitions. This gradient boosting framework provides parallel tree augmentation to solve many data science problems quickly. XGBoost runs in a distributed environment [Hadoop, SGE, MPI] and can solve big data problems. The term gradient boosting originates in the idea of enhancing or improving a single weak model by combining it with several other weak models to produce a collectively robust model. The weak learning models are reinforced through iterative learning.

Interfaces for C ++, Java, Python, R, and Julia and works on Linux, Windows, and Mac OS.

Supports the Apache Hadoop/Spark/Flink, and DataFlow distributed processing frameworks.

High performance in model training and execution speed.

Parallelization of the tree construction using all CPU cores during the training.

Distributed computing for training large models with the help of a machine cluster.

Out-of-core computing for huge amounts of data that do not fit in memory.

Cache optimized data structures for optimal use of the hardware.

Boosting library designed for tabular data, hence, won't work for other tasks like NLP or Computer Vision.

Getting Started with Data Governance

Ideally, data governance is, eventually, an enterprise-wide effort involving multiple business groups and IT. In most companies, though, it starts with a project or two and then grows. If you’re responsible for getting a governance effort launched, start by creating the organization, then focus on defining processes and data.