There has been data explosion over the past few years and is continuing as you are reading this. Every day we generate a lot of data, without being aware of doing so, through social media websites, digital pictures, Meteorological data, transaction related data, Mobil phone etc. 90% of the of the heap of data that is available today is created in the last 2-3 years. This is called big Data. In other words, Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.
Big data is characterized by three V’s: Volume, Velocity and Variety
Volume:
Data volume is resulted in by many factors. Like Transaction-based data stored over the years, social media(eg: 12 Terabytes of data is generated each day by Tweets alone). Call details, Sensor and machine-to-machine data. With decreasing storage costs,data is being collected every data and being analysed to predict trends and sentiments. In other words, making sense out of the vast amount of data.This volume presents the most immediate challenge to conventional IT structures. It calls for scalable storage, and a distributed approach to querying. Many companies already have large amounts of archived data, perhaps in the form of logs, but not the capacity to process it.
Velocity:
Data is being generated at great speeds, and analyzing it as soon as it gets generated is equally important to get benefit out of it in most cases. Say, data collected in an aero plane needs to be analyzed immediately to avoid possible mishaps. Other the data collected is of no use. Another example is to analyze 5 million trade events created each day to identify potential fraud. It’s not just the velocity of the incoming data that’s the issue: it’s possible to stream fast-moving data into bulk storage for later batch processing, for example. The importance lies in the speed of the feedback loop, taking data from input through to decision.
Variety:
Data today comes in all types of formats. For eg: Pictures, Music files all come in multiple formats. Structured, numeric data in traditional databases. Information created from line-of-business applications. Unstructured text documents, email, video, audio, stock ticker data and financial transactions. Rarely does data present itself in a form perfectly ordered and ready for processing. A common theme in big data systems is that the source data is diverse, and doesn’t fall into neat relational structures. It could be text from social networks, image data, a raw feed directly from a sensor source. None of these things come ready for integration into an application.
Even on the web, where computer-to-computer communication ought to bring some guarantees, the reality of data is messy. Different browsers send different data, users withhold information, they may be using differing software versions or vendors to communicate with you. And you can bet that if part of the process involves a human, there will be error and inconsistency. A common use of big data processing is to take unstructured data and extract ordered meaning, for consumption either by humans or as a structured input to an application. One such example is entity resolution, the process of determining exactly what a name refers to. Is this city London, England, or London, Texas? By the time your business logic gets to it, you don’t want to be guessing.
Traditional Data Handling:
Data Storage:
Big data ranges from several terabytes to petabytes and it’s difficult to read/write huge amount of data on a single disk. At these volumes access speed of the data devices will dominate overall analysis time. Have multiple disk drives, split your data file into small enough pieces across the drives and do parallel read and processing.
Hardware reliability (failure of any drive) is a challenge
Resolving data interdependency between drives is a notorious challenge.
Number of disk drives that can be added to a server is limited.
• Analysis on the stored data
Much of Big Data is unstructured. Traditional RDBMS/EDW cannot handle easily. Log of Big Data analysis is adhoc in nature, involves whole data scan, referencing itself, joining, combing etc. Traditional RDBMS/ EDW cannot handle these with their limited scalability options and architectural limitations. You can incorporate better servers, processors and throw in more RAM but there is a limit to it.
Need Some Drastically Different Approach
• A distributed file system with high capacity and high reliability to store large amount of data (like HDFS file system).
• A process engine that can handle structure / unstructured data.
• A computation model that can operate on distributed data and abstract data dispersion (MapReduce model)
Is Hadoop a Right Solution for Big Data?
•Hadoop is an open source Apache project for handling Big Data
•It addresses data storage issue and analysis (processing) issue through
•HDFS file system
•Implementing MapReduce computation model
•It is designed for massive scalability and reliability
•The model enables leveraging cheap commodity servers keeping the cost in check.
At a high-level, Hadoop architectural components can be classified into two categories
Distributed file management system – HDFS
• Name Node: Centrally Monitors and controls the whole file system
• Data Node: Take care of the local file segments & constantly communicates with Name Node
• Secondary Name Node: This just backs up the file system status from the Name Node periodically
Distributed computing system – MapReduce Framework
• Job Tracker:Centrally Monitors the submitted Job and controls all processes funning on the nodes (computers) of the cluster.
This communicated with Name Node for file system access
• Task Tracker:
Takes care of the local job execution on the local file segment.
Talks to Data Node for file information.
This constantly communicates with job Tracker daemon to report the task progress
In other words, Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.
A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.
The key aspect of the MapReduce algorithm is that if every Map and Reduce is independent of all other ongoing Maps and Reduces, then the operation can be run in parallel on different keys and lists of data. On a large cluster of machines, you can go one step further, and run the Map operations on servers where the data lives. Rather than copy the data over the network to the program, you push out the program to the machines. The output list can then be saved to the distributed filesystem, and the reducers run to merge the results. Again, it may be possible to run these in parallel, each reducing different keys.
Typically the compute nodes and the storage nodes are the same, that is, the MapReduce framework and the Hadoop Distributed File System are running on the same set of nodes. This configuration allows the framework to effectively schedule tasks on the nodes where data is already present, resulting in very high aggregate bandwidth across the cluster.
The MapReduce framework consists of a single master JobTracker and one slave TaskTracker per cluster-node. The master is responsible for scheduling the jobs’ component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.
Minimally, applications specify the input/output locations and supply map and reduce functions via implementations of appropriate interfaces and/or abstract-classes. These, and other job parameters, comprise the job configuration. The Hadoop job client then submits the job (jar/executable etc.) and configuration to the JobTrackerwhich then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.
The value of big data to an organization falls into two categories: analytical use, and enabling new products. Big data analytics can reveal insights hidden previously by data too costly to process, such as peer influence among customers, revealed by analyzing shoppers’ transactions, social and geographical data. Being able to process every item of data in reasonable time removes the troublesome need for sampling and promotes an investigative approach to data, in contrast to the somewhat static nature of running predetermined reports.
No comments:
Post a Comment