Sign up FAST! Login

Apache Spark is another increasingly popular alternative to replace MapReduce

There are different approaches in big data world to make Hadoop more suitable for ad-hoc, interactive queries and iterative data processing. As it is very well known, Hadoop MapReduce framework is primarily designed for batch processing and that makes it less suitable for ad-hoc data exploration, machine learning processes and the like. Big data vendors are trying to address this challenge by replacing MaReduce with alternatives. In case of SQL on Hadoop, there are various initiatives; Cloudera ImpalaPivotal HAWQ orHortonworks Stinger initiative that aims to improve Hive performance significantly.

apache spark

Apache Spark is another increasingly popular alternative to replace MapReduce with a more performant execution engine but still use Hadoop HDFS as storage engine for large data sets.

Spark ArchitectureFrom architecture perspective Apache Spark is based on two key concepts; Resilient Distributed Datasets (RDD) and directed acyclic graph (DAG) execution engine. With regards to datasets, Spark supports two types of RDDs: parallelized collections that are based on existing Scala collections and Hadoop datasets that are created from the files stored on HDFS. RDDs support two kinds of operations: transformations and actions. Transformations create new datasets from the input (e.g. map or filter operations are transformations), whereas actions return a value after executing calculations on the dataset (e.g. reduce or count operations are actions).The DAG engine helps to eliminate the MapReduce multi-stage execution model and offers significant performance improvements.

Apache Spark â a Fast Big Data Analytics Engine | BigHadoop

Stashed in: Big Data!, internet

To save this post, select a stash from drop-down menu or type in a new one:

So Hadoop is for ad hoc interactive queries and Spark is for structured regular queries?

You May Also Like: