A brief on Mapreduce

Data Science, AI and ML
#1

MapReduce

MapReduce is a programming model that is associated with the implementation of processing and generating big data sets with the help of parallel, distributed algorithmic rules on a cluster.

A MapReduce program consists of a map procedure, that performs filtering and sorting, and a reduce technique, that performs an outline operation.

mapreduce
mapreduce1948×960 79.8 KB

Image source: by me

  • MapReduce could be a data processing framework to process data on the cluster.
  • Two consecutive phases: Map and reduce.
  • Each map task operates on separate parts of data.
  • After the map, the reducer works on the data generated by the mapper on distributed data nodes.
  • MapReduce used Disk I/O to perform operations on data.

map reduce disk io
map reduce disk io1948×960 59.1 KB

Image source: by me

Apache Spark

Apache Spark is an open-source data analytics engine for large-scale processing of structure or unstructured data. To work with the Python including the Spark functionalities, the Apache Spark community had released a tool called PySpark.

The Spark Python API (PySpark) discloses the Spark programming model to Python. Using the PySpark, we can work with RDDs in the Python programming language. It’s attributable to a library referred to as the Py4j that they’re able to reach this.

hadoop | apche spark
hadoop | apche spark753×323 15.1 KB

Advantages of Apache spark:

  • The Spark is 10 to 100 times faster than the Hadoop MapReduce when talking about data processing.
  • It has simple data processing framework and Interactive APIs for Python that helps in faster application development.
  • Also, It is more efficient as it has multiple tools for complex analytics operations.
  • It can be easily integrated with the existing Hadoop infrastructure.