One of Hadoop’s limitations is that by distributing jobs across several nodes, there’s a possibility that a few sluggish nodes can slow down the remainder of the application. There are a variety of reasons why jobs are sluggish, some of which are difficult to notice. Rather than detecting and resolving slow-running jobs, Hadoop seeks to recognize when a task is taking longer than planned and then starts backup tasks. Speculative execution is Hadoop’s backup mechanism.
It duplicates the task on a different disc. The same input can be processed in parallel numerous times. When the majority of a job’s tasks have been completed, the speculative execution mechanism arranges duplicate copies of the remaining (slower) jobs among the nodes that are now free. The JobTracker gets notified when these tasks are completed. Hadoop informs the TaskTrackers to stop those tasks and refuse their output if other copies are operating speculatively.
In Hadoop, speculative execution is enabled by default. Set the JobConf options mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution to false to deactivate it.