Monday, 4 January 2016

Binary compatibility of org.apache.hadoop.mapred APIs

Binary compatibility here means that the compiled binaries should be able to run without any modification on the new framework.

For those Hadoop 1.x users who use the  org.apache.hadoop.mapred  APIs, they can simply run their MapReduce jobs on YARN just by pointing them to their Apache Hadoop 2.x cluster via the configuration settings.

They will not need any recompilation. All they will need to do is point their application to the YARN installation and point  HADOOP_CONF_DIR  to the corresponding configuration directory. The  yarn-site.xml  (configuration for YARN) and  mapred-site.xml  files (configuration for MapReduce apps) are present in the conf directory.

Also,  mapred.job.tracker  in  mapred-site.xml  is no longer necessary in Apache Hadoop 2.x. Instead, the following property needs to be added in the  mapred-site.xml  file to make MRv1 applications run on top of YARN:

Related Posts Plugin for WordPress, Blogger...