Prior to Hadoop 0.20.x, a Map class had to extend a MapReduceBase and implement a Mapper as such:
public static class Map extends MapReduceBase implements Mapper {
...
}
and similarly, a map function had to use an OutputCollector and a Reporter object to emit (key,value) pairs and send progress updates to the main program. A typical map function looked like:
public void map(K1, V1, OutputCollector o, Reporter r) throws IOException {
...
output.collect(key,value);
}
With the new Hadoop API, a mapper or reducer has to extend classes from the package org.apache.hadoop.mapreduce.* and there is no need to implement an interface anymore. Here is how a Map class is defined in the new API:
public class MapClass extends Mapper {
...
}
and a map function uses Context objects to emit records and send progress updates. A typical map function is now defined as:
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
...
context.write(key,value);
}
All of the changes for a Mapper above go the same way for a Reducer.
Another major change has been done in the way a job is configured and controlled. Earlier, a map reduce job was configured through a JobConf object and the job control was done using an instance of JobClient. The main body of a driver class used to look like:
JobConf conf = new JobConf(Driver.class);
conf.setPropertyX(..);
conf.setPropertyY(..);
...
...
...
JobClient.runJob(conf);
In the new Hadoop API, the same functionality is achieved as follows:
Configuration conf = new Configuration();
Job job = new Job(conf);
job.setJarByClass(Driver.class);
job.setPropertyX(..);
job.setPropertyY(..);
...
...
...
job.waitForCompletion(true);
The current Map Reduce tutorial on the official Hadoop page is written for the old API, so I uploaded a working word count example that uses the new methods and constructs here. Please let me know if you have any comments or suggestions.
