org.apache.hadoop.tools.mapred.lib
Class DynamicRecordReader<K,V>

java.lang.Object
  org.apache.hadoop.mapreduce.RecordReader<K,V>
      org.apache.hadoop.tools.mapred.lib.DynamicRecordReader<K,V>

All Implemented Interfaces:

Closeable

public class DynamicRecordReader<K,V>
extends org.apache.hadoop.mapreduce.RecordReader<K,V>

The DynamicRecordReader is used in conjunction with the DynamicInputFormat to implement the "Worker pattern" for DistCp. The DynamicRecordReader is responsible for: 1. Presenting the contents of each chunk to DistCp's mapper. 2. Acquiring a new chunk when the current chunk has been completely consumed, transparently.

Constructor Summary

DynamicRecordReader(int numEntriesPerChunk)

Method Summary

void	close() Implementation of RecordReader::close().
K	getCurrentKey() Implementation of RecordReader::getCurrentKey().
V	getCurrentValue() Implementation of RecordReader::getCurrentValue().
float	getProgress() Implementation of RecordReader::getProgress().
void	initialize(org.apache.hadoop.mapreduce.InputSplit inputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext) Implementation for RecordReader::initialize().
boolean	nextKeyValue() Implementation of RecordReader::nextValue().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DynamicRecordReader

public DynamicRecordReader(int numEntriesPerChunk)

Method Detail

initialize

public void initialize(org.apache.hadoop.mapreduce.InputSplit inputSplit,
                       org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
                throws IOException,
                       InterruptedException

Implementation for RecordReader::initialize(). Initializes the internal RecordReader to read from chunks.

Specified by:

initialize in class org.apache.hadoop.mapreduce.RecordReader<K,V>

Parameters:

inputSplit: - The InputSplit for the map. Ignored entirely.

taskAttemptContext: - The AttemptContext.

Throws:

IOException, - on failure.

InterruptedException

IOException

nextKeyValue

public boolean nextKeyValue()
                     throws IOException,
                            InterruptedException

Implementation of RecordReader::nextValue(). Reads the contents of the current chunk and returns them. When a chunk has been completely exhausted, an new chunk is acquired and read, transparently.

Specified by:

nextKeyValue in class org.apache.hadoop.mapreduce.RecordReader<K,V>

Returns:

True, if the nextValue() could be traversed to. False, otherwise.

Throws:

IOException, - on failure.

InterruptedException

IOException

getCurrentKey

public K getCurrentKey()
                throws IOException,
                       InterruptedException

Implementation of RecordReader::getCurrentKey().

Specified by:

getCurrentKey in class org.apache.hadoop.mapreduce.RecordReader<K,V>

Returns:

The key of the current record. (i.e. the source-path.)

Throws:

IOException, - on failure.

InterruptedException

IOException

getCurrentValue

public V getCurrentValue()
                  throws IOException,
                         InterruptedException

Implementation of RecordReader::getCurrentValue().

Specified by:

getCurrentValue in class org.apache.hadoop.mapreduce.RecordReader<K,V>

Returns:

The value of the current record. (i.e. the target-path.)

Throws:

IOException, - on failure.

InterruptedException

IOException

getProgress

public float getProgress()
                  throws IOException,
                         InterruptedException

Implementation of RecordReader::getProgress().

Specified by:

getProgress in class org.apache.hadoop.mapreduce.RecordReader<K,V>

Returns:

A fraction [0.0,1.0] indicating the progress of a DistCp mapper.

Throws:

IOException, - on failure.

InterruptedException

IOException

close

public void close()
           throws IOException

Implementation of RecordReader::close(). Closes the RecordReader.

Specified by:

close in interface Closeable

Specified by:

close in class org.apache.hadoop.mapreduce.RecordReader<K,V>

Throws:

IOException, - on failure.

IOException