org.apache.nutch.crawl

Class Generator.Selector

java.lang.Object

org.apache.nutch.crawl.Generator.Selector

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,CrawlDatum,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>, org.apache.hadoop.mapred.Partitioner<org.apache.hadoop.io.FloatWritable,org.apache.hadoop.io.Writable>, org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>

Enclosing class:

Generator

public static class Generator.Selector
extends Object
implements org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,CrawlDatum,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>, org.apache.hadoop.mapred.Partitioner<org.apache.hadoop.io.FloatWritable,org.apache.hadoop.io.Writable>, org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>

Selects entries due for fetch.

Constructor Summary

Constructors

Constructor and Description
Generator.Selector()

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	configure(org.apache.hadoop.mapred.JobConf job)
int	getPartition(org.apache.hadoop.io.FloatWritable key, org.apache.hadoop.io.Writable value, int numReduceTasks) Partition by host / domain or IP.
void	map(org.apache.hadoop.io.Text key, CrawlDatum value, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry> output, org.apache.hadoop.mapred.Reporter reporter) Select & invert subset due for fetch.
void	reduce(org.apache.hadoop.io.FloatWritable key, Iterator<Generator.SelectorEntry> values, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry> output, org.apache.hadoop.mapred.Reporter reporter) Collect until limit is reached.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Generator.Selector

public Generator.Selector()

Method Detail

configure

public void configure(org.apache.hadoop.mapred.JobConf job)

Specified by:

configure in interface org.apache.hadoop.mapred.JobConfigurable

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

map

public void map(org.apache.hadoop.io.Text key,
       CrawlDatum value,
       org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry> output,
       org.apache.hadoop.mapred.Reporter reporter)
         throws IOException

Select & invert subset due for fetch.

Specified by:

map in interface org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,CrawlDatum,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>

Throws:

IOException

getPartition

public int getPartition(org.apache.hadoop.io.FloatWritable key,
               org.apache.hadoop.io.Writable value,
               int numReduceTasks)

Partition by host / domain or IP.

Specified by:

getPartition in interface org.apache.hadoop.mapred.Partitioner<org.apache.hadoop.io.FloatWritable,org.apache.hadoop.io.Writable>

reduce

public void reduce(org.apache.hadoop.io.FloatWritable key,
          Iterator<Generator.SelectorEntry> values,
          org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry> output,
          org.apache.hadoop.mapred.Reporter reporter)
            throws IOException

Collect until limit is reached.

Specified by:

reduce in interface org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry,org.apache.hadoop.io.FloatWritable,Generator.SelectorEntry>

Throws:

IOException