org.apache.manifoldcf.crawler.connectors.webcrawler
Class RobotsManager.Record

java.lang.Object
  org.apache.manifoldcf.crawler.connectors.webcrawler.RobotsManager.Record

Enclosing class:

RobotsManager

protected static class RobotsManager.Record
extends java.lang.Object

This class represents a record in a robots.txt file. It contains one or more user-agents, and one or more disallows.

Field Summary

protected java.util.ArrayList	allows
protected java.util.ArrayList	disallows
protected java.util.ArrayList	userAgents

Constructor Summary

RobotsManager.Record()
Constructor.

Method Summary

void	addAgent(java.lang.String agentName) Add a user-agent.
void	addAllow(java.lang.String allowPath) Add an allow.
void	addDisallow(java.lang.String disallowPath) Add a disallow.
boolean	isAgentMatch(java.lang.String agentNameUpper, boolean exactMatch) See if user-agent matches.
boolean	isAllowed(java.lang.String path) See if path is allowed.
boolean	isDisallowed(java.lang.String path) See if path is disallowed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

userAgents

protected java.util.ArrayList userAgents

disallows

protected java.util.ArrayList disallows

allows

protected java.util.ArrayList allows

Constructor Detail

RobotsManager.Record

public RobotsManager.Record()

Constructor.

Method Detail

addAgent

public void addAgent(java.lang.String agentName)

Add a user-agent.

addDisallow

public void addDisallow(java.lang.String disallowPath)

Add a disallow.

addAllow

public void addAllow(java.lang.String allowPath)

Add an allow.

isAgentMatch

public boolean isAgentMatch(java.lang.String agentNameUpper,
                            boolean exactMatch)

See if user-agent matches.

isDisallowed

public boolean isDisallowed(java.lang.String path)

See if path is disallowed. Only called if user-agent has already matched. (This checks if there's an explicit match with one of the Disallows clauses.)

isAllowed

public boolean isAllowed(java.lang.String path)

See if path is allowed. Only called if user-agent has already matched. (This checks if there's an explicit match with one of the Allows clauses).