org.apache.nutch.protocol.http.api
Class RobotRulesParser.RobotRuleSet

java.lang.Object
  org.apache.nutch.protocol.http.api.RobotRulesParser.RobotRuleSet

Enclosing class:

RobotRulesParser

public static class RobotRulesParser.RobotRuleSet

extends Object

This class holds the rules which were parsed from a robots.txt file, and can test paths against those rules.

Constructor Summary

RobotRulesParser.RobotRuleSet()

Method Summary

long	getCrawlDelay() Get Crawl-Delay, in milliseconds.
long	getExpireTime() Get expire time
boolean	isAllowed(String path) Returns false if the robots.txt file prohibits us from accessing the given path, or true otherwise.
void	setCrawlDelay(long crawlDelay) Set Crawl-Delay, in milliseconds
void	setExpireTime(long expireTime) Change when the ruleset goes stale.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

RobotRulesParser.RobotRuleSet

public RobotRulesParser.RobotRuleSet()

Method Detail

setExpireTime

public void setExpireTime(long expireTime)

Change when the ruleset goes stale.

getExpireTime

public long getExpireTime()

Get expire time

getCrawlDelay

public long getCrawlDelay()

Get Crawl-Delay, in milliseconds. This returns -1 if not set.

setCrawlDelay

public void setCrawlDelay(long crawlDelay)

Set Crawl-Delay, in milliseconds

isAllowed

public boolean isAllowed(String path)

Returns false if the robots.txt file prohibits us from accessing the given path, or true otherwise.

toString

public String toString()