Web crawling of KNOT@FIT
(Knowledge Technology Research Group)

As a part of our research, we regularly crawl the Internet using different tools like Bubing or Nutch. Although our policies are set to be very polite, we ask you to inform us in cases of any hard violations. At the same time, please treat our crawlers as any internet crawling robot and do not restrict it too much...

We continually enhance our configuration based on violations and experience so the crawling should be seamless over the time.

Bubing politeness settings:

• The minimum delay between two consecutive fetches from the same scheme+authority is 20 sec.
• The minimum delay between two consecutive fetches from the same IP address is 5 sec.
• How to stop Bubing?

Contacts

• Smrz Pavel, doc. RNDr., Ph.D. (research group leader) - smrz at fit.vutbr.cz
• Skoda Petr, RNDr. (responsible person for the crawling) - iskoda at fit.vutbr.cz
• Dytrych Jaroslav, Ing. (administrator of the servers of KNOT) - dytrych at fit.vutbr.cz