spiderbar

Parse and Test Robots Exclusion Protocol Files and Rules

CRAN Package

The 'Robots Exclusion Protocol' documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' C++ library for processing these 'robots.txt' files.

Documentation

Team

Insights

Last 30 days

Last 365 days

The following line graph shows the downloads per day. You can hover over the graph to see the exact number of downloads per day.

Data provided by CRAN

Binaries

Dependencies

  • Depends1 package
  • Imports1 package
  • Suggests3 packages
  • Linking To1 package
  • Reverse Imports1 package