PHP class for parse all directives from robots.txt files according to specifications
-
Updated
May 1, 2025 - DIGITAL Command Language
#
robots-txt-parser
Here are 10 public repositories matching this topic...
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
-
Updated
Jun 23, 2023 - JavaScript
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
web-scraping robots-txt web-crawling hacktoberfest robots-exclusion-standard robots-exclusion-protocol robots-txt-parser
-
Updated
Jan 6, 2026 - PHP
Modern robots.txt analyzer with instant analysis, security recommendations, and export capabilities. Built with Qwik and deployed on Cloudflare Pages.
-
Updated
Apr 5, 2026 - TypeScript
⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.
-
Updated
Dec 2, 2024 - Dart
Google's robots.txt parser ported to TypeScript - RFC 9309 compliant
parser crawler seo geo robots-txt robots-parser robotstxt search-engine-optimization robots-txt-parser
-
Updated
Dec 8, 2025 - TypeScript
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
-
Updated
Jul 22, 2024 - Python
Simple robots txt parser for python
-
Updated
Mar 20, 2026 - Python
A web tool for validating URLs against robots.txt rules. Built with RFC 9309 compliant parsing.
parser crawler seo geo robots-txt robots-parser robotstxt search-engine-optimization robots-txt-parser
-
Updated
Dec 8, 2025 - TypeScript
Improve this page
Add a description, image, and links to the robots-txt-parser topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the robots-txt-parser topic, visit your repo's landing page and select "manage topics."