crawler.sh

Rapidly scan entire websites in seconds with adjustable speed settings, depth controls, and respectful pauses to maintain server courtesy. Pull out core page content into pristine Markdown format, complete with word counts, author details, and summaries for seamless processing. Deliver results via streaming NDJSON for real-time pipelines, structured JSON, standards-compliant Sitemap XML for SEO boosts, or bundled Markdown files.