crawler.sh

crawler.sh is a desktop application and command-line tool for crawling websites. It systematically navigates through a site, discovering and analyzing every page it finds. The primary function is to extract the main content from each page, which it converts into clean, readable Markdown. This includes details like the word count, author byline, and a short excerpt. It also performs a comprehensive technical analysis of each page, checking for common SEO issues. The tool is designed to run on your own computer, giving you full control over the data and process.

The tool works by starting at a given URL and following links within the same domain. You can configure how deep it goes and how many pages it processes. A key feature is its set of 23 automated SEO checks, which scan for problems like missing page titles, duplicate meta descriptions, 'noindex' directives, thin content, and broken links. After the crawl, you can export all the gathered data in several useful formats, including JSON arrays for developers, NDJSON for streaming, and standard Sitemap XML files for search engines. SEO reports can be exported as CSV or plain text.

crawler.sh is most useful for professionals who need to understand and manage website structure and content. SEO specialists can use it to perform detailed site audits and identify issues that might affect search rankings. Content managers can archive website articles in a portable Markdown format for backups or migrations. Web developers will find it helpful for generating accurate sitemaps and monitoring sites for broken links or other technical problems that could impact user experience.