Docs Crawler Agent Skill
>- Crawl an entire documentation or design-system website into one LLM-ready Markdown corpus. Discovers every page from the site's sitemap.xml (with a same-origin link-following fallback when there is no sitemap), extracts each page's main content as clean Markdown, downloads each page's images (and inline base64 images) into a local folder referenced by relative paths, and renders JavaScript-heavy pages with a headless browser. Use this whenever someone wants the WHOLE of a multi-page docs site, design-system site, API reference, component library, or knowledge base turned into Markdown — for example "crawl this docs site", "turn this design system site into markdown", "make an LLM or RAG corpus from these docs", "mirror the entire documentation", "archive the whole knowledge base", or "give an AI the full X documentation". It applies even when the user never says the word "crawl" — any request to capture, ingest, archive, or convert a whole site's documentation pages into text fits. The design-md skill also calls it during research to gather a brand's published design-system docs. Do not use it for reading or summarizing a single page, for scraping structured data such as prices into a spreadsheet, or for generating a sitemap.
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/CaesiumY/ko-design-md --skill docs-crawlerOther skills for the same job
different authors, same section of the catalogue
Automate web scraping and data extraction with Apify -- run Actors, manage datasets, create reusable tasks, and retrieve crawl results through the Composio Apify integration.
Scrapes content based on a preset URL list, filters high-quality technical information, and generates daily Markdown reports.
Web search, content extraction, crawling, and research capabilities using Tavily API
| Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extract", or needs content from many pages on the same site. Handles depth limits, path filtering, and concurrent extraction.
Deep web scraping, screenshots, PDF parsing, and website crawling using Firecrawl API. Use when you need deep content extraction from web pages, page interaction is required (clicking, scrolling, etc.), or you want screenshots or PDF parsing.
Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.
Scrapes content based on a preset URL list, filters high-quality technical information, and generates daily Markdown reports.
Scrape, crawl, search, and extract structured data from websites using Firecrawl API - converts web pages to LLM-ready markdown
How to use it
Take caesiumy/docs-crawler from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.