Harvest Deep Crawl Skill for Claude
Multi-page deep crawling - documentation sites, wikis, knowledge bases
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/vibeeval/vibecosystem --skill harvest-deep-crawlWhat it tells the agent to use
found in the instruction textThe instruction itself
11 sections, as written by the authorHarvest Deep Crawl
Crawl multi-page websites following internal links to a specified depth. Ideal for building complete knowledge bases from documentation sites, wikis, and reference materials.
Usage
/crawl <url> --depth <N>
Examples
# Crawl docs site 3 levels deep
/crawl https://docs.example.com --depth 3
# Crawl a specific section
/crawl https://docs.example.com/api --depth 2
# Crawl with page limit
/crawl https://wiki.example.com --depth 5 --max-pages 50
Parameters
| Param | Default | Description |
|-------|---------|-------------|
| --depth | 2 | Max link-following depth |
| --max-pages | 100 | Max pages to crawl |
| --same-domain | true | Stay on same domain |
| --include | * | URL pattern to include |
| --exclude | - | URL pattern to exclude |
How It Works
- Start at root URL, extract all internal links
- Follow links up to specified depth (BFS order)
- Extract content from each page
- Deduplicate pages with > 90% content overlap
- Build table of contents from page hierarchy
- Merge into coherent knowledge base
- Save to
.claude/cache/agents/harvest/crawl-{domain}/
Output Structure
crawl-{domain}-{timestamp}/
index.md # Table of contents + summary
page-001.md # First page content
page-002.md # Second page content
...
metadata.json # Crawl stats, URLs, timings
Crawl Engine
Primary: crawl4ai (Docker port 11235)
curl -s http://localhost:11235/crawl \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://docs.example.com"],
"max_depth": 3,
"same_domain": true,
"word_count_threshold": 50
}'
Fallback: Manual Link Following
When Docker unavailable:
- WebFetch root URL
- Parse links from markdown output
- WebFetch each linked page (depth-limited)
- Compile results
Use Cases
| Scenario | Depth | Max Pages |
|----------|-------|-----------|
| API reference | 2-3 | 50 |
| Full documentation site | 3-5 | 100 |
| Wiki section | 2 | 30 |
| Changelog history | 1-2 | 20 |
| Tutorial series | 2-3 | 30 |
Rules
- Respect robots.txt
- Max 2 requests/second
- Skip binary files (PDF, images, videos)
- Detect and skip infinite pagination
- Cache results for 24 hours
Other skills for the same job
different authors, same section of the catalogue
Automate web scraping and data extraction with Apify -- run Actors, manage datasets, create reusable tasks, and retrieve crawl results through the Composio Apify integration.
Scrapes content based on a preset URL list, filters high-quality technical information, and generates daily Markdown reports.
Web search, content extraction, crawling, and research capabilities using Tavily API
| Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extract", or needs content from many pages on the same site. Handles depth limits, path filtering, and concurrent extraction.
Deep web scraping, screenshots, PDF parsing, and website crawling using Firecrawl API. Use when you need deep content extraction from web pages, page interaction is required (clicking, scrolling, etc.), or you want screenshots or PDF parsing.
Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.
Scrapes content based on a preset URL list, filters high-quality technical information, and generates daily Markdown reports.
Scrape, crawl, search, and extract structured data from websites using Firecrawl API - converts web pages to LLM-ready markdown
How to use it
Take vibeeval/harvest-deep-crawl from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.