Web Fetch Skill for Claude
Fetches web content as clean markdown by preferring markdown-native responses and falling back to selector-based HTML extraction. Use for documentation, articles, and reference pages at http/https URLs.
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/aiskillstore/marketplace --skill web-fetchWhat comes with it
50 759 bytes besides the instructionThe instruction itself
8 sections, as written by the authorWeb Content Fetching
Fetch web content in this order:
- Prefer markdown-native endpoints (
content-type: text/markdown) - Use selector-based HTML extraction for known sites
- Use the bundled Bun fallback script when selectors fail
Prerequisites
Verify required tools before extracting:
command -v curl >/dev/null || echo "curl is required"
command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction"
command -v bun >/dev/null || echo "bun is required for fetch.ts fallback"
Install Bun dependencies for the bundled script:
cd ~/.claude/skills/web-fetch && bun install
Default Workflow
Use this as the default flow for any URL:
URL="<url>"
CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)"
if echo "$CONTENT_TYPE" | grep -q "markdown"; then
curl -sL "$URL"
else
curl -sL "$URL" \
| html2markdown \
--include-selector "article,main,[role=main]" \
--exclude-selector "nav,header,footer,script,style"
fi
Known Site Selectors
| Site | Include Selector | Exclude Selector |
|------|------------------|------------------|
| platform.claude.com | #content-container | - |
| docs.anthropic.com | #content-container | - |
| developer.mozilla.org | article | - |
| github.com (docs) | article | nav,.sidebar |
| Generic | article,main,[role=main] | nav,header,footer,script,style |
Example:
curl -sL "<url>" \
| html2markdown \
--include-selector "#content-container" \
--exclude-selector "nav,header,footer"
Finding the Right Selector
When a site isn't in the patterns list:
# Check what content containers exist
curl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -10
# Test a selector
curl -sL "<url>" | html2markdown --include-selector "<selector>" | head -30
# Check line count
curl -sL "<url>" | html2markdown --include-selector "<selector>" | wc -l
Universal Fallback Script
When selectors produce poor output, run the bundled parser:
bun ~/.claude/skills/web-fetch/fetch.ts "<url>"
If already in the skill directory:
bun fetch.ts "<url>"
Options Reference
--include-selector "CSS" # Keep only matching elements
--exclude-selector "CSS" # Remove matching elements
--domain "https://..." # Convert relative links to absolute
Troubleshooting
Empty output with selectors: The page might be markdown-native. Check headers first:
curl -sIL "<url>" | grep -i '^content-type:'
Wrong content selected: The site may have multiple article/main regions:
curl -s "<url>" | grep -o '<article[^>]*>'
html2markdown not found: Install it, then retry selector-based extraction.
bun or script deps missing: Run cd ~/.claude/skills/web-fetch && bun install.
Missing code blocks: Check if the site uses non-standard code formatting.
Client-rendered content: If HTML only has "Loading..." placeholders, the content is JS-rendered. Neither curl nor the Bun script can extract it; use browser-based tools.
Other skills for the same job
different authors, same section of the catalogue
Integration with protocols.io API for managing scientific protocols. This skill should be used when working with protocols.io to search, create, update, or publish protocols; manage protocol steps and materials; handle discussions and comments; organize workspaces; upload and manage files; or integrate protocols.io functionality into workflows. Applicable for protocol discovery, collaborative protocol development, experiment tracking, lab protocol management, and scientific documentation.
Analyzes job descriptions and generates tailored resumes that highlight relevant experience, skills, and achievements to maximize interview chances
Generate Excalidraw diagrams from natural language descriptions. Use when asked to "create a diagram", "make a flowchart", "visualize a process", "draw a system architecture", "create a mind map", or "generate an Excalidraw file". Supports flowcharts, relationship diagrams, mind maps, and system architecture diagrams. Outputs .excalidraw JSON files that can be opened directly in Excalidraw.
Build and distribute Expo development clients locally or via TestFlight
Use when you have a written implementation plan to execute in a separate session with review checkpoints
Data structure for annotated matrices in single-cell analysis. Use when working with .h5ad files or integrating with the scverse ecosystem. This is the data format skill—for analysis workflows use scanpy; for probabilistic models use scvi-tools; for population-scale queries use cellxgene-census.
Benchling R&D platform integration. Access registry (DNA, proteins), inventory, ELN entries, workflows via API, build Benchling Apps, query Data Warehouse, for lab data management automation.
Comprehensive molecular biology toolkit. Use for sequence manipulation, file parsing (FASTA/GenBank/PDB), phylogenetics, and programmatic NCBI/PubMed access (Bio.Entrez). Best for batch processing, custom bioinformatics pipelines, BLAST automation. For quick lookups use gget; for multi-service integration use bioservices.
How to use it
Take aiskillstore/web-fetch from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.