Fetch URL As Markdown Skill for Claude

Fetch a web page (URL) and return clean Markdown via local trafilatura, with Exa MCP as a fallback for JS-rendered or anti-bot pages. Use when the user asks to read, fetch, scrape, summarize, or quote a URL — prefer this over the built-in WebFetch tool. Don't use for binary files (PDFs, images, archives) or for fetching API/JSON endpoints.

5k tokens

context cost

the whole folder, loaded on every use

files

ships runnable scripts

copies elsewhere

how many repositories repackaged it

115

stars on the repo

on the repository, not the skill itself

Install

one command, takes just this skill from the repository

npx skills add https://github.com/CodeAlive-AI/ai-driven-development --skill fetch-url-as-markdown

What comes with it

15 486 bytes besides the instruction

README.md

scripts/fetch_url.py

scripts/settings.cfg

What it tells the agent to use

found in the instruction text

WebFetch fetches pages from the network

The instruction itself

6 sections, as written by the author

URL to Markdown

Fetch any web URL and get clean, readable Markdown — main content only, no

navigation/footer/ads. Local + free by default; smart fallback to Exa MCP

when the page can't be extracted locally.

Workflow (the only thing the agent needs to remember)

Try trafilatura first:

   python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "<URL>"

If exit code is 1 or 2 → fall back to Exa MCP with the same URL:

   mcp__exa__web_search_advanced_exa(
       query="<URL>",
       includeDomains=["<host of URL>"],
       numResults=1,
       textMaxCharacters=50000,
       type="auto"
   )

(mcp__exa__crawling works too if the server exposes it; the web_search_advanced_exa

call above is the always-available variant — pin the host with includeDomains and

use the URL itself as the query.)

Exit code 3 means trafilatura is not installed — install once:

   python3 -m pip install --break-system-packages trafilatura

Exit codes (what they mean for the fallback decision)

| Code | Meaning | Action |

|---|---|---|

| 0 | Markdown printed to stdout | done |

| 1 | DownloadError — network/HTTP/timeout/anti-bot block at fetch | fall back to Exa |

| 2 | ExtractionError — empty extract, JS/Cloudflare wall, or stub body (<200 chars) | fall back to Exa |

| 3 | trafilatura missing | install (see above), then retry |

| 4 | UnsupportedContentTypeError — URL is binary (PDF, image, archive) | don't fall back to Exa; use the right specialized skill (e.g. pdf for PDFs) |

Defaults baked into the script

output_format="markdown", include_formatting=True — keeps headings/lists/code structure where the source HTML uses real <h1..h6> etc.
include_links=True, include_tables=True
with_metadata=True → emits a YAML frontmatter (title, author, date, url, hostname)
favor_recall=True, deduplicate=True — readable but trims duplicates
Real-browser User-Agent + 30s timeout configured in scripts/settings.cfg
Anti-stub guards (built into the script):
rejects Content-Type other than text/html|application/xhtml+xml|text/plain|application/xml|text/xml → exit 4
sniffs raw HTML for Cloudflare / "Please enable JavaScript" / Imperva / DataDome wall markers → exit 2
rejects extracted bodies under 50 chars (configurable via --min-body N, 0 to disable) → exit 2

Useful flags

... fetch_url.py "<URL>" --no-links     # strip hyperlinks
... fetch_url.py "<URL>" --no-tables    # strip tables
... fetch_url.py "<URL>" --no-metadata  # omit YAML header
... fetch_url.py "<URL>" --comments     # include user comments (off by default — usually noise)
... fetch_url.py "<URL>" --images       # include image refs (experimental)
... fetch_url.py "<URL>" --precision    # terser output, drops borderline content

When to choose what

| Situation | Tool |

|---|---|

| Article, blog post, docs, README, wiki | trafilatura (default) — local, free |

| JS-heavy SPA, login-walled, Cloudflare | Exa fallback (the script will signal exit 2) |

| Bulk / many URLs | trafilatura — no quota, no API key |

| Already failed twice on a domain | Exa directly |

Other skills for the same job

different authors, same section of the catalogue

Firecrawl Parse

by firecrawl

vendor ×1

| Efficiently extract and convert the contents of any local file—such as PDF, DOCX, DOC, ODT, RTF, XLSX, XLS, or HTML—into clean, well-formatted markdown saved to disk. Use this skill whenever the user requests to parse, read, or extract information from a file on their computer, including phrases like “parse this PDF”, “convert this document”, “read this file”, “extract text from”, or when a local file path (not a URL) is provided. This skill offers advanced options like generating AI-powered summaries and answering questions based on the file's content. Prefer this tool over `scrape` when handling local files to deliver precise, structured outputs for downstream tasks.

685 tokens

Latchshot Page Capture

by github

vendor

Use this skill when a user needs a screenshot, website thumbnail, full-page capture, or PDF of a public HTTP(S) webpage saved as a local artifact through Latchshot, including report, QA, archive, and social-preview workflows. Do not use it for private or authenticated pages, raw HTML, scraping or extraction, arbitrary browser actions, CAPTCHA or anti-bot bypass, or local-file capture.

6k tokens scripts

Feishu Doc Scraper

by daymade

Extract Feishu (Lark) Docs, Wiki pages/collections, spreadsheets, and Minutes (妙记) transcripts into faithful local Markdown via the lark-cli API (no LLM rewriting of the body; browser-DOM fallback when lark-cli can't reach the content). Use whenever the source is a Feishu/Lark URL and fidelity matters — 导出飞书文档/合集/妙记转写, 把飞书 wiki/知识库转 markdown, archiving a Feishu collection, exporting a 妙记 transcript, or saving a Feishu page — even if the user only says clipping, archiving, converting, or "save this". Also covers the owner-exported .docx → faithful Markdown path.

30k tokens scripts

Kb Refresh

by techwolf-ai

| Add new sources to your knowledge base or re-scrape existing ones to pick up changes. Supports Notion, Slack, Confluence, and local files. Can be run anytime after /setup-knowledge-base.

1k tokens

Tapestry

by NatsuFox

AI-native web intelligence workflow for crawling, organizing, and synthesizing web content from multiple platforms (Zhihu, Reddit, HN, X/Twitter, Xiaohongshu, Weibo). Use when users share URLs, want to archive web content, build knowledge bases, or analyze online discussions.

Comprehensive document creation, editing, and analysis with support for tracked changes, comments, formatting preservation, and text extraction. When Claude needs to work with professional documents (.docx files) for: (1) Creating new documents, (2) Modifying or editing content, (3) Working with tracked changes, (4) Adding comments, or any other document tasks

Comprehensive PDF manipulation toolkit for extracting text and tables, creating new PDFs, merging/splitting documents, and handling forms. When Claude needs to fill in a PDF form or programmatically process, generate, or analyze PDF documents at scale.

Presentation creation, editing, and analysis. When Claude needs to work with presentations (.pptx files) for: (1) Creating new presentations, (2) Modifying or editing content, (3) Working with layouts, (4) Adding comments or speaker notes, or any other presentation tasks

308k tokens scripts

How to use it

Copy the folder

Take codealive-ai/fetch-url-as-markdown from the repository into ~/.claude/skills for personal use, or into .claude/skills inside a project.

Check the name does not clash

The agent identifies a skill by the name field in its header. Two skills with the same name cannot sit side by side — one of them will be ignored.

Install what it needs

The instructions reference pip. Without those the skill loads but fails at the first command.