Scrapling Skill for Codex
CLI-first web scraping & content extraction with optional MCP server. Use when you have target URLs and need clean, selector-based outputs (html/md/txt).
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/foryourhealth111-pixel/Vibe-Skills --skill scraplingWhat comes with it
1 142 bytes besides the instructionThe instruction itself
16 sections, as written by the authorScrapling Skill (VCO)
Scrapling is a Python-based web scraping / extraction toolkit that exposes:
- a CLI (
scrapling ...) for fetching + extracting content into files - an optional MCP server (
scrapling mcp) so an agent can call structured scraping tools
This skill is CLI-first. Prefer it when you already have URLs and need reliable, repeatable extraction (CSS selector → file).
When to use
Use scrapling when you need:
- Extract specific parts of a web page (CSS selector / XPath) into
.txt/.md/.html - Run repeatable scraping jobs (batch URLs with a small wrapper script)
- Reduce token usage by extracting only the relevant DOM region before passing to the LLM
- Provide a local MCP endpoint for scraping tools (agent → MCP → scrapling)
Boundaries (vs Playwright / Search)
vs playwright
scrapling: best for “get URL → extract selector → write file” workflows; simpler, faster iterationplaywright: best for interactive UI flows (login, multi-step navigation, downloads, complex JS actions, stateful sessions)
If you must *navigate* or *click through* a UI, use playwright.
If you can directly fetch the target page and just need extraction, use scrapling.
vs search tools
- Search tools are for discovering sources/URLs (query → result list → choose URLs).
scraplingis for acquisition + extraction once you already know the URL(s).
A common pipeline:
1) Search → find candidate URLs
2) Scrapling → extract focused content from chosen URLs
3) LLM → summarize / transform / analyze extracted outputs
Prerequisite check (required)
1) Python version (Scrapling requires Python >= 3.10):
python --version
2) Scrapling CLI availability:
scrapling --help
Installation (recommended)
Scrapling’s CLI and MCP features are enabled via extras.
Recommended (CLI + MCP + fetchers):
python -m pip install "scrapling[ai]"
If you only want CLI fetch/extract without MCP:
python -m pip install "scrapling[fetchers]"
If you use browser-based fetchers, you may need browser binaries:
# Option A: via Scrapling helper (after install)
scrapling install
# Option B: directly via Playwright
python -m playwright install
Wrapper script (Windows convenience)
This skill ships a thin PowerShell wrapper:
C:/Users/羽裳/.codex/skills/scrapling/scripts/scrapling.ps1
It checks whether scrapling exists and prints install hints if missing.
Common CLI patterns
1) Extract full page body (to Markdown)
scrapling extract get "https://example.com" out.md
2) Extract a specific element (CSS selector) to text
scrapling extract get "https://example.com" out.txt --css-selector "main article"
3) Extract HTML for downstream parsing
scrapling extract get "https://example.com" out.html --css-selector "#content"
4) Use browser-backed fetcher mode (when simple GET is blocked / dynamic)
scrapling extract fetch "https://example.com" out.md --css-selector "main"
Tip: keep outputs in files and only feed the smallest relevant snippet to the LLM.
MCP server relationship (optional)
Scrapling can run as an MCP server. This is useful when:
- the agent needs tool-style scraping calls
- you want scraping results to be structured and deterministic
Start MCP server (stdio transport by default):
scrapling mcp
Optional: run MCP server with HTTP transport:
scrapling mcp --http --host 127.0.0.1 --port 8765
Example MCP server config snippet
{
"servers": {
"scrapling": {
"mode": "stdio",
"command": "scrapling",
"args": ["mcp"],
"required": false,
"note": "Requires: python -m pip install \"scrapling[ai]\""
}
}
}
Safety & ops notes
- Prefer selector-based extraction to minimize data volume.
- Treat scraping as an external dependency: handle timeouts, retries, and failures explicitly.
- For aggressive bot protection, consider switching fetchers or using
playwright.
Other skills for the same job
different authors, same section of the catalogue
Discover and extract sitemaps from any website using SitemapKit. Use this skill whenever the user wants to find pages on a website, get a list of URLs from a domain, audit a site's structure, crawl a sitemap, check what pages exist on a site, or do anything involving sitemaps or site URL discovery — even if they don't explicitly say "sitemap". Requires the sitemapkit MCP server configured with a valid SITEMAPKIT_API_KEY.
This skill helps users automatically extract Amazon product reviews via the Amazon Reviews API. Agent should proactively apply this skill when users express needs like getting reviews for Amazon product with ASIN B07TS6R1SF, analyzing customer feedback for a specific Amazon item, getting ratings and comments for a competitive product, tracking sentiment of recent Amazon reviews, extracting verified purchase reviews for quality assessment, summarizing user experiences from Amazon product pages, monitoring product performance through customer reviews, collecting reviewer profiles and links for market research, gathering review titles and descriptions for content analysis, scraping Amazon reviews without requiring a login.
Use this skill when the user wants to retrieve a URL without a full browser session: fetch HTML or JSON from static pages, inspect status codes or headers, follow redirects, or get page source for simple scraping. Prefer it over a browser when JavaScript rendering and page interaction are not needed. Supports proxies and redirect control.
> Use this skill when the user is doing hands-on DOCA Telemetry Exporter programming on a host where DOCA is installed — defining a doca_telemetry_exporter_schema, creating sources, picking counter/gauge/event types, running capability queries before assuming limits, registering schemas before the first emit, or debugging DOCA_ERROR_* failures from the exporter API. Trigger even when the user does not explicitly mention "DOCA Telemetry Exporter" or "doca_telemetry_exporter_*" — typical implicit phrasings include "publishing counters from my DOCA app", "emit returns AGAIN under bulk load", "consumer sees nothing but emit reports success", "NOT_FOUND on first submit", or "should I link the exporter or the telemetry service". Refuse and route elsewhere for the receiving DOCA Telemetry Service, plain stdout logging via doca_log, non-DOCA Prometheus scrape sinks, or real-time event subscription back into the app via doca-comch — those belong to other skills.
End-to-end pipeline for scraping X/Twitter for GTM/DevRel tech startup jobs using Apify, or optionally TweetClaw for OpenClaw and Hermes Tweet for Hermes Agent, then validating them against an Ideal Customer Profile (ICP) using Gemini's native Google Search Grounding. Use this skill when OpenClaw needs to find developer-first, funded startups actively hiring for GTM, DevRel, or Growth roles.
X (Twitter) data platform skill — tweet search, user lookup, follower extraction, engagement metrics, giveaway draws, monitoring, webhooks, 19 extraction tools, MCP server.
Skills for web search and content scraping via DuckDuckGo MCP Server. Used when users need online searching and web scraping.
X (Twitter) data platform skill — tweet search, user lookup, follower extraction, engagement metrics, giveaway draws, monitoring, webhooks, 19 extraction tools, MCP server.
How to use it
Take foryourhealth111-pixel/scrapling from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.
The instructions reference pip.
Without those the skill loads but fails at the first command.