Web Scraping Skills

Multi-engine web search (SearXNG) + browsing/scraping (Camofox, CloakBrowser). Use whenever you need to do web research.

> This skill should be used when performing web research beyond a simple single search -- looking into topics, comparing options, investigating questions, finding recommendations, or any task where effective use of Exa, Firecrawl, and Reddit tools matters. Triggers on "research", "look into", "investigate", "compare", "find out about", "search for", "find information", "what do people think about", "what are the best", "look up", or multi-source search tasks. Also invocable explicitly by deep-research team members via the Skill tool.

Use the ax CLI instead of curl + throwaway parsing scripts whenever you fetch a URL, explore an unknown web page, or extract structured data from HTML. Trigger whenever you are about to write an inline script (python3 heredoc, node -e, regex over HTML) or a bare curl for one-off web fetching, scraping, or page exploration.

| Kimi WebBridge lets AI control the user's real browser — navigate, click, type, read, screenshot, and interact with any website using the user's actual login sessions. Use this skill whenever the user wants to interact with websites, automate browser tasks, scrape web content, or perform any action requiring a real browser. Also use when the user mentions "browser", "webpage", "open URL", "screenshot", or asks to read/interact with any website. Use even for simple-sounding browser requests — the daemon handles all complexity.

Expert blueprint for roguelikes including procedural generation (Walker method, BSP rooms), permadeath with meta-progression (unlock persistence), run state vs meta state separation, seeded RNG (shareable runs), loot/relic systems (hook-based modifiers), and difficulty scaling (floor-based progression). Use for dungeon crawlers, action roguelikes, or roguelites. Trigger keywords: roguelike, procedural_generation, permadeath, meta_progression, seeded_RNG, relic_system, run_state.

> Information architecture, URL hierarchy, internal linking strategy, navigation design, and silo structure for websites. Covers architectural audits, new site planning, crawl equity optimization, and topic cluster mapping.

| Discover and list all URLs on a website without extracting content, via the Tavily CLI. Use this skill when the user wants to find a specific page on a large site, list all URLs, see the site structure, find where something is on a domain, or says "map the site", "find the URL for", "what pages are on", "list all pages", or "site structure". Faster than crawling — returns URLs only. Essential when you know the site but not the exact page. Combine with extract for targeted content retrieval.

Build production-ready Tavily integrations with best practices baked in. Reference documentation for developers using coding assistants (Claude Code, Cursor, etc.) to implement web search, content extraction, crawling, and research in agentic workflows, RAG systems, or autonomous agents.

| Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download documentation, extract an entire docs section, bulk-extract pages, save a site as local markdown files, or says "crawl", "get all the pages", "download the docs", "extract everything under /docs", "bulk extract", or needs content from many pages on the same domain. Supports depth/breadth control, path filtering, semantic instructions, and saving each page as a local markdown file.

Create Firecrawl monitors that watch pages, sites, or web search results and send change alerts by email or webhook.

| Extract clean markdown from any URL, including JavaScript-rendered SPAs. Use this skill whenever the user provides a URL and wants its content, says "scrape", "grab", "fetch", "pull", "get the page", "extract from this URL", or "read this webpage". Handles JS-rendered pages, multiple concurrent URLs, and returns LLM-optimized markdown. Use this instead of WebFetch for any webpage content extraction.

| Web search with full page content extraction. Use this skill whenever the user asks to search the web, find articles, research a topic, look something up, find recent news, discover sources, or says "search for", "find me", "look up", "what are people saying about", or "find articles about". Returns real search results with optional full-page markdown — not just snippets. Provides capabilities beyond Claude's built-in WebSearch.

Check robots.txt and sitemap.xml for crawl-blocking mistakes. Use when the user says "check my robots.txt", "validate my sitemap", or "is my site crawlable".

Search, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON from a site, crawl documentation or site sections, monitor a page for changes, inspect request history, check ScrapeGraph credits, or validate API setup.

Use when the user needs X (Twitter) data or confirmation-gated X actions through Xquik: tweet search, user lookup, follower extraction, media download, monitoring, webhooks, MCP, SDKs, posting, likes, DMs, and profile updates. Requires a Xquik API key. Never ask for X login material.

行业客户名单爬取 — 地区+行业→企业名称/联系人/电话/地址CSV

爬虫/数据采集工程师 Agent — 覆盖网页数据采集、API抓取、动态内容渲染、反爬对抗、数据清洗存储、分布式爬虫架构、App抓包逆向等全领域数据采集工作。能动手搭建完整爬虫系统，不只是出方案。

> and loot tables. Use for a roguelike/roguelite or turn-based grid dungeon crawler with procedural levels.

| 使用 scrapling 进行网页抓取和数据提取。根据目标网站特征自动选择最佳 Fetcher， (1) 抓取/爬取网页内容或数据（scrape, crawl, fetch page, extract data） (2) 需要绕过 Cloudflare/WAF 等反爬保护 (3) 登录后抓取受保护页面 (4) 解析已有 HTML 提取结构化数据 (5) 用户提供 URL 并要求获取页面内容或特定元素 (6) 批量采集多个页面

Bypasses anti-bot protections using Oxylabs Web Unblocker, an AI-powered proxy that handles fingerprinting, JavaScript rendering, and retries automatically. Use when the user needs to scrape protected websites, bypass CAPTCHAs, access blocked content, or when regular proxies fail due to anti-bot measures.

Production-grade web scraping with automatic anti-bot bypass, structured JSON parsing for 40+ targets, and geo-targeting. Use when the user needs to scrape web pages, extract product data, get search results, or collect structured data from supported e-commerce and search platforms without worrying about getting blocked and when geo targeting is required.

>- Dedicated Datacenter/ISP proxies with geo-targeting, IP rotation, session persistence, and port-based sticky IPs. Use when routing traffic through proxies, building scrapers with proxy auth, rotating or sticky sessions, whitelisting IPs, or accessing geo-restricted content.

Authorized web content extraction with trust-boundary controls, scraping cascades, poison-pill detection, browser rendering, observed API analysis, and social-media archiving. Use when extracting public content, diagnosing access failures, implementing respectful scrapers, or processing social-media sources with requests, trafilatura, Playwright, yt-dlp, or instaloader.

This skill should be used when the user requests to research topics using FireCrawl, enrich notes with web sources, search and scrape information, or write scientific/academic papers. It extracts research topics from markdown files, creates research documents with scraped sources, generates BibTeX bibliographies from research results, and provides Pandoc/MyST templates for academic writing with citation management.

| Progressive four-tier URL content scraping with automatic fallback strategy. USE WHEN user says "scrape this URL", "fetch this page", "get content from", "can't access this site", "use Bright Data", "pull content from URL", or needs to retrieve web content that may have bot detection or access restrictions.

Minimal Chrome DevTools Protocol tools for browser automation and scraping. Use when you need to start Chrome, navigate pages, execute JavaScript, take screenshots, or interactively pick DOM elements. Triggers include "browse website", "scrape page", "take screenshot", "automate browser", "extract DOM", "web scraping".

Use this skill for web search, extraction, mapping, crawling, and research via Tavily’s REST API when web searches are needed and no built-in tool is available, or when Tavily’s LLM-friendly format is beneficial.

SerpApi search engine results API via curl. Use this skill to scrape Google, Bing, YouTube, and other search engines.

Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.

Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.

Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.

构建一个全自动化的AI驱动数据收集代理，适用于任何公共来源——招聘网站、价格信息、新闻、GitHub、体育赛事等任何内容。按计划进行抓取，使用免费LLM（Gemini Flash）丰富数据，将结果存储在Notion/Sheets/Supabase中，并从用户反馈中学习。完全免费在GitHub Actions上运行。适用于用户希望自动监控、收集或跟踪任何公共数据的场景。

Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.

X (Twitter) data extraction and scraping. Use when asked to scrape tweets, extract followers, search Twitter/X users, download media from tweets, monitor X accounts, or analyze Twitter engagement. Triggers on twitter, x.com, tweet, follower, following, retweet, quote tweet, scrape, OSINT.

| Firecrawl integration. Manage data, records, and automate workflows. Use when the user wants to interact with Firecrawl data.

| Scrape-It.Cloud integration. Manage Projects, Users, Organizations. Use when the user wants to interact with Scrape-It.Cloud data.

Bulk download images from login-protected gallery websites using an attached browser session. Use when asked to scrape, download, or save images from authenticated gallery pages, extract full-size images from thumbnails, or batch download from multi-page galleries.

Find and enrich B2B leads from Booking.com - hotels, apartments, and vacation rentals - and pull each host's or property manager's real contact details (email, phone, company name, registration number). Use when the user says things like "get emails from Booking.com", "Booking.com lead generation", "find tour operator / property manager / host contacts", "scrape accommodation owner emails", or "build a list of Booking hosts". The host's email is usually already inside the Booking scraper's traderInfo field (EU/AU trader-transparency disclosure) - this skill leads with that and uses a Google Maps email scraper only as a fallback, instead of relying on Google Maps first (which mostly returns the wrong business).

Look up official company data from European public registries across 11 countries/regions (CZ, SK, PL, DE, UK, NL, RO, HR, SE + EU-level + ESG). Covers company registration, ownership, financial filings, VAT status, ESG data. Use when the user asks to "look up a company", "check registry", "find company info", "look up IČO/KRS/LEI/CRN", "company due diligence", "check VAT status", "find ownership structure", or needs official data from European registries. Reads tracked companies from data/companies.json. Some lookups use Python scripts (stdlib), some fall back to Apify actors for scraping-based registries.

Scrape e-commerce data for pricing, reviews, bestsellers, and seller discovery across 30+ platforms including Amazon, Walmart, eBay, Shopify, WooCommerce, and more. Use when user asks about product prices, competitor analysis, store scraping, tech stack detection, food delivery, real estate, or marketplace intelligence.

| Extract structured data from websites, scrape page content, and collect information across multiple pages. pull content from web pages, gather data across multiple pages, or download page content.

Fetch any X/Twitter post as clean LLM-friendly JSON. Converts x.com, twitter.com, or adhx.com links into structured data with full article content, author info, and engagement metrics. No scraping or browser required.

Scrape reviews, ratings, and brand mentions from multiple platforms using Apify Actors.

Scrape leads from multiple platforms using Apify Actors.

AI-driven data extraction from 55+ Actors across all major platforms. This skill automatically selects the best Actor for your task.

Extract product data, prices, reviews, and seller information from any e-commerce platform using Apify's E-commerce Scraping Tool.

Use DeepAPI for supported scraping, research, and email workflows with explicit credentials and approval.

Command-line access to search, scraping, and structured web data.