Using Web Scraping Agent Skill

by besoeasy

Search and scrape public web content with headless Chrome and DuckDuckGo using safe practices.

866 tokens

context cost

the whole folder, loaded on every use

files

instructions only

copies elsewhere

how many repositories repackaged it

127

stars on the repo

on the repository, not the skill itself

Install

one command, takes just this skill from the repository

npx skills add https://github.com/besoeasy/open-skills --skill using-web-scraping

The instruction itself

10 sections, as written by the author

Web Scraping Skill — Chrome (Playwright) + DuckDuckGo

A privacy-minded, agent-facing web-scraping skill that uses headless Chrome (Playwright/Puppeteer) and DuckDuckGo for search. Focuses on: reliable navigation, extracting structured text, obeying robots.txt, and rate-limiting.

When to use

Collect public webpage content for summarization, metadata extraction, or link discovery.
Use DuckDuckGo for queries when you want a privacy-respecting search source.
NOT for bypassing paywalls, scraping private/logged-in content, or violating Terms of Service.

Safety & etiquette

Always check and respect /robots.txt before scraping a site.
Rate-limit requests (default: 1 request/sec) and use polite User-Agent strings.
Avoid executing arbitrary user-provided JavaScript on scraped pages.
Only scrape public content; if login is required, return login_required instead of attempting to bypass.

Capabilities

Search DuckDuckGo and return top-N result links.
Visit result pages in headless Chrome and extract title, meta description, main text (or best-effort article text), and canonical URL.
Return results as structured JSON for downstream consumption.

Examples

Node.js (Playwright)

const { chromium } = require('playwright');

async function ddgSearchAndScrape(query) {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage({ userAgent: 'open-skills-bot/1.0' });

  // DuckDuckGo search
  await page.goto('https://duckduckgo.com/');
  await page.fill('input[name="q"]', query);
  await page.keyboard.press('Enter');
  await page.waitForSelector('.result__title a');

  // collect top result URL
  const href = await page.getAttribute('.result__title a', 'href');
  if (!href) { await browser.close(); return []; }

  // visit result and extract
  await page.goto(href, { waitUntil: 'domcontentloaded' });
  const title = await page.title();
  const description = await page.locator('meta[name="description"]').getAttribute('content').catch(() => null);
  const article = await page.locator('article, main, #content').first().innerText().catch(() => null);

  await browser.close();
  return [{ url: href, title, description, text: article }];
}

// usage
// ddgSearchAndScrape('open-source agent runtimes').then(console.log);

Agent prompt (copy/paste)

You are an agent with a web-scraping skill. For any `search:` task, use DuckDuckGo to find relevant pages, then open each page in a headless Chrome instance (Playwright/Puppeteer) and extract `title`, `meta description`, `main text`, and `canonical` URL. Always:
- Check and respect robots.txt
- Rate-limit requests (<=1 req/sec)
- Use a clear `User-Agent` and do not execute arbitrary page JS
Return results as JSON: [{url,title,description,text}] or `login_required` if a page needs authentication.

Quick setup

Node: npm i playwright and run npx playwright install for browser binaries.
Python: pip install playwright and playwright install.

Tips

Use page.route to block large assets (images, fonts) when you only need text.
Respect site terms and introduce exponential backoff for retries.

Other skills for the same job

different authors, same section of the catalogue

Apify Automation

by christophacham

×2

Automate web scraping and data extraction with Apify -- run Actors, manage datasets, create reusable tasks, and retrieve crawl results through the Composio Apify integration.

Scrapes content based on a preset URL list, filters high-quality technical information, and generates daily Markdown reports.

Web search, content extraction, crawling, and research capabilities using Tavily API

| Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extract", or needs content from many pages on the same site. Handles depth limits, path filtering, and concurrent extraction.

Deep web scraping, screenshots, PDF parsing, and website crawling using Firecrawl API. Use when you need deep content extraction from web pages, page interaction is required (clicking, scrolling, etc.), or you want screenshots or PDF parsing.

Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.

Scrapes content based on a preset URL list, filters high-quality technical information, and generates daily Markdown reports.

Scrape, crawl, search, and extract structured data from websites using Firecrawl API - converts web pages to LLM-ready markdown

6k tokens scripts