Site Content Catalog Agent Skill

> Crawl a website's sitemap and blog index to build a complete content inventory. Lists every page with URL, title, publish date, content type, and topic cluster. Groups content by category and topic. Optionally deep-reads top N pages for quality analysis and funnel stage tagging. Use before SEO audits, content gap analysis, or brand voice extraction.

8k tokens

context cost

the whole folder, loaded on every use

files

ships runnable scripts

copies elsewhere

how many repositories repackaged it

1086

stars on the repo

on the repository, not the skill itself

Install

one command, takes just this skill from the repository

npx skills add https://github.com/gooseworks-ai/goose-skills --skill site-content-catalog

What comes with it

25 623 bytes besides the instruction

scripts/catalog_content.py

skill.meta.json

What it tells the agent to use

found in the instruction text

WebFetch fetches pages from the network

WebSearch reads your files

The instruction itself

21 sections, as written by the author

Site Content Catalog

Crawl a website's sitemap and blog to build a complete content inventory — every page cataloged with URL, title, date, content type, and topic cluster. Groups content by category, identifies publishing patterns, and optionally deep-analyzes top pages.

Quick Start

# Basic content inventory
python3 scripts/catalog_content.py --domain "example.com"

# With deep analysis of top 20 pages
python3 scripts/catalog_content.py --domain "example.com" --deep-analyze 20

# Output to specific file
python3 scripts/catalog_content.py --domain "example.com" --output content-inventory.json

Inputs

|-----------|----------|---------|-------------|

| domain | Yes | — | Domain to catalog (e.g., "example.com") |

| deep-analyze | No | 0 | Number of top pages to deep-read for content analysis |

Cost

Sitemap/RSS crawling: Free (direct HTTP requests)
Apify sitemap extractor (fallback): ~$0.50 per site
Deep analysis: Free (WebFetch on individual pages)

Process

Phase 1: Discover All Pages

The script attempts multiple methods to find all pages on a site, in order:

A) Sitemap.xml

Fetch https://[domain]/sitemap.xml
If it's a sitemap index, recursively fetch all child sitemaps
Common alternate locations: /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml
Check robots.txt for Sitemap: directives

B) RSS/Atom Feeds

Check /feed, /rss, /atom.xml, /blog/feed, etc.
Extract posts with titles, dates, and URLs
RSS typically only surfaces recent content (last 10-50 posts)

C) Blog Index Crawl

Fetch /blog, /resources, /insights, /news, /articles
Extract links from the page
Follow pagination if present (/blog/page/2, ?page=2, etc.)

D) Site: Search (fallback)

WebSearch: site:[domain] to estimate total indexed pages
WebSearch: site:[domain]/blog to find blog content
WebSearch: site:[domain] intitle: to discover page title patterns

E) Apify Sitemap Extractor (fallback for JS-heavy sites)

Actor: onescales/sitemap-url-extractor
Use when sitemap.xml is missing and the site is JS-rendered

Phase 2: Classify Each Page

For each discovered URL, classify by:

Content Type

Classify based on URL patterns and page titles:

| Type | URL Patterns | Examples |

|------|-------------|----------|

| blog-post | /blog/, /posts/, /articles/ | How-to guides, opinion pieces |

| case-study | /case-study/, /customers/, /success-stories/ | Customer stories |

| comparison | /vs/, /compare/, /alternative/ | X vs Y pages |

| landing-page | /solutions/, /use-cases/, /for-/ | Product marketing pages |

| docs | /docs/, /help/, /documentation/, /api/ | Technical documentation |

| changelog | /changelog/, /releases/, /whats-new/ | Product updates |

| pricing | /pricing/ | Pricing page |

| about | /about/, /team/, /careers/ | Company pages |

| legal | /privacy/, /terms/, /security/ | Legal/compliance |

| resource | /resources/, /guides/, /ebooks/, /webinars/ | Gated/downloadable content |

| glossary | /glossary/, /dictionary/, /terms/ | SEO glossary pages |

| integration | /integrations/, /apps/, /marketplace/ | Integration pages |

| other | — | Anything else |

Topic Cluster

Group by extracting topic signals from URL slugs and titles:

Extract keywords from URL path segments
Group similar keywords into clusters (e.g., "aws-cost", "cloud-spending", "finops" → "Cloud Cost Management")
Use simple keyword co-occurrence for clustering

Phase 3: Analyze Publishing Patterns

From the dated content (primarily blog posts):

Total content pieces by type
Publishing frequency: Posts per month over last 12 months
Trend: Increasing, decreasing, or stable output
Recency: Date of most recent publish
Author diversity: Unique authors (if extractable from RSS)

Phase 4: Deep Analysis (Optional)

If --deep-analyze N is specified, fetch the top N pages (prioritizing blog posts) and extract:

Word count (approximate)
Target keyword (inferred from title + H1 + URL)
Funnel stage: TOFU (awareness), MOFU (consideration), BOFU (decision)
Content depth: Shallow (<500 words), Medium (500-1500), Deep (1500+)
Has images/video: Boolean
Has CTA: Boolean (detected by common CTA patterns)
Internal links count

Phase 5: Output

JSON Output (default)

{
  "domain": "example.com",
  "crawl_date": "2026-02-25",
  "total_pages": 347,
  "discovery_methods": ["sitemap.xml", "rss"],
  "pages": [
    {
      "url": "https://example.com/blog/reduce-aws-costs",
      "title": "How to Reduce Your AWS Bill by 40%",
      "date": "2025-11-15",
      "type": "blog-post",
      "topic_cluster": "Cloud Cost Optimization",
      "deep_analysis": {
        "word_count": 2100,
        "target_keyword": "reduce aws costs",
        "funnel_stage": "TOFU",
        "content_depth": "deep",
        "has_images": true,
        "has_cta": true
      }
    }
  ],
  "summary": {
    "by_type": {"blog-post": 89, "landing-page": 23, "case-study": 12, ...},
    "by_topic": {"Cloud Cost Optimization": 34, "FinOps": 18, ...},
    "publishing_cadence": {
      "posts_per_month_avg": 4.2,
      "trend": "increasing",
      "most_recent": "2026-02-20"
    }
  }
}

Markdown Summary (also generated)

# Content Inventory: example.com
**Crawled:** 2026-02-25 | **Total pages:** 347

## Content by Type
| Type | Count | % |
|------|-------|---|
| Blog Posts | 89 | 25.6% |
| Landing Pages | 23 | 6.6% |
| ...

## Content by Topic Cluster
| Topic | Posts | Most Recent |
|-------|-------|-------------|
| Cloud Cost Optimization | 34 | 2026-02-20 |
| ...

## Publishing Cadence
- Average: 4.2 posts/month
- Trend: Increasing (3.1 → 5.4 over last 6 months)
- Most recent: 2026-02-20

## Full Catalog
| # | Date | Type | Topic | Title | URL |
|---|------|------|-------|-------|-----|
| 1 | 2026-02-20 | blog-post | Cloud Cost | How to Reduce... | https://... |

Tips

Sitemap.xml is the best source. Most well-maintained sites have one. If missing, it's itself an SEO signal (negative).
RSS only shows recent content. If you need the full catalog, sitemap is essential. RSS is supplementary.
Deep analysis is optional but valuable. Use it when feeding into brand-voice-extractor or when you need funnel stage mapping.
JS-rendered sites may need the Apify fallback. Signs: sitemap.xml returns HTML, or blog page returns mostly JavaScript.
Combine with seo-domain-analyzer to overlay traffic data on the content inventory — see which content actually performs.

Dependencies

Python 3.8+
requests library (pip install requests)
APIFY_API_TOKEN env var (only for Apify fallback mode)

Other skills for the same job

different authors, same section of the catalogue

Apify Ultimate Scraper

by apify

vendor ×1

Universal AI-powered web scraper for any platform. Scrape data from Instagram, Facebook, TikTok, YouTube, LinkedIn, X/Twitter, Google Maps, Google Search, Google Trends, Reddit, Airbnb, Yelp, and 15+ more platforms. Use for lead generation, brand monitoring, competitor analysis, influencer discovery, trend research, content analytics, audience analysis, review analysis, SEO intelligence, recruitment, or any data extraction task.

20k tokens

Ebay Sold Listings Search

by browser-act

eBay sold-listings scraper across 8 marketplaces (ebay.com/.co.uk/.de/.fr/.it/.es/.ca/.com.au). Takes keyword plus filters (category, price range, item condition, item location, sort order, completed toggle) and returns paginated real-sale records with itemId, url, title, condition, conditionId, endedAt, soldPrice, soldCurrency, listingType (best_offer_accepted / buy_it_now / auction), isBestOfferAccepted, buyingFormat, bidCount, shipping, totalPrice, thumbnails, seller info, sellerType. Use when user mentions ebay sold, ebay sold listings, ebay sold prices, ebay completed listings, ebay comps, ebay resale prices, ebay auction sold results, ebay best offer accepted, scrape ebay sold, ebay market research, ebay pricing intelligence, ebay flipping research, real sold prices ebay, get sold prices from ebay. Also applies to price benchmarking, sold-price analytics, resale valuation, cross-marketplace price arbitrage, appraisal for collectibles, brand demand tracking on ebay.

10k tokens scripts

Etsy Keyword Search

by browser-act

Etsy keyword search scraper: given a search keyword and optional page number, returns paginated product listings with listingId, shopId, title, url, image, salePrice, originalPrice, currency, rating, reviewCount, shopName, isAd, freeShipping, badge from etsy.com search results. Use when user mentions Etsy, etsy.com, Etsy search, search Etsy by keyword, scrape Etsy listings, extract Etsy products, Etsy product search, Etsy marketplace search, find products on Etsy, Etsy handmade search, Etsy vintage search, Etsy craft search, bulk Etsy product export, Etsy price monitoring, Etsy competitor research, Etsy top listings, Etsy bestseller extraction, Etsy sales data, Etsy shop discovery via keyword. Also applies to trend research on handmade or craft niches, competitor keyword ranking on Etsy, sourcing Etsy suppliers by product type, and any paginated bulk product collection driven by a search keyword.

4k tokens scripts

Etsy Product Detail

by browser-act

Etsy product detail scraper: given an Etsy listing URL, returns full product detail including listingId, title, priceCurrent, priceOriginal, currency, images (all), description, shopName, shopUrl, rating, reviewCount, favorites, inCartCount, variations (with per-option price ranges), highlights, listedDate, relatedTags. Use when user mentions Etsy product, Etsy listing detail, Etsy item info, Etsy product page, scrape Etsy listing, extract Etsy product data, Etsy price and variations, Etsy product images, Etsy product description, Etsy shop from listing, Etsy favorites count, Etsy sale count, Etsy variations extraction, Etsy listing metadata, single Etsy product scrape, bulk enrich Etsy listing URLs, Etsy product details export. Also applies to competitor product monitoring, price and variation tracking on a specific listing, favorites/wishlist popularity tracking, description mining for SEO analysis, and any per-listing enrichment task.

4k tokens scripts

Goofish Search List

by browser-act

Scrapes second-hand item search results from Goofish (闲鱼/xianyu, goofish.com) — China's largest second-hand marketplace. Input: keyword, optional sort/filter params. Output: list of items with id, title, price, image, location, want-count per page (30 items/page). Use when user mentions goofish, 闲鱼, xianyu, 二手交易, second-hand marketplace China, 二手商品搜索, search used goods, scrape goofish listings, xianyu search results, collect second-hand prices, monitor used item prices, 闲鱼关键词搜索, 闲鱼数据采集, 批量抓取闲鱼, goofish scraper, goofish data, xianyu data extraction, 二手商品价格监控, used iPhone prices, 二手手机价格. Also applies to: price research on Chinese second-hand market, competitor product monitoring via used goods listings, inventory analysis.

4k tokens scripts

Google Maps API Skill

by browser-act

This skill helps users automatically scrape business data from Google Maps using the BrowserAct Google Maps API. Agent should proactively trigger this skill for needs like finding restaurants in a specific city, extracting contact info of dental clinics, researching local competitors, collecting addresses of coffee shops, generating lead lists for specific industries, monitoring business ratings and reviews, getting opening hours of local services, finding specialized stores (e.g., Turkish-style restaurants), analyzing business categories in a region, extracting website links from local businesses, gathering phone numbers for sales outreach, mapping out service providers in a specific country.

2k tokens scripts

Google Search Serp

by browser-act

Extracts Google Search results page (SERP) data including organic results, paid ads, related searches, People Also Ask questions, AI Overview text, and total result count from google.com. Use when user mentions Google search results, SERP scraping, google search data, search engine results page, organic rankings, keyword SERP, Google SERP extraction, scrape Google search, Google search API alternative, SEO ranking data, paid search ads, PPC ads on Google, Google search monitoring, keyword research, search results export, check Google rankings, what shows up on Google, search engine scraper, google results checker.

3k tokens scripts

Indeed Job Search

by browser-act

Scrape job listings from Indeed.com by keyword, location, and country. Returns job title, company, salary, rating, description, benefits, and apply links. Use when user mentions Indeed, Indeed scraper, Indeed jobs, scrape Indeed, job search Indeed, Indeed job listings, extract Indeed data, Indeed job data, get jobs from Indeed, Indeed employment data, job market research Indeed, Indeed salary data, bulk job extraction Indeed, Indeed job scraper, monitor Indeed listings, Indeed hiring data, job postings Indeed, Indeed career search. Also applies to: job market analysis, salary benchmarking from Indeed, competitor hiring monitoring, recruitment data collection, building job databases from Indeed search results.

4k tokens scripts

How to use it

Copy the folder

Take gooseworks-ai/site-content-catalog from the repository into ~/.claude/skills for personal use, or into .claude/skills inside a project.

Check the name does not clash

The agent identifies a skill by the name field in its header. Two skills with the same name cannot sit side by side — one of them will be ignored.

Install what it needs

The instructions reference pip. Without those the skill loads but fails at the first command.