Apify Ultimate Scraper Agent Skill
Universal AI-powered web scraper for any platform. Scrape data from Instagram, Facebook, TikTok, YouTube, LinkedIn, X/Twitter, Google Maps, Google Search, Google Trends, Reddit, Airbnb, Yelp, and 15+ more platforms. Use for lead generation, brand monitoring, competitor analysis, influencer discovery, trend research, content analytics, audience analysis, review analysis, SEO intelligence, recruitment, or any data extraction task.
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/apify/agent-skills --skill apify-ultimate-scraperWhat comes with it
73 626 bytes besides the instructionWhat it tells the agent to use
found in the instruction textThe instruction itself
10 sections, as written by the authorUniversal web scraper
AI-driven data extraction from ~100 Actors across 15+ platforms via the Apify CLI.
Rule: Pass --json and redirect stderr with 2>/dev/null on data-returning commands (actors call, actors start, actors info, actors search, datasets get-items, runs info). JSON output is stable across CLI versions. stderr contains progress messages and version warnings that break JSON parsers if not redirected.
This rule does not apply to status/auth commands (apify info, apify --version, apify login). For those, use 2>&1 so authentication and version errors are visible.
Exception: if --input returns no data, re-run with 2>&1 to confirm whether the cause is a missing schema vs. a network/auth error.
Prerequisites
- Apify CLI v1.4.0+ (
npm install -g apify-cli) - Authenticated session (see below)
Authentication
If a CLI command fails with an auth error, authenticate using one of these methods:
- OAuth (interactive):
apify login(opens browser) - Environment variable:
export APIFY_TOKEN=your_token_here - From .env file:
source .env(if the file containsAPIFY_TOKEN=...)
Generate token: https://console.apify.com/settings/integrations
Workflow
Step 0: Verify CLI readiness before doing anything else
Before using the Apify CLI, always verify the local environment:
- Check that the CLI is installed:
apify --help
If this fails, install the CLI first:
npm install -g apify-cli
- Check that the CLI is authenticated:
# Auth check — do NOT pipe to /dev/null, you need to see errors
apify info 2>&1
If this shows the user is not logged in, instruct them to authenticate with a token:
apify login --token TOKEN
- Run Apify CLI commands with
allpermissions when needed by the agent sandbox.
- Assume many Apify commands block with zero output until completion. For blocking runs, set
block_until_msto at least60000.
- For long or unknown-duration runs, prefer the async pattern:
apify actors start "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null
Then poll the run status:
apify info actor-runs/RUN_ID --json
Check .status for SUCCEEDED or FAILED.
Step 1: Understand goal and select Actor
Identify the target platform and use case. Read references/actor-index.md to find the right Actor.
Prefer apify-tier actors; use community-tier only when no apify actor covers the task.
For input schemas, fetch dynamically: apify actors info "ACTOR_ID" --input --json 2>/dev/null
If the output is empty, re-run without the redirect (2>&1) to surface auth or network errors before proceeding.
If the task involves a multi-step pipeline, also read the matching workflow guide:
| Task involves... | Read |
|-----------------|------|
| leads, contacts, emails, B2B | references/workflows/lead-generation.md |
| competitor, ads, pricing | references/workflows/competitive-intel.md |
| influencer, creator | references/workflows/influencer-vetting.md |
| brand, mentions, sentiment | references/workflows/brand-monitoring.md |
| reviews, ratings, reputation | references/workflows/review-analysis.md |
| SEO, SERP, crawl, content, RAG | references/workflows/content-and-seo.md |
| analytics, engagement, performance | references/workflows/social-media-analytics.md |
| trends, keywords, hashtags | references/workflows/trend-research.md |
| jobs, recruiting, candidates | references/workflows/job-market-and-recruitment.md |
| real estate, listings, hotels | references/workflows/real-estate-and-hospitality.md |
| price monitoring, e-commerce, products | references/workflows/ecommerce-price-monitoring.md |
| contact enrichment, email extraction | references/workflows/contact-enrichment.md |
| knowledge base, RAG, LLM data feed | references/workflows/knowledge-base-and-rag.md |
| company research, due diligence | references/workflows/company-research.md |
If no Actor matches in the index, search dynamically:
apify actors search "KEYWORDS" --json --limit 10 2>/dev/null
From results: items[].username/items[].name (Actor ID), items[].title, items[].stats.totalUsers30Days, items[].currentPricingInfo.pricingModel.
Step 2: Fetch Actor schema and check gotchas
Some Actors don't register an input schema with the platform (their schema lives in code). Try schema sources in this order — fall through on empty/error:
- Input schema (human-readable):
apify actors info "ACTOR_ID" --input 2>/dev/null
If output is Error: No input schema found for this Actor, skip to source 2.
- Input schema (JSON keys only):
apify actors info "ACTOR_ID" --input --json 2>/dev/null | jq '.input.schema.properties // empty | keys'
Empty result means no registered schema — fall through to source 3. To drill into a specific field:
apify actors info "ACTOR_ID" --input --json 2>/dev/null | jq '.input.schema.properties.FIELD_NAME'
- README fallback (always works, contains usage examples):
apify actors info "ACTOR_ID" --readme 2>/dev/null
Grep the README for an "Input" / "Example input" section to copy the JSON shape.
- Last resort — call with minimal known input (e.g.
{"startUrls":[{"url":"..."}]}for crawlers) and let the Actor surface validation errors that reveal required fields. Seereferences/gotchas.mdfor known-good minimal inputs for common Actors.
Also read references/gotchas.md to check for common pitfalls and cost guardrails for the selected Actor.
Step 3: Configure and run
Skip user preferences for simple lookups (e.g., "Nike's follower count"). Go straight to running with quick answer mode.
For larger tasks, confirm output format (quick answer / CSV / JSON) and result count.
Before starting the run, double-check whether the task is short enough for a blocking call or should use the async pattern from Step 0.
Standard run (blocking):
apify actors call "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null
From output: .id (run ID), .status, .defaultDatasetId, .stats.durationMillis
Fetch results:
apify datasets get-items DATASET_ID --format json
For CSV: apify datasets get-items DATASET_ID --format csv
Quick answer mode: Fetch results as JSON, pick top 5, present formatted in chat.
Save to file: Fetch results, use Write tool to save as YYYY-MM-DD_descriptive-name.csv or .json.
Large/long-running scrapes:
apify actors start "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null
Poll: apify info actor-runs/RUN_ID --json (check .status for SUCCEEDED or FAILED).
Step 4: Deliver results
Report: result count, file location (if saved), key data fields, and links:
- Dataset:
https://console.apify.com/storage/datasets/DATASET_ID - Run:
https://console.apify.com/actors/runs/RUN_ID
For multi-step workflows: suggest the next pipeline step from the workflow guide.
Troubleshooting
Common errors and pitfalls are documented in references/gotchas.md. Read it before running PPE (pay-per-event) Actors.
Other skills for the same job
different authors, same section of the catalogue
eBay sold-listings scraper across 8 marketplaces (ebay.com/.co.uk/.de/.fr/.it/.es/.ca/.com.au). Takes keyword plus filters (category, price range, item condition, item location, sort order, completed toggle) and returns paginated real-sale records with itemId, url, title, condition, conditionId, endedAt, soldPrice, soldCurrency, listingType (best_offer_accepted / buy_it_now / auction), isBestOfferAccepted, buyingFormat, bidCount, shipping, totalPrice, thumbnails, seller info, sellerType. Use when user mentions ebay sold, ebay sold listings, ebay sold prices, ebay completed listings, ebay comps, ebay resale prices, ebay auction sold results, ebay best offer accepted, scrape ebay sold, ebay market research, ebay pricing intelligence, ebay flipping research, real sold prices ebay, get sold prices from ebay. Also applies to price benchmarking, sold-price analytics, resale valuation, cross-marketplace price arbitrage, appraisal for collectibles, brand demand tracking on ebay.
Etsy keyword search scraper: given a search keyword and optional page number, returns paginated product listings with listingId, shopId, title, url, image, salePrice, originalPrice, currency, rating, reviewCount, shopName, isAd, freeShipping, badge from etsy.com search results. Use when user mentions Etsy, etsy.com, Etsy search, search Etsy by keyword, scrape Etsy listings, extract Etsy products, Etsy product search, Etsy marketplace search, find products on Etsy, Etsy handmade search, Etsy vintage search, Etsy craft search, bulk Etsy product export, Etsy price monitoring, Etsy competitor research, Etsy top listings, Etsy bestseller extraction, Etsy sales data, Etsy shop discovery via keyword. Also applies to trend research on handmade or craft niches, competitor keyword ranking on Etsy, sourcing Etsy suppliers by product type, and any paginated bulk product collection driven by a search keyword.
Etsy product detail scraper: given an Etsy listing URL, returns full product detail including listingId, title, priceCurrent, priceOriginal, currency, images (all), description, shopName, shopUrl, rating, reviewCount, favorites, inCartCount, variations (with per-option price ranges), highlights, listedDate, relatedTags. Use when user mentions Etsy product, Etsy listing detail, Etsy item info, Etsy product page, scrape Etsy listing, extract Etsy product data, Etsy price and variations, Etsy product images, Etsy product description, Etsy shop from listing, Etsy favorites count, Etsy sale count, Etsy variations extraction, Etsy listing metadata, single Etsy product scrape, bulk enrich Etsy listing URLs, Etsy product details export. Also applies to competitor product monitoring, price and variation tracking on a specific listing, favorites/wishlist popularity tracking, description mining for SEO analysis, and any per-listing enrichment task.
Scrapes second-hand item search results from Goofish (闲鱼/xianyu, goofish.com) — China's largest second-hand marketplace. Input: keyword, optional sort/filter params. Output: list of items with id, title, price, image, location, want-count per page (30 items/page). Use when user mentions goofish, 闲鱼, xianyu, 二手交易, second-hand marketplace China, 二手商品搜索, search used goods, scrape goofish listings, xianyu search results, collect second-hand prices, monitor used item prices, 闲鱼关键词搜索, 闲鱼数据采集, 批量抓取闲鱼, goofish scraper, goofish data, xianyu data extraction, 二手商品价格监控, used iPhone prices, 二手手机价格. Also applies to: price research on Chinese second-hand market, competitor product monitoring via used goods listings, inventory analysis.
This skill helps users automatically scrape business data from Google Maps using the BrowserAct Google Maps API. Agent should proactively trigger this skill for needs like finding restaurants in a specific city, extracting contact info of dental clinics, researching local competitors, collecting addresses of coffee shops, generating lead lists for specific industries, monitoring business ratings and reviews, getting opening hours of local services, finding specialized stores (e.g., Turkish-style restaurants), analyzing business categories in a region, extracting website links from local businesses, gathering phone numbers for sales outreach, mapping out service providers in a specific country.
Extracts Google Search results page (SERP) data including organic results, paid ads, related searches, People Also Ask questions, AI Overview text, and total result count from google.com. Use when user mentions Google search results, SERP scraping, google search data, search engine results page, organic rankings, keyword SERP, Google SERP extraction, scrape Google search, Google search API alternative, SEO ranking data, paid search ads, PPC ads on Google, Google search monitoring, keyword research, search results export, check Google rankings, what shows up on Google, search engine scraper, google results checker.
Scrape job listings from Indeed.com by keyword, location, and country. Returns job title, company, salary, rating, description, benefits, and apply links. Use when user mentions Indeed, Indeed scraper, Indeed jobs, scrape Indeed, job search Indeed, Indeed job listings, extract Indeed data, Indeed job data, get jobs from Indeed, Indeed employment data, job market research Indeed, Indeed salary data, bulk job extraction Indeed, Indeed job scraper, monitor Indeed listings, Indeed hiring data, job postings Indeed, Indeed career search. Also applies to: job market analysis, salary benchmarking from Indeed, competitor hiring monitoring, recruitment data collection, building job databases from Indeed search results.
Trustpilot company profile lookup on trustpilot.com — input a company domain (e.g. apple.com, shopify.com, shopwagandtail.com) and extract company metadata: official display name, businessUnitId, TrustScore (1-5), star rating, total review count, last-12-months review count, primary category and all categories, verification flags (verifiedByGoogle, verifiedPaymentMethod, verifiedUserIdentity), claimed/closed status, website URL, contact info (email, phone, country, city, address, zip), profile image, and optional reply behavior metrics (averageDaysToReply, replyPercentage, negativeReviewsWithRepliesCount, companyUsesAIResponses). Detects company-not-found and returns isCompanyFound=false with the searched domain. Use when user mentions Trustpilot company info, Trustpilot TrustScore, look up Trustpilot rating, check Trustpilot stars, Trustpilot business profile, Trustpilot review count, Trustpilot company lookup, Trustpilot verification status, Trustpilot reply percentage, Trustpilot AI responses, business unit ID Trustpilot, get company rating from Trustpilot, scrape Trustpilot company, batch enrich domains with Trustpilot, bulk company info Trustpilot, fast domain enrichment with TrustScore, Trustpilot company metadata, Trustpilot brand monitoring, competitor TrustScore comparison, trustpilot.com domain enrichment, trustpilot company profile, trustpilot domain check, trustpilot business listing, check if company is on Trustpilot, Trustpilot category lookup, company contact info from Trustpilot. Also applies to lead-list enrichment with reputation scores, supplier vetting, agency or KOL vetting via review reputation, aggregator sites that need TrustScores for many domains, due-diligence workflows, and pre-outreach reputation checks.
How to use it
Take apify/apify-ultimate-scraper from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.
The instructions reference npm.
Without those the skill loads but fails at the first command.