Gallery Scraper Agent Skill
Bulk download images from login-protected gallery websites using an attached browser session. Use when asked to scrape, download, or save images from authenticated gallery pages, extract full-size images from thumbnails, or batch download from multi-page galleries.
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/jdrhyne/agent-skills --skill gallery-scraperThe instruction itself
14 sections, as written by the authorGallery Scraper
Bulk download images from authenticated gallery websites via browser relay.
Safety Boundaries
- Do not access gallery sites or user accounts that the user has not explicitly attached and authorized.
- Do not download beyond the selected gallery, profile, or page range without confirmation.
- Do not store cookies, tokens, or hidden form values in local output files.
- Do not keep retrying blocked downloads indefinitely; surface rate limits or auth failures instead.
Prerequisites
- User must have Chrome with OpenClaw Browser Relay extension
- User must be logged into the target site
- User must attach the browser tab (click relay toolbar button, badge ON)
Workflow
1. Attach Browser Tab
Ask user to:
- Log into the gallery site in Chrome
- Navigate to the target gallery/profile page
- Click the OpenClaw Browser Relay toolbar button (badge shows ON)
2. Discover Image URL Pattern
Most gallery sites store full-size URLs in data attributes. Common patterns:
// Extract via browser evaluate
() => {
// Try common patterns
const patterns = [
'img[data-max]', // data-max attribute
'img[data-src]', // lazy-load pattern
'img[data-full]', // full-size pattern
'a[data-lightbox] img', // lightbox galleries
'.gallery-item img' // generic gallery
];
for (const sel of patterns) {
const imgs = document.querySelectorAll(sel);
if (imgs.length > 0) {
return {
selector: sel,
count: imgs.length,
sample: imgs[0].outerHTML.substring(0, 200)
};
}
}
return null;
}
3. Extract Full-Size URLs
Once pattern identified, extract all URLs:
// For data-max pattern (common)
() => Array.from(document.querySelectorAll('img[data-max]'))
.map(img => img.dataset.max)
// For thumbnail→full conversion (replace path segment)
() => Array.from(document.querySelectorAll('.gallery img'))
.map(img => img.src.replace('/thumb/', '/full/'))
4. Handle Pagination
Check for multiple pages:
() => {
const pagination = document.querySelectorAll('.pagination a, [class*="page"] a');
return Array.from(pagination).map(a => ({text: a.textContent, href: a.href}));
}
Navigate to each page and collect URLs.
4b. Batch scrape multiple galleries (iframe trick)
When you need multiple galleries quickly and can’t automate CDP, you can load each gallery in a hidden iframe and extract data-max URLs:
async () => {
const urls = [
'https://site.example/galleries/view/123',
'https://site.example/galleries/view/456'
];
const results = [];
for (const url of urls) {
const iframe = document.createElement('iframe');
iframe.style.position = 'fixed';
iframe.style.left = '-9999px';
iframe.style.width = '800px';
iframe.style.height = '600px';
iframe.src = url;
document.body.appendChild(iframe);
await new Promise((resolve, reject) => {
const t = setTimeout(() => reject(new Error('timeout load')), 20000);
iframe.onload = () => { clearTimeout(t); resolve(); };
});
const doc = iframe.contentDocument;
const start = Date.now();
let imgs = [];
while (Date.now() - start < 20000) {
imgs = Array.from(doc.querySelectorAll('img[data-max]')).map(i => i.dataset.max);
if (imgs.length) break;
await new Promise(r => setTimeout(r, 500));
}
results.push({ id: url.split('/').pop(), urls: imgs });
iframe.remove();
}
return results;
}
5. Check CDN Access
Test if CDN requires authentication or just Referer:
# Test direct access
curl -I "CDN_URL" 2>/dev/null | head -3
# Test with Referer
curl -I -H "Referer: https://SITE_DOMAIN/" "CDN_URL" 2>/dev/null | head -3
6. Bulk Download
Collect the URLs into a text file, then parallel download:
# Create output directory
mkdir -p ~/Downloads/gallery_name
# Download with Referer header (parallel)
cd ~/Downloads/gallery_name
while IFS= read -r url; do
filename=$(basename "$url")
curl -s -H "Referer: https://SITE_DOMAIN/" -o "$filename" "$url" &
[ $(jobs -r | wc -l) -ge 8 ] && wait -n
done < urls.txt
wait
Python ThreadPool fallback (avoids shell quoting + wait -n issues):
import os
import requests
from concurrent.futures import ThreadPoolExecutor
outdir = os.path.expanduser('~/Downloads/gallery_name')
os.makedirs(outdir, exist_ok=True)
headers = {'Referer': 'https://SITE_DOMAIN/', 'User-Agent': 'Mozilla/5.0'}
with open('urls.txt') as f:
urls = [line.strip() for line in f if line.strip()]
def download(url):
filename = os.path.join(outdir, os.path.basename(url))
if os.path.exists(filename) and os.path.getsize(filename) > 0:
return
r = requests.get(url, headers=headers, timeout=60)
r.raise_for_status()
with open(filename, 'wb') as f:
f.write(r.content)
with ThreadPoolExecutor(max_workers=8) as ex:
for url in urls:
ex.submit(download, url)
Handling Lock Buttons
Some galleries have "lock" buttons to reveal hidden content. Look for:
// Find lock/unlock buttons
() => {
const locks = document.querySelectorAll(
'[class*="lock"], [class*="unlock"], ' +
'button[title*="lock"], .premium-unlock'
);
return Array.from(locks).map(el => ({
tag: el.tagName,
class: el.className,
text: el.innerText?.substring(0, 30)
}));
}
Click each lock button before extracting URLs.
Output Organization
Optionally organize by gallery:
# Derive a gallery-specific folder name from the selected URL
mkdir -p "gallery_<id>"
Troubleshooting
- 403 Forbidden: Add Referer header or extract cookies from browser
- Rate limited: Reduce parallel downloads, add delays
- Missing images: Check for JavaScript-loaded content, may need scroll injection
- Login required for CDN: Extract session cookies via
document.cookie
Other skills for the same job
different authors, same section of the catalogue
Fetches complete Airbnb listing details for a given numeric listing ID via the internal GraphQL API, returning title, room type, description, amenities, photos, coordinates, city, house rules, highlights, ratings, review count, bedroom configuration, and property overview. Use when user mentions Airbnb listing details, Airbnb property info, Airbnb room details, get Airbnb listing data, Airbnb amenities list, Airbnb house rules, Airbnb property description, Airbnb detail page scraper, Airbnb rooms detail, Airbnb property page data, Airbnb listing info, fetch Airbnb room details, pull Airbnb listing.
Searches Douyin (douyin.com) for videos by keyword and returns structured video data including author info, stats, cover, description, hashtags, and download URL. Supports date range filtering and sorting by relevance, likes, or recency. Use when user mentions Douyin search, scrape Douyin videos, collect TikTok China videos, extract douyin video data, grab douyin results, fetch douyin keyword videos, douyin video list, douyin content mining, search douyin by keyword, douyin likes filter, douyin date filter, douyin video download links, douyin creator info, douyin hashtag extraction, douyin video scraper, douyin KOL research, douyin content analysis.
Fetch full product detail from a Taobao or Tmall product page by itemId, returning title, price, shop info, images, SKU variants, and product attributes. Use when user asks to get product details from Taobao, scrape a Taobao item page, extract product info by item ID, fetch Tmall product data, 抓取淘宝商品详情, 获取淘宝商品信息, 淘宝商品页面采集, 天猫商品详情, 按商品ID获取信息. Also applies to building product databases, price tracking by itemId, and product comparison research.
Fetch customer reviews for a Taobao or Tmall product by itemId, returning reviewer name, date, purchased variant, review text, and photo URLs. Use when user asks to get product reviews from Taobao, scrape Taobao customer feedback, extract buyer reviews by item ID, collect Tmall ratings and comments, 采集淘宝商品评价, 抓取淘宝买家评论, 获取淘宝商品评论, 天猫商品评价抓取, 按商品ID获取评价. Also applies to sentiment analysis of product reviews, building review datasets, and monitoring product rating changes.
TikTok hashtag video scraper: input a hashtag name → output paginated video list with full metadata (author profile, engagement stats, music, video meta, hashtag list). Use when user mentions TikTok hashtag scraping, TikTok tag videos, scrape TikTok by hashtag, extract TikTok hashtag data, TikTok challenge videos, get videos from a TikTok tag, bulk collect TikTok hashtag posts, TikTok video collection by tag, TikTok topic videos, collect TikTok tag data, batch fetch TikTok videos by hashtag, tiktok tag scraper, tiktok challenge scraper. Also applies to competitive research on TikTok trending topics, influencer discovery by hashtag, content monitoring for specific TikTok tags, or any task requiring video lists from a specific TikTok hashtag or challenge.
TikTok user profile video scraper: input a TikTok username → output the user's profile info plus paginated video list with full metadata (engagement stats, music, video meta). Use when user mentions TikTok profile scraping, scrape TikTok user videos, get TikTok creator videos, extract TikTok profile data, TikTok user posts, TikTok account video collection, collect TikTok profile page videos, TikTok creator video list, TikTok creator data, TikTok profile scraper, tiktok user scraper, tiktok creator scraper. Also applies to influencer research, competitor analysis, content archiving for a specific TikTok creator, or extracting all posts from a TikTok account.
TikTok keyword search video scraper: input search keyword → output paginated video list with full metadata (author, engagement stats, music, video meta). Use when user mentions TikTok search scraping, search TikTok by keyword, TikTok search results, extract TikTok search data, scrape TikTok videos by keyword, TikTok keyword videos, TikTok keyword search, TikTok search results collection, find TikTok videos by topic, tiktok search scraper, tiktok keyword scraper. Also applies to market research on TikTok content for specific topics, competitor content monitoring, or discovering videos and creators around a keyword.
Search Xiaohongshu (RedNote / xhs) notes by keyword and return a paginated list with title, author, engagement stats (likes, collects, comments), cover image URL, and xsecToken for detail lookup. Use when user mentions find notes on xiaohongshu, search rednote, search xhs, scrape xiaohongshu search, xiaohongshu keyword search, rednote post search, xhs search results, monitor xiaohongshu topics, KOL content discovery via xiaohongshu, xiaohongshu note list, rednote scrape, xhs data collection, collect xiaohongshu posts, xiaohongshu topic search, xiaohongshu content monitoring, rednote post list, xhs keyword scrape.
How to use it
Take jdrhyne/gallery-scraper from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.