🧪 Web Scraping
Web scraping with anti-bot bypass, content extraction, and poison pill detection. Use when extracting content from websites, handling paywalls, implementing scraping cascades, or processing social media. Covers trafilatura, Playwright with stealth mode, yt-dlp, and instaloader patterns.
npx playbooks add skill jamditis/claude-skills-journalism --skill web-scraping
About Web Scraping
Web scraping with anti-bot bypass, content extraction, and poison pill detection. Use when extracting content from websites, handling paywalls, implementing scraping cascades, or processing social media. Covers trafilatura, Playwright with stealth mode, yt-dlp, and instaloader patterns.
This compact 36-word instruction set is purpose-built for testing & qa work in AI coding agents. Install with a single command.
Use Cases
- Writing unit, integration, and end-to-end tests
- Setting up test coverage and CI pipelines
- Refactoring legacy code with confidence using tests
- Creating test plans and QA checklists
Example Prompts
System Prompt (36 words)
Web scraping with anti-bot bypass, content extraction, and poison pill detection. Use when extracting content from websites, handling paywalls, implementing scraping cascades, or processing social media. Covers trafilatura, Playwright with stealth mode, yt-dlp, and instaloader patterns.
Frequently Asked Questions
What is Web Scraping?
Web Scraping is a free testing & qa skill for AI coding agents. Web scraping with anti-bot bypass, content extraction, and poison pill detection. Use when extracting content from websites, handling paywalls, implementing scraping cascades, or processing social media. Covers trafilatura, Playwright with stealth mode, yt-dlp, and instaloader patterns.. It provides a specialized system prompt that configures your agent with testing & qa expertise.
How do I use Web Scraping with Claude Code?
Run npx playbooks add skill jamditis/claude-skills-journalism --skill web-scraping in your terminal to install Web Scraping into your Claude Code session. It works immediately after installation.
Which AI coding agents work with Web Scraping?
Web Scraping is compatible with Claude Code, Cursor, GitHub Copilot, Windsurf, OpenClaw, Cline, and any AI agent that supports custom system prompts or .cursorrules files.
Is Web Scraping free to use?
Yes, Web Scraping is completely free and open source. The full source is available on GitHub at https://github.com/jamditis/claude-skills-journalism/tree/main/web-scraping. You only need a subscription to the AI agent you use it with.
