📝 Crawl Url
Crawl any website and save pages as local markdown files. Use when you need to download documentation, knowledge bases, or web content for offline access or analysis. No code required - just provide a URL.
npx playbooks add skill tavily-ai/tavily-plugins --skill crawl-url
About Crawl Url
Crawl any website and save pages as local markdown files. Use when you need to download documentation, knowledge bases, or web content for offline access or analysis. No code required - just provide a URL.
The 35-word prompt provides structured documentation guidance — covering detailed methodology and consistent output formats. Install it in one command.
Use Cases
- Writing API references and README files
- Generating inline code comments and docstrings
- Creating architecture decision records (ADRs)
- Keeping docs in sync with code changes
Example Prompts
System Prompt (35 words)
Crawl any website and save pages as local markdown files. Use when you need to download documentation, knowledge bases, or web content for offline access or analysis. No code required - just provide a URL.
Frequently Asked Questions
What is Crawl Url?
Crawl Url is a free documentation skill for AI coding agents. Crawl any website and save pages as local markdown files. Use when you need to download documentation, knowledge bases, or web content for offline access or analysis. No code required - just provide a URL.. It provides a specialized system prompt that configures your agent with documentation expertise.
How do I use Crawl Url with Claude Code?
Run npx playbooks add skill tavily-ai/tavily-plugins --skill crawl-url in your terminal to install Crawl Url into your Claude Code session. It works immediately after installation.
Which AI coding agents work with Crawl Url?
Crawl Url is compatible with Claude Code, Cursor, GitHub Copilot, Windsurf, OpenClaw, Cline, and any AI agent that supports custom system prompts or .cursorrules files.
Is Crawl Url free to use?
Yes, Crawl Url is completely free and open source. The full source is available on GitHub at https://github.com/tavily-ai/tavily-plugins/tree/main/skills/crawl-url. You only need a subscription to the AI agent you use it with.
