Firecrawl Just Fixed the Pagination Bug in Every AI Agent
Your AI agent scrapes page 1 and calls it done, because detecting where pagination actually ends is harder than it looks. Static HTML, JS-rendered infinite scroll, numbered pages, offset pagination: every site handles it differently, and hand-rolled detection breaks the moment a site's markup changes.
In this video we show how to scrape every page of a paginated website automatically using Firecrawl and the Node SDK. It knows exactly when to stop: the last page is just the one that comes back with zero results. No custom regex, no guessing page counts, just structured JSON back on every page of your web scraper or AI agent pipeline.
We build it up step by step:
Scrape a single page as plain markdown
Add a Zod schema for structured data extraction, returning real JSON
Wrap it in a loop that walks every page until results run out, saving everything to a data store
Firecrawl is also available via MCP, CLI, and SDKs like Python and Node, so you can use whatever fits your stack.
Use the coupon FIRECRAWLYT for 500 extra credits
Code from this video: https://gist.github.com/RichardBray/d...
👇 Learn more about Firecrawl 👇
🔥 Website: https://www.firecrawl.dev?via=youtube
⚡ Get started: https://www.firecrawl.dev/search
📚 Docs: https://docs.firecrawl.dev/
📱 More Links:
💻 GitHub: https://github.com/firecrawl/firecrawl
🎧 Discord: / discord
🐦 X (Twitter): https://x.com/firecrawl_dev
💼 Linkedin: / firecrawl