Web Scraping with Puppeteer and Cheerio: A Practical Guide for Developers

Web scraping remains one of the most powerful techniques for gathering data from websites that lack public APIs. Whether you need to monitor competitor pricing, aggregate research data, or build datasets for machine learning, having a solid scraping toolkit is essential. In the Node.js ecosystem, two libraries stand out: Puppeteer for browser automation and dynamic content, and Cheerio for fast, lightweight HTML parsing.

This guide walks you through building production-ready web scrapers using both tools. You will learn when to use each one, how to combine them for maximum efficiency, and how to handle the real-world challenges that tutorials rarely cover — anti-bot detection, pagination, rate limiting, and error recovery.

Why Puppeteer and Cheerio?

Before diving into code, it helps to understand why these two libraries complement each other so well. If you are new to the Node.js ecosystem, our guide to getting started with Node.js covers the fundamentals you will need.

Cheerio: The Lightweight Parser

Cheerio provides a jQuery-like API for parsing and manipulating HTML strings. It does not render pages or execute JavaScript — it simply parses raw HTML and lets you traverse the DOM using familiar CSS selectors. This makes it extremely fast and memory-efficient. For static pages where the content is present in the initial HTML response, Cheerio is the ideal choice.

Key advantages of Cheerio include minimal resource consumption (no browser instance required), parsing speeds measured in milliseconds rather than seconds, a familiar jQuery syntax that reduces the learning curve, and excellent compatibility with Node.js streams for processing large volumes of data.

Puppeteer: The Browser Automation Engine

Puppeteer controls a headless (or headful) Chromium browser instance. It can click buttons, fill forms, scroll pages, wait for network requests, and capture screenshots. When websites rely on JavaScript to render content — single-page applications, infinite scroll feeds, or pages behind login walls — Puppeteer is your tool.

The tradeoff is resource usage. Each Puppeteer instance launches a full browser process, consuming significant CPU and memory. For large-scale scraping, you need to be strategic about when to use it. If you are interested in browser automation beyond scraping, our Playwright end-to-end testing guide explores a similar tool built by the same team.

The Hybrid Approach

The most effective scraping architecture uses both tools together. Puppeteer handles JavaScript-rendered pages, authentication flows, and anti-bot challenges. Once the fully rendered HTML is captured, Cheerio takes over to parse and extract data efficiently. This hybrid approach gives you the flexibility of browser automation with the speed of static parsing.

Setting Up Your Scraping Environment

Start by initializing a new project and installing dependencies. If you prefer type safety in your scraping code, consider following our TypeScript for JavaScript developers guide to set up TypeScript support.

mkdir web-scraper && cd web-scraper
npm init -y
npm install puppeteer cheerio axios
npm install -D @types/cheerio typescript

For containerized deployments, Puppeteer requires specific system dependencies. Our Docker for web developers guide covers container setup in detail, but here is a minimal Dockerfile for scraping:

FROM node:20-slim
RUN apt-get update && apt-get install -y \
    chromium fonts-ipafont-gothic fonts-freefont-ttf \
    --no-install-recommends && rm -rf /var/lib/apt/lists/*
ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
CMD ["node", "scraper.js"]

Building a Cheerio HTML Parsing Pipeline

Let us start with Cheerio since it covers the simpler use case. The following pipeline fetches a page, parses the HTML, extracts structured data, and handles errors gracefully. This pattern works well for static websites, documentation pages, and any content that does not require JavaScript execution.

const axios = require('axios');
const cheerio = require('cheerio');
const fs = require('fs/promises');

// Configuration
const CONFIG = {
  baseUrl: 'https://example-blog.com',
  maxRetries: 3,
  retryDelay: 2000,
  requestDelay: 1500,
  outputFile: 'scraped_articles.json',
  headers: {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
  }
};

// Delay helper for rate limiting
const delay = (ms) => new Promise(resolve => setTimeout(resolve, ms));

// Fetch with retry logic
async function fetchWithRetry(url, retries = CONFIG.maxRetries) {
  for (let attempt = 1; attempt <= retries; attempt++) {
    try {
      const response = await axios.get(url, {
        headers: CONFIG.headers,
        timeout: 15000,
        maxRedirects: 5,
      });
      return response.data;
    } catch (error) {
      const status = error.response?.status;
      console.error(`Attempt ${attempt}/${retries} failed for ${url}: ${status || error.message}`);

      if (status === 404) throw new Error(`Page not found: ${url}`);
      if (status === 403) throw new Error(`Access forbidden: ${url}`);

      if (attempt < retries) {
        const backoff = CONFIG.retryDelay * Math.pow(2, attempt - 1);
        console.log(`Retrying in ${backoff}ms...`);
        await delay(backoff);
      }
    }
  }
  throw new Error(`All ${retries} attempts failed for ${url}`);
}

// Parse a single article page
function parseArticle(html, url) {
  const $ = cheerio.load(html);

  // Remove unwanted elements before extraction
  $('script, style, nav, footer, .advertisement, .sidebar').remove();

  const article = {
    url,
    title: $('h1.article-title').first().text().trim(),
    author: $('meta[name="author"]').attr('content') || $('[rel="author"]').text().trim(),
    publishDate: $('time[datetime]').attr('datetime') || null,
    categories: [],
    content: '',
    wordCount: 0,
    images: [],
    links: [],
  };

  // Extract categories
  $('.category-tag, .post-category').each((_, el) => {
    const cat = $(el).text().trim();
    if (cat) article.categories.push(cat);
  });

  // Extract main content with cleanup
  const contentEl = $('article, .post-content, .entry-content').first();
  article.content = contentEl.text().replace(/\s+/g, ' ').trim();
  article.wordCount = article.content.split(/\s+/).filter(Boolean).length;

  // Extract images with alt text
  contentEl.find('img').each((_, el) => {
    const src = $(el).attr('src') || $(el).attr('data-src');
    if (src) {
      article.images.push({
        src: new URL(src, url).href,
        alt: $(el).attr('alt') || '',
      });
    }
  });

  // Extract internal links
  contentEl.find('a[href]').each((_, el) => {
    const href = $(el).attr('href');
    if (href && !href.startsWith('#') && !href.startsWith('javascript:')) {
      article.links.push({
        href: new URL(href, url).href,
        text: $(el).text().trim(),
      });
    }
  });

  return article;
}

// Discover article URLs from listing pages
function extractArticleLinks(html, baseUrl) {
  const $ = cheerio.load(html);
  const links = new Set();

  $('a[href]').each((_, el) => {
    const href = $(el).attr('href');
    if (href && href.includes('/article/')) {
      links.add(new URL(href, baseUrl).href);
    }
  });

  return [...links];
}

// Main scraping pipeline
async function scrapeSite() {
  console.log('Starting Cheerio scraping pipeline...');
  const allArticles = [];
  let page = 1;
  let hasMore = true;

  while (hasMore) {
    const listUrl = `${CONFIG.baseUrl}/articles?page=${page}`;
    console.log(`Fetching listing page ${page}: ${listUrl}`);

    try {
      const html = await fetchWithRetry(listUrl);
      const articleUrls = extractArticleLinks(html, CONFIG.baseUrl);

      if (articleUrls.length === 0) {
        hasMore = false;
        break;
      }

      for (const articleUrl of articleUrls) {
        try {
          await delay(CONFIG.requestDelay);
          const articleHtml = await fetchWithRetry(articleUrl);
          const article = parseArticle(articleHtml, articleUrl);

          if (article.title && article.wordCount > 50) {
            allArticles.push(article);
            console.log(`Parsed: "${article.title}" (${article.wordCount} words)`);
          }
        } catch (err) {
          console.error(`Skipping ${articleUrl}: ${err.message}`);
        }
      }

      page++;
    } catch (err) {
      console.error(`Failed to fetch listing page ${page}: ${err.message}`);
      hasMore = false;
    }
  }

  // Save results
  await fs.writeFile(CONFIG.outputFile, JSON.stringify(allArticles, null, 2));
  console.log(`Saved ${allArticles.length} articles to ${CONFIG.outputFile}`);
  return allArticles;
}

scrapeSite().catch(console.error);

This pipeline demonstrates several production best practices. The retry logic with exponential backoff handles transient network failures gracefully. The configurable request delay prevents overwhelming the target server — a critical consideration covered in our API rate limiting and throttling guide. The parser strips non-content elements before extraction, reducing noise in the output data.

Building a Puppeteer Scraper with Anti-Detection and Pagination

When websites render content dynamically with JavaScript or employ bot detection, you need Puppeteer. The following scraper navigates paginated results, handles lazy-loaded content, implements anti-detection measures, and extracts data from JavaScript-rendered pages.

const puppeteer = require('puppeteer');
const cheerio = require('cheerio');
const fs = require('fs/promises');

const CONFIG = {
  targetUrl: 'https://example-spa.com/products',
  maxPages: 50,
  navigationTimeout: 30000,
  scrollDelay: 800,
  pageDelay: 2000,
  outputFile: 'products.json',
};

// Launch browser with anti-detection settings
async function createBrowser() {
  const browser = await puppeteer.launch({
    headless: 'new',
    args: [
      '--no-sandbox',
      '--disable-setuid-sandbox',
      '--disable-blink-features=AutomationControlled',
      '--disable-features=IsolateOrigins,site-per-process',
      '--window-size=1920,1080',
      '--disable-web-security',
      '--disable-dev-shm-usage',
    ],
  });

  const page = await browser.newPage();

  // Mask automation signals
  await page.evaluateOnNewDocument(() => {
    // Override navigator.webdriver property
    Object.defineProperty(navigator, 'webdriver', {
      get: () => undefined,
    });

    // Override chrome runtime indicator
    window.chrome = {
      runtime: {},
      loadTimes: function () {},
      csi: function () {},
    };

    // Override permissions API
    const originalQuery = window.navigator.permissions.query;
    window.navigator.permissions.query = (parameters) =>
      parameters.name === 'notifications'
        ? Promise.resolve({ state: Notification.permission })
        : originalQuery(parameters);

    // Override plugins length
    Object.defineProperty(navigator, 'plugins', {
      get: () => [1, 2, 3, 4, 5],
    });

    // Override languages
    Object.defineProperty(navigator, 'languages', {
      get: () => ['en-US', 'en'],
    });
  });

  // Set realistic viewport and user agent
  await page.setViewport({ width: 1920, height: 1080 });
  await page.setUserAgent(
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
  );

  // Block unnecessary resources to speed up loading
  await page.setRequestInterception(true);
  page.on('request', (req) => {
    const blocked = ['image', 'stylesheet', 'font', 'media'];
    if (blocked.includes(req.resourceType())) {
      req.abort();
    } else {
      req.continue();
    }
  });

  return { browser, page };
}

// Scroll page to trigger lazy loading
async function autoScroll(page) {
  await page.evaluate(async (scrollDelay) => {
    await new Promise((resolve) => {
      let totalHeight = 0;
      const distance = 400;
      const timer = setInterval(() => {
        const scrollHeight = document.body.scrollHeight;
        window.scrollBy(0, distance);
        totalHeight += distance;

        if (totalHeight >= scrollHeight) {
          clearInterval(timer);
          resolve();
        }
      }, scrollDelay);
    });
  }, CONFIG.scrollDelay);
}

// Extract products from rendered page using Cheerio
function extractProducts(html) {
  const $ = cheerio.load(html);
  const products = [];

  $('.product-card, [data-product-id]').each((_, el) => {
    const card = $(el);
    const product = {
      id: card.attr('data-product-id') || null,
      name: card.find('.product-name, h3').first().text().trim(),
      price: parseFloat(
        card.find('.price, .product-price').first().text().replace(/[^0-9.]/g, '')
      ) || null,
      rating: parseFloat(card.find('[data-rating]').attr('data-rating')) || null,
      reviewCount: parseInt(
        card.find('.review-count').text().replace(/[^0-9]/g, ''), 10
      ) || 0,
      availability: card.find('.in-stock, .availability').text().trim(),
      imageUrl: card.find('img').attr('data-src') || card.find('img').attr('src') || null,
      productUrl: card.find('a[href]').first().attr('href') || null,
    };

    if (product.name) products.push(product);
  });

  return products;
}

// Handle pagination — click next button or detect end
async function goToNextPage(page, currentPage) {
  try {
    // Strategy 1: Look for a "Next" button
    const nextButton = await page.$('button.next-page, a.next-page, [aria-label="Next page"]');
    if (nextButton) {
      const isDisabled = await page.evaluate(
        (btn) => btn.disabled || btn.classList.contains('disabled'),
        nextButton
      );
      if (isDisabled) return false;

      await Promise.all([
        page.waitForNavigation({ waitUntil: 'networkidle2', timeout: CONFIG.navigationTimeout }),
        nextButton.click(),
      ]);
      return true;
    }

    // Strategy 2: URL-based pagination
    const url = new URL(page.url());
    url.searchParams.set('page', currentPage + 1);
    await page.goto(url.href, {
      waitUntil: 'networkidle2',
      timeout: CONFIG.navigationTimeout,
    });

    // Check if page actually has new content
    const hasProducts = await page.$('.product-card, [data-product-id]');
    return !!hasProducts;
  } catch (error) {
    console.error(`Pagination failed at page ${currentPage}: ${error.message}`);
    return false;
  }
}

// Main scraper
async function scrapeProducts() {
  const { browser, page } = await createBrowser();
  const allProducts = [];
  let currentPage = 1;

  try {
    console.log(`Navigating to ${CONFIG.targetUrl}...`);
    await page.goto(CONFIG.targetUrl, {
      waitUntil: 'networkidle2',
      timeout: CONFIG.navigationTimeout,
    });

    // Wait for initial content to render
    await page.waitForSelector('.product-card, [data-product-id]', {
      timeout: 10000,
    });

    while (currentPage <= CONFIG.maxPages) {
      console.log(`Scraping page ${currentPage}...`);

      // Scroll to load lazy content
      await autoScroll(page);

      // Small delay after scrolling
      await new Promise((r) => setTimeout(r, 1000));

      // Get rendered HTML and parse with Cheerio
      const html = await page.content();
      const products = extractProducts(html);

      console.log(`Found ${products.length} products on page ${currentPage}`);
      allProducts.push(...products);

      // Try to navigate to next page
      const hasNext = await goToNextPage(page, currentPage);
      if (!hasNext) {
        console.log('No more pages available.');
        break;
      }

      currentPage++;

      // Respectful delay between pages
      await new Promise((r) => setTimeout(r, CONFIG.pageDelay));
    }

    // Deduplicate by product ID
    const uniqueProducts = [...new Map(
      allProducts
        .filter((p) => p.id)
        .map((p) => [p.id, p])
    ).values()];

    // Save results
    await fs.writeFile(CONFIG.outputFile, JSON.stringify(uniqueProducts, null, 2));
    console.log(`Saved ${uniqueProducts.length} unique products to ${CONFIG.outputFile}`);

    return uniqueProducts;
  } catch (error) {
    console.error(`Scraping failed: ${error.message}`);
    // Save partial results
    if (allProducts.length > 0) {
      await fs.writeFile(CONFIG.outputFile, JSON.stringify(allProducts, null, 2));
      console.log(`Saved ${allProducts.length} partial results`);
    }
    throw error;
  } finally {
    await browser.close();
  }
}

scrapeProducts().catch(console.error);

Notice how this scraper uses Cheerio inside the Puppeteer workflow. After Puppeteer renders the page and triggers lazy loading through scrolling, the rendered HTML is passed to Cheerio for fast, reliable data extraction. This hybrid pattern avoids complex Puppeteer DOM queries and leverages Cheerio’s efficient CSS selector engine instead.

Anti-Detection Techniques in Detail

Modern websites employ sophisticated bot detection systems. The anti-detection measures in the Puppeteer example above address the most common detection vectors, but understanding why they work helps you adapt to new challenges.

Browser Fingerprint Masking

Detection systems check the navigator.webdriver property, which Puppeteer sets to true by default. Overriding this property is the first line of defense. Similarly, headless Chrome exposes specific properties through the window.chrome object that differ from regular Chrome. The script mimics a real browser’s properties to avoid triggering these checks.

Request Interception

Blocking images, stylesheets, and fonts serves dual purposes. It dramatically speeds up page loads (often by 60-80%), and it reduces the number of requests your scraper makes — each of which is an opportunity for detection. However, be cautious: some anti-bot systems monitor whether resources are loaded and flag sessions that skip them.

Behavioral Mimicry

Beyond fingerprinting, advanced systems analyze browsing behavior. Adding randomized delays between actions, scrolling at variable speeds, and occasionally moving the mouse cursor all help your scraper appear more human. Some teams go further by recording real user sessions and replaying timing patterns.

Choosing Between Puppeteer and Cheerio

The decision tree is straightforward. Use Cheerio when the target page serves content in the initial HTML response, when you need to process thousands of pages quickly, when server resources are limited, or when you are building a pipeline that processes HTML from multiple sources (files, APIs, databases). Use Puppeteer when the target uses client-side rendering (React, Vue, Angular), when you need to interact with the page (login, click, scroll), when content loads dynamically via AJAX or WebSocket, or when you need to handle CAPTCHAs or anti-bot challenges. Use both together when you need Puppeteer’s rendering but Cheerio’s parsing speed, when scraping SPAs with complex DOM structures, or when building a pipeline that handles both static and dynamic sources.

Error Handling and Resilience

Production scrapers must handle failures gracefully. Network timeouts, changed selectors, rate limiting, and unexpected page structures are daily realities. Building resilience into your scraper from the start saves hours of debugging later.

Implementing Circuit Breakers

If a target server returns errors repeatedly, continuing to hammer it is counterproductive. A circuit breaker pattern stops requests after a threshold of failures and resumes after a cooldown period. This protects both your scraper and the target server — an important ethical consideration covered in our OWASP web security guide.

Selector Resilience

Hardcoded CSS selectors are the most fragile part of any scraper. When the target website updates its markup, your selectors break. Mitigate this by using multiple fallback selectors, preferring data attributes over class names, validating extracted data against expected formats, and setting up monitoring alerts for when extraction yields change significantly.

Session Management

For long-running scraping jobs, managing browser sessions and cookies is critical. Puppeteer allows you to persist cookies between sessions, reuse authentication tokens, and rotate user agents. Storing session state in a file or database lets you resume interrupted scraping jobs without starting from scratch.

Performance Optimization

Scraping performance directly impacts project timelines and infrastructure costs. There are several strategies to maximize throughput. For more general performance tips, see our web performance optimization guide.

Connection Pooling

For Cheerio-based scrapers, reusing HTTP connections through an Axios instance with keepAlive: true reduces TCP handshake overhead. For Puppeteer, reusing browser instances across multiple page navigations avoids the expensive startup cost of launching Chromium.

Concurrent Scraping

Running multiple scraping tasks in parallel dramatically increases throughput. With Cheerio, you can use Promise.allSettled() to process batches of URLs concurrently. With Puppeteer, open multiple tabs or even multiple browser instances, though watch your memory usage. A concurrency limit of 5-10 simultaneous pages per Puppeteer instance is a reasonable starting point.

Caching and Deduplication

Cache raw HTML responses to avoid re-fetching pages during development and debugging. Use content hashes or unique identifiers to deduplicate results. A simple SQLite database or Redis instance works well for tracking which URLs have been processed.

Ethical and Legal Considerations

Web scraping occupies a complex legal landscape. While the technical ability to scrape exists, developers should always consider the ethical and legal implications of their work.

Always check and respect robots.txt directives. Implement rate limiting to avoid overloading target servers — treat scraping like consuming an API, as we discuss in our REST APIs introduction. Identify your scraper in its User-Agent string when scraping cooperatively. Review the target website’s terms of service for explicit prohibitions. Consider whether the data you are collecting is personally identifiable and subject to privacy regulations like GDPR.

For teams building scraping infrastructure for business intelligence or competitive analysis, tools like Taskee can help organize scraping projects with task management workflows that track data collection milestones and team responsibilities. For agencies managing scraping projects across multiple clients, Toimi provides project planning features that help coordinate technical data collection efforts with broader business objectives.

Deploying Scrapers to Production

Moving from a local development script to a production scraping system requires additional infrastructure. Consider containerization for consistent execution environments, scheduled execution via cron jobs or task queues, centralized logging and monitoring, proxy rotation for large-scale operations, and data storage pipelines that validate and transform scraped data.

Cloud platforms like AWS Lambda or Google Cloud Functions can run Cheerio-based scrapers serverlessly, keeping costs proportional to usage. Puppeteer requires more substantial compute resources — a container orchestration platform like Kubernetes or a dedicated VM is more appropriate for headless browser workloads.

Common Pitfalls and How to Avoid Them

Experience teaches hard lessons in web scraping. Here are the most common mistakes and their solutions.

Not handling encoding correctly. Some pages serve content in non-UTF-8 encodings. Always check the Content-Type header and convert encoding before parsing with Cheerio.

Ignoring relative URLs. Links and image sources often use relative paths. Always resolve them against the page’s base URL using the URL constructor.

Memory leaks in long-running Puppeteer sessions. Each new page and context consumes memory. Close pages and contexts when done, and periodically restart the browser instance for jobs that process thousands of pages.

Hardcoding wait times instead of using intelligent waits. Instead of page.waitForTimeout(5000), use page.waitForSelector() or page.waitForNetworkIdle(). Fixed waits are either too long (wasting time) or too short (causing failures).

Not validating extracted data. Always check that extracted values match expected formats. A price field containing “Add to Cart” means your selector is wrong. Build validation into your parsing logic from day one.

FAQ