How do you handle infinite scroll with Playwright?

Use a scroll loop: scroll to the bottom with page.evaluate(() => window.scrollTo(0, document.body.scrollHeight)), wait for new content with waitForTimeout or waitForResponse, check if new items loaded by comparing element counts, and repeat until no new items appear. Always add a maximum iteration limit to prevent infinite loops.

Is web scraping with Playwright legal?

Web scraping legality depends on what you scrape, how you use the data, and the website's terms of service. Generally: scraping publicly available data for personal use or research is legal in most jurisdictions. Always check the website's robots.txt and ToS, don't overload servers with too many requests, respect rate limits, and never scrape personal data without consent (GDPR/CCPA). When in doubt, consult legal advice.

Playwright Web Scraping Tutorial: Complete Guide 2026

Q: Can Playwright be used for web scraping?

Yes. Playwright is one of the best tools for web scraping in 2026 because it controls a real browser that renders JavaScript, handles dynamic content, SPAs, infinite scroll, and authentication flows. Unlike static HTTP scrapers (like requests or cheerio), Playwright executes JavaScript exactly as a user's browser does, so it can scrape any content visible to a user.

Q: Can Playwright bypass anti-bot detection?

Playwright alone does not bypass anti-bot systems like Cloudflare or DataDome. However, you can reduce detection by using headed mode, setting realistic user agents and viewports, adding random delays between actions, using stealth plugins (playwright-extra with stealth plugin), and routing through residential proxies. For heavy-duty anti-bot bypass, use dedicated services like ScraperAPI or ZenRows.

Traditional HTTP-based scrapers (like fetch + cheerio) fail on modern websites because the content is rendered by JavaScript after the initial HTML loads. Single-page applications, lazy-loaded images, infinite scroll feeds, and client-side routing all require a real browser to render the page before you can extract data.

Playwright gives you that real browser — headless, fast, and programmable. If you're new to Playwright, our beginner's guide covers the fundamentals. Let's build a production-ready scraper from scratch.

Ethics first: Always check a website's robots.txt and Terms of Service before scraping. Respect rate limits, don't overload servers, and never scrape personal data without consent. This tutorial teaches the technique — use it responsibly.

Why Playwright for Web Scraping?

Renders JavaScript — SPAs, React/Vue/Angular apps, dynamic content all render correctly
Auto-waiting — waits for elements to appear before interacting, reducing timing issues
Network interception — capture API responses directly instead of parsing HTML
Multi-browser — Chromium, Firefox, or WebKit (some sites render differently)
Headless + headed mode — run headless for speed, headed for debugging
Built-in screenshots — capture visual proof of scraped pages
TypeScript native — full type safety for your scraping scripts (see our Playwright TypeScript tutorial)

Setup: Your First Scraper

Terminal

# Create a new project
mkdir playwright-scraper && cd playwright-scraper
npm init -y
npm install playwright

# Install browser binaries
npx playwright install chromium

TypeScript — scraper.ts (your first scraper)

import { chromium } from 'playwright';

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  await page.goto('https://news.ycombinator.com');

  // Extract all story titles and links
  const stories = await page.$$eval('.titleline > a', (links) =>
    links.map((a) => ({
      title: a.textContent,
      url: a.href,
    }))
  );

  console.log(JSON.stringify(stories, null, 2));
  await browser.close();
})();

Terminal — Run it

npx tsx scraper.ts

That's it — 15 lines of code to scrape Hacker News. You can also use Playwright Codegen to record browser interactions and generate scraping scripts visually. Now let's handle the real-world scenarios.

Scraping Dynamic JavaScript Pages

Most modern websites load content after the initial HTML. Playwright handles this automatically because it runs a real browser, but you need to wait for the right content before extracting:

TypeScript — Wait for dynamic content

await page.goto('https://example.com/products');

// Wait for product cards to load (they're rendered by JavaScript)
await page.waitForSelector('.product-card', { state: 'visible' });

// Or wait for a specific API response that populates the page
await page.waitForResponse((res) =>
  res.url().includes('/api/products') && res.status() === 200
);

// Now extract the data
const products = await page.$$eval('.product-card', (cards) =>
  cards.map((card) => ({
    name: card.querySelector('h3')?.textContent?.trim(),
    price: card.querySelector('.price')?.textContent?.trim(),
    image: card.querySelector('img')?.getAttribute('src'),
  }))
);

Handling Infinite Scroll

Social media feeds, product listings, and news sites use infinite scroll. Here's a reliable pattern:

TypeScript — Infinite scroll scraper

async function scrapeInfiniteScroll(page: Page, itemSelector: string, maxItems = 100) {
  let items: string[] = [];
  let previousHeight = 0;
  let retries = 0;

  while (items.length < maxItems && retries < 5) {
    // Scroll to bottom
    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));

    // Wait for new content to load
    await page.waitForTimeout(1500);

    // Check if new content appeared
    const currentHeight = await page.evaluate(() => document.body.scrollHeight);

    if (currentHeight === previousHeight) {
      retries++;  // No new content loaded
    } else {
      retries = 0;
      previousHeight = currentHeight;
    }

    // Extract items so far
    items = await page.$$eval(itemSelector, (els) =>
      els.map((el) => el.textContent?.trim() ?? '')
    );
  }

  return items.slice(0, maxItems);
}

Pro tip: Instead of scrolling and parsing HTML, intercept the API call that loads more data. Most infinite scroll pages make XHR/fetch requests — capture the JSON response directly with page.route(). It's faster and more reliable.

The Power Move: Intercept API Responses

Many websites load data from APIs and render it with JavaScript. Instead of parsing the rendered HTML, intercept the API response directly — a technique also covered in depth in our Playwright API testing guide. You get clean, structured JSON without any HTML parsing:

TypeScript — Intercept API responses

const products: Product[] = [];

// Listen for API responses before navigating
page.on('response', async (response) => {
  if (response.url().includes('/api/products') && response.status() === 200) {
    const data = await response.json();
    products.push(...data.items);
  }
});

await page.goto('https://example.com/products');
await page.waitForLoadState('networkidle');

console.log(`Captured ${products.length} products from API`);

This technique is 10x more reliable than HTML parsing because you get the exact data the application uses, with proper types, IDs, and relationships.

Scraping Behind Authentication

Need to scrape pages that require login? Playwright makes this straightforward:

TypeScript — Login then scrape

// Log in once
await page.goto('https://example.com/login');
await page.getByLabel('Email').fill('your@email.com');
await page.getByLabel('Password').fill('yourPassword');
await page.getByRole('button', { name: 'Sign In' }).click();
await page.waitForURL('**/dashboard');

// Save session for reuse (no re-login needed)
await page.context().storageState({ path: 'auth.json' });

// Later: reuse the saved session
const context = await browser.newContext({ storageState: 'auth.json' });
const page2 = await context.newPage();
await page2.goto('https://example.com/private-data');
// Already logged in — scrape away

Handling Pagination

TypeScript — Click-through pagination

const allProducts: Product[] = [];

while (true) {
  // Extract data from current page
  const pageProducts = await page.$$eval('.product-card', (cards) =>
    cards.map((c) => ({
      name: c.querySelector('h3')?.textContent?.trim() ?? '',
      price: c.querySelector('.price')?.textContent?.trim() ?? '',
    }))
  );
  allProducts.push(...pageProducts);

  // Try to click "Next" button
  const nextBtn = page.getByRole('link', { name: 'Next' });
  if (await nextBtn.isVisible()) {
    await nextBtn.click();
    await page.waitForLoadState('networkidle');
  } else {
    break;  // No more pages
  }
}

console.log(`Scraped ${allProducts.length} products across all pages`);

Taking Screenshots & PDFs

TypeScript

// Full page screenshot
await page.screenshot({ path: 'page.png', fullPage: true });

// Screenshot of a specific element
await page.locator('.product-card').first().screenshot({ path: 'product.png' });

// Generate PDF (Chromium only)
await page.pdf({
  path: 'report.pdf',
  format: 'A4',
  printBackground: true,
  margin: { top: '1cm', bottom: '1cm' },
});

Avoiding Anti-Bot Detection

Websites use anti-bot systems (Cloudflare, DataDome, PerimeterX) that detect automation. Playwright alone doesn't bypass these, but you can reduce detection:

1. Use realistic browser settings

TypeScript — Stealth configuration

const browser = await chromium.launch({ headless: false }); // headed = less detection

const context = await browser.newContext({
  userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
  viewport: { width: 1920, height: 1080 },
  locale: 'en-US',
  timezoneId: 'America/New_York',
  geolocation: { latitude: 40.7128, longitude: -74.006 },
  permissions: ['geolocation'],
});

2. Add human-like delays

TypeScript

// Random delay between actions (500-2000ms)
function randomDelay(min = 500, max = 2000) {
  return new Promise((resolve) =>
    setTimeout(resolve, Math.floor(Math.random() * (max - min) + min))
  );
}

await page.goto('https://example.com');
await randomDelay();
await page.getByRole('link', { name: 'Products' }).click();
await randomDelay();

3. Use stealth plugins

Terminal

npm install playwright-extra puppeteer-extra-plugin-stealth

TypeScript

import { chromium } from 'playwright-extra';
import stealth from 'puppeteer-extra-plugin-stealth';

chromium.use(stealth());

const browser = await chromium.launch();
// navigator.webdriver is now hidden, WebGL fingerprint randomized, etc.

Important: No tool guarantees bypass of all anti-bot systems. For heavy-duty scraping at scale, consider dedicated proxy services (ScraperAPI, ZenRows, BrightData) that handle anti-bot detection, CAPTCHAs, and IP rotation for you.

Saving Scraped Data

TypeScript — Save to JSON and CSV

import { writeFileSync } from 'fs';

// Save as JSON
writeFileSync('products.json', JSON.stringify(products, null, 2));

// Save as CSV
const csv = [
  'Name,Price,URL',
  ...products.map((p) => `"${p.name}","${p.price}","${p.url}"`),
].join('\n');
writeFileSync('products.csv', csv);

Production Scraper Pattern

Here's a complete, production-ready scraper with error handling, retries, and data output:

TypeScript — production-scraper.ts

import { chromium, type Page } from 'playwright';
import { writeFileSync } from 'fs';

type Product = { name: string; price: string; url: string };

async function scrapePage(page: Page, url: string): Promise<Product[]> {
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  await page.waitForSelector('.product-card', { timeout: 10000 });

  return page.$$eval('.product-card', (cards) =>
    cards.map((card) => ({
      name: card.querySelector('h3')?.textContent?.trim() ?? '',
      price: card.querySelector('.price')?.textContent?.trim() ?? '',
      url: card.querySelector('a')?.href ?? '',
    }))
  );
}

async function main() {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  const allProducts: Product[] = [];

  const urls = [
    'https://example.com/products?page=1',
    'https://example.com/products?page=2',
    'https://example.com/products?page=3',
  ];

  for (const url of urls) {
    try {
      const products = await scrapePage(page, url);
      allProducts.push(...products);
      console.log(`Scraped ${products.length} from ${url}`);
    } catch (err) {
      console.error(`Failed: ${url}`, err);
    }
    // Respectful delay between requests
    await page.waitForTimeout(1000 + Math.random() * 2000);
  }

  writeFileSync('products.json', JSON.stringify(allProducts, null, 2));
  console.log(`Done! ${allProducts.length} products saved.`);
  await browser.close();
}

main();

Performance Tips

Block unnecessary resources — images, fonts, CSS slow down scraping
Use domcontentloaded instead of load — don't wait for every image
Intercept API responses instead of parsing HTML when possible
Reuse browser contexts — creating a new browser per page is expensive
Run headless — skip UI rendering for 2–3x speed boost

TypeScript — Block images and CSS for speed

await page.route('**/*.{png,jpg,jpeg,gif,svg,webp,css,woff,woff2}', (route) =>
  route.abort()
);

// Now page loads only HTML + JS — much faster
await page.goto('https://example.com/products');

Frequently Asked Questions

Can Playwright be used for web scraping?

Yes. Playwright is one of the best web scraping tools in 2026 because it controls a real browser that renders JavaScript, handles dynamic content, SPAs, infinite scroll, and authentication. It can scrape any content visible to a user.

Is Playwright better than Selenium for scraping?

Yes for most use cases. Playwright is faster (native browser protocols), has auto-waiting, supports all three browser engines, and provides built-in network interception. Read the full comparison.

How do you handle infinite scroll?

Use a scroll loop: scroll to bottom, wait for content, check if new items loaded, repeat. Better yet, intercept the API response that loads more data — it's faster and more reliable than parsing HTML.

Can Playwright bypass anti-bot detection?

Playwright alone doesn't bypass anti-bot systems. Reduce detection with headed mode, realistic user agents, random delays, and stealth plugins. For heavy-duty scraping, use dedicated proxy services.

Is web scraping legal?

Legality depends on what you scrape, how you use the data, and the website's ToS. Public data for personal use is generally legal. Always check robots.txt, respect rate limits, and never scrape personal data without consent.

Meet Your Instructor

Asim Noaman

Senior QA Automation Engineer & AI Testing Specialist

With years of hands-on experience building test automation frameworks for production applications, Asim specializes in combining traditional QA methodologies with cutting-edge AI tools. He has helped teams adopt Playwright and AI-driven testing workflows to ship faster with fewer bugs.

Udemy Instructor Published course author

Playwright + AI Expert Specialized in AI-powered QA

Production Experience Enterprise-grade frameworks

Connect on LinkedIn

Playwright Web Scraping Tutorial: Scrape Dynamic Pages with TypeScript (2026)

Why Playwright for Web Scraping?

Setup: Your First Scraper

Scraping Dynamic JavaScript Pages

Handling Infinite Scroll

The Power Move: Intercept API Responses

Scraping Behind Authentication

Handling Pagination

Taking Screenshots & PDFs

Avoiding Anti-Bot Detection

1. Use realistic browser settings

2. Add human-like delays

3. Use stealth plugins

Saving Scraped Data

Production Scraper Pattern

Performance Tips

Frequently Asked Questions

Can Playwright be used for web scraping?

Is Playwright better than Selenium for scraping?

How do you handle infinite scroll?

Can Playwright bypass anti-bot detection?

Is web scraping legal?

Asim Noaman