Python Web Scraping: The Complete Guide (requests + BeautifulSoup + Playwright)

Web scraping in Python lets you extract data from websites automatically. This guide covers the full stack: static pages with requests + BeautifulSoup, JavaScript-rendered sites with Playwright, full crawlers with Scrapy, and the anti-bot countermeasures you need to know.

Quick reference

Task	Tool	Code
Fetch HTML	requests	`r = requests.get(url)`
Parse HTML	BeautifulSoup	`soup = BeautifulSoup(r.text, "html.parser")`
Select by CSS	BS4	`soup.select("div.price")`
First match	BS4	`soup.find("h1")`
All matches	BS4	`soup.find_all("a", class_="link")`
Get attribute	BS4	`tag["href"]`
Get text	BS4	`tag.get_text(strip=True)`
JavaScript site	Playwright	`page.goto(url); page.content()`
Full crawler	Scrapy	`scrapy crawl myspider`
Parse JSON API	requests	`r.json()`
Set headers	requests	`requests.get(url, headers={...})`
Session cookies	requests	`s = requests.Session()`

Installation

# Core scraping stack
pip install requests beautifulsoup4 lxml

# For JavaScript-rendered pages
pip install playwright
playwright install chromium

# Full-featured crawler
pip install scrapy

Fetching pages with requests

import requests

# Basic GET
r = requests.get("https://example.com")
r.raise_for_status()          # raises HTTPError on 4xx/5xx
print(r.status_code)          # 200
print(r.text[:500])           # raw HTML
print(r.headers["Content-Type"])

# Always set a User-Agent (bare Python gets blocked easily)
headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/125.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}
r = requests.get("https://example.com", headers=headers, timeout=10)

# POST with form data
r = requests.post("https://httpbin.org/post", data={"q": "python"})

# POST with JSON body
r = requests.post("https://api.example.com/search",
                  json={"query": "python"},
                  headers={"Authorization": "Bearer TOKEN"})
data = r.json()

# Session — shares cookies across requests (login flows)
s = requests.Session()
s.headers.update(headers)
s.post("https://example.com/login", data={"user": "me", "pass": "secret"})
r = s.get("https://example.com/dashboard")   # authenticated

Parsing HTML with BeautifulSoup

from bs4 import BeautifulSoup

html = """
<html>
  <body>
    <h1 class="title">Python Scraping</h1>
    <ul id="results">
      <li class="item"><a href="/1">Item 1</a><span class="price">$10</span></li>
      <li class="item"><a href="/2">Item 2</a><span class="price">$20</span></li>
    </ul>
    <p data-id="42">Footer paragraph</p>
  </body>
</html>
"""

soup = BeautifulSoup(html, "lxml")   # lxml is faster than html.parser

# --- Finding elements ---
h1 = soup.find("h1")                         # first <h1>
h1 = soup.find("h1", class_="title")         # with class
h1 = soup.find("h1", {"class": "title"})     # dict form
items = soup.find_all("li", class_="item")   # list of all matches
first = soup.find_all("li", limit=1)         # stop after N

# CSS selectors (most powerful)
items = soup.select("ul#results li.item")    # descendant
prices = soup.select("li.item span.price")
link = soup.select_one("li.item a")          # first match only

# --- Extracting data ---
print(h1.get_text())                 # "Python Scraping"
print(h1.get_text(strip=True))       # stripped whitespace
print(link["href"])                  # "/1"
print(link.get("href", ""))          # safe, returns "" if missing
print(link.string)                   # "Item 1" (direct text child only)

# Attribute selectors
para = soup.find("p", attrs={"data-id": "42"})

# Navigate the tree
ul = soup.find("ul")
first_li = ul.find("li")
next_li = first_li.find_next_sibling("li")
parent = first_li.parent               # <ul>

# Extract structured data from a list
results = []
for li in soup.select("li.item"):
    results.append({
        "name": li.select_one("a").get_text(strip=True),
        "url": li.select_one("a")["href"],
        "price": li.select_one(".price").get_text(strip=True),
    })
print(results)
# [{'name': 'Item 1', 'url': '/1', 'price': '$10'}, ...]

Handling pagination

import requests
from bs4 import BeautifulSoup
import time

BASE_URL = "https://books.toscrape.com/catalogue"

def scrape_all_pages():
    url = f"{BASE_URL}/page-1.html"
    all_books = []

    while url:
        r = requests.get(url, timeout=10)
        soup = BeautifulSoup(r.text, "lxml")

        for article in soup.select("article.product_pod"):
            all_books.append({
                "title": article.h3.a["title"],
                "price": article.select_one(".price_color").get_text(strip=True),
                "rating": article.p["class"][1],   # "Three", "Five", ...
            })

        # Find "next" button
        next_btn = soup.select_one("li.next a")
        if next_btn:
            url = f"{BASE_URL}/{next_btn['href']}"
            time.sleep(1)   # be polite — 1 second delay between pages
        else:
            url = None

    return all_books

books = scrape_all_pages()
print(f"Scraped {len(books)} books")

JSON APIs (the easy win)

Many sites load data via XHR/fetch. Check the browser Network tab first — you may be able to hit the API directly, no HTML parsing needed.

import requests

# Intercept the API call from browser DevTools → Network → XHR
r = requests.get(
    "https://api.example.com/products",
    params={"page": 1, "limit": 50, "category": "books"},
    headers={"User-Agent": "Mozilla/5.0", "Referer": "https://example.com"},
)
data = r.json()   # dict/list — no BeautifulSoup needed

products = data["results"]
for p in products:
    print(p["title"], p["price"])

JavaScript-rendered pages with Playwright

Use Playwright when the content is injected by JavaScript (React/Vue/Angular SPAs, infinite scroll, lazy-load).

from playwright.sync_api import sync_playwright
import time

def scrape_spa(url: str) -> list[dict]:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()

        # Block images and fonts to speed up scraping
        page.route("**/*.{png,jpg,gif,svg,woff,woff2}", lambda r: r.abort())

        page.goto(url, wait_until="networkidle")   # wait for JS to finish

        # Wait for a specific element to appear
        page.wait_for_selector("div.product-card", timeout=10_000)

        # Scroll to trigger lazy-load
        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        time.sleep(2)

        # Extract data using Playwright locators
        cards = page.locator("div.product-card")
        results = []
        for i in range(cards.count()):
            card = cards.nth(i)
            results.append({
                "title": card.locator("h2").inner_text(),
                "price": card.locator(".price").inner_text(),
                "link": card.locator("a").get_attribute("href"),
            })

        browser.close()
        return results

# Async version (better for scraping many pages in parallel)
from playwright.async_api import async_playwright
import asyncio

async def scrape_async(urls: list[str]):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        results = []

        for url in urls:
            page = await browser.new_page()
            await page.goto(url)
            title = await page.title()
            results.append({"url": url, "title": title})
            await page.close()

        await browser.close()
        return results

data = asyncio.run(scrape_async(["https://example.com", "https://python.org"]))

Scrapy — production crawler

Scrapy is ideal for large-scale crawls: built-in concurrency, middleware, pipelines, and retry logic.

scrapy startproject myproject
cd myproject
scrapy genspider books books.toscrape.com

# myproject/spiders/books.py
import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com"]

    def parse(self, response):
        # Extract items on this page
        for article in response.css("article.product_pod"):
            yield {
                "title": article.css("h3 a::attr(title)").get(),
                "price": article.css(".price_color::text").get(),
                "rating": article.css("p.star-rating::attr(class)").get(),
            }

        # Follow "next" pagination link
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

# Run and export to JSON
scrapy crawl books -o output.json

# Export to CSV
scrapy crawl books -o output.csv

Scrapy settings (settings.py):

DOWNLOAD_DELAY = 1          # seconds between requests per domain
CONCURRENT_REQUESTS = 8     # parallel requests
AUTOTHROTTLE_ENABLED = True # adaptive rate limiting
ROBOTSTXT_OBEY = True       # respect robots.txt
USER_AGENT = "Mozilla/5.0 ..."

Handling anti-bot measures

Measure	What it does	Bypass
User-Agent check	Blocks "python-requests/x.x" UA	Set realistic UA
Rate limiting	429 Too Many Requests	`time.sleep(1–3)` between requests
IP blocking	Bans your IP after N requests	Rotate proxies
Cloudflare JS challenge	Requires JS execution	Use Playwright + `cloudscraper`
CAPTCHAs	reCAPTCHA / hCaptcha	2captcha API or avoid
Login wall	Requires authentication	`requests.Session` + cookies
Dynamic tokens	CSRF token in form	Extract from HTML before POST

import requests
import time
import random

# Rotate User-Agents
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ...",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ...",
]

# Polite scraping with jitter delay
def polite_get(url: str, session: requests.Session) -> requests.Response:
    session.headers["User-Agent"] = random.choice(USER_AGENTS)
    time.sleep(random.uniform(1.0, 3.0))   # random delay 1–3 seconds
    return session.get(url, timeout=15)

# Retry with exponential backoff
import time

def get_with_retry(url: str, max_retries: int = 3) -> requests.Response:
    for attempt in range(max_retries):
        try:
            r = requests.get(url, timeout=10,
                             headers={"User-Agent": random.choice(USER_AGENTS)})
            r.raise_for_status()
            return r
        except requests.RequestException as e:
            if attempt == max_retries - 1:
                raise
            wait = 2 ** attempt   # 1s, 2s, 4s
            time.sleep(wait)

# Extract CSRF token before form submission
def login(session: requests.Session, login_url: str,
          username: str, password: str):
    r = session.get(login_url)
    soup = BeautifulSoup(r.text, "lxml")
    csrf = soup.find("input", {"name": "csrf_token"})["value"]
    session.post(login_url, data={
        "username": username,
        "password": password,
        "csrf_token": csrf,
    })

Saving scraped data

import json
import csv
from pathlib import Path

data = [
    {"title": "Item 1", "price": "$10"},
    {"title": "Item 2", "price": "$20"},
]

# JSON
Path("output.json").write_text(json.dumps(data, indent=2, ensure_ascii=False))

# CSV
with open("output.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["title", "price"])
    writer.writeheader()
    writer.writerows(data)

# pandas (for analysis)
import pandas as pd
df = pd.DataFrame(data)
df.to_csv("output.csv", index=False)
df.to_excel("output.xlsx", index=False)
df.to_parquet("output.parquet")    # efficient for large datasets

Practical patterns

Scrape only new items (incremental):

import json
from pathlib import Path

def load_seen_urls(path="seen.json") -> set:
    if Path(path).exists():
        return set(json.loads(Path(path).read_text()))
    return set()

def save_seen_urls(urls: set, path="seen.json"):
    Path(path).write_text(json.dumps(list(urls)))

seen = load_seen_urls()
new_items = []

for item in scraped_items:
    if item["url"] not in seen:
        new_items.append(item)
        seen.add(item["url"])

save_seen_urls(seen)
print(f"{len(new_items)} new items found")

Parse tables directly:

import pandas as pd

# pandas reads <table> HTML directly — no BeautifulSoup needed
tables = pd.read_html("https://en.wikipedia.org/wiki/List_of_countries_by_GDP")
df = tables[2]   # pick the right table by index
print(df.head())

Common mistakes

Mistake	Problem	Fix
No `raise_for_status()`	Silently processes 404/500 responses	Call it after every request
No timeout	Hangs forever on slow servers	`timeout=10`
No delay	Gets IP-banned immediately	`time.sleep(1–3)`
Parsing with `str.find()` / regex on HTML	Fragile, breaks on whitespace changes	Use BeautifulSoup CSS selectors
Storing raw HTML in DB	Expensive, hard to query	Extract and store structured fields
Scraping JS-rendered pages with requests	Gets empty body	Use Playwright
Using `r.text` for binary (PDF/images)	Corrupts data	Use `r.content` (bytes)
No `User-Agent` header	Returns 403/blocks	Set realistic UA string

Library comparison

Library	Use case	JS support	Speed	Learning curve
requests	HTTP client	✗	Fast	Low
BeautifulSoup4	Parse HTML	✗	Medium	Low
lxml	Parse HTML/XML	✗	Fastest	Medium
Scrapy	Full crawler	✗	Fast (async)	High
Playwright	Browser automation	✓	Slow	Medium
Selenium	Browser automation	✓	Slow	Medium
httpx	Async HTTP client	✗	Fast	Low
Parsel	CSS/XPath selectors	✗	Fast	Low

FAQ

Is web scraping legal?
It depends: scraping publicly available data is generally legal, but violating a site's robots.txt, ToS, or scraping behind authentication may be legally and ethically problematic. Always check robots.txt at /robots.txt and respect Crawl-delay directives.

requests vs httpx — what's the difference?
httpx is an async-capable HTTP client that mirrors the requests API. Use httpx when you need async scraping without Playwright's browser overhead:

import asyncio, httpx

async def fetch_all(urls):
    async with httpx.AsyncClient() as client:
        tasks = [client.get(u) for u in urls]
        return await asyncio.gather(*tasks)

How do I scrape a site that requires JavaScript?
Use Playwright (playwright install chromium, then page.goto(url)). Alternatively, inspect the Network tab in DevTools — many "JS-only" sites actually call a JSON API you can hit directly with requests.

How do I avoid getting blocked?
Rotate User-Agents, add random delays (time.sleep(random.uniform(1, 3))), respect robots.txt, use a session to mimic a real browser, and consider rotating residential proxies for large-scale scraping.

How do I handle infinite scroll?
With Playwright, scroll to the bottom and wait for new content to load:

prev_count = 0
while True:
    page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    page.wait_for_timeout(2000)
    count = page.locator(".item").count()
    if count == prev_count:
        break   # no new items loaded
    prev_count = count

How do I speed up scraping?

Check if there's a JSON API (no parsing needed).
Use requests.Session (reuses TCP connections).
Use asyncio + httpx for I/O-bound parallel requests.
Block images/fonts in Playwright.
Use Scrapy for large crawls (built-in async concurrency).