What Is Web Scraping? How It Works, Best Tools, Costs & Legal Risks
Every price tag, job post and property listing you see online began as a web page built for human eyes. Web scraping is how software reads those same pages at scale and turns them into rows in a spreadsheet. It powers price tracking, academic research and market intelligence. It also sits in contested legal territory: LinkedIn and hiQ Labs fought over it in court for years, and European regulators opened a public consultation on new web scraping guidelines in July 2026. This guide explains how scraping works, what it really costs and where the risks are, so you can decide with your eyes open.
TL;DR
Web scraping is automated extraction of data from web pages into a structured format such as CSV or JSON. Crawling discovers pages, and an API is a data feed the site owner offers on purpose.
The core pipeline is: fetch or render the page, parse the HTML, extract fields, clean and validate them, store them, then schedule and monitor the job.
Free libraries such as Requests, Beautiful Soup, Scrapy, Playwright and Selenium cost nothing to license, but you still pay for developer time, servers and maintenance.
Paid tools bill in different units (credits, compute units, successful requests, records or monthly tasks), so compare cost per usable record instead of the headline price.
Legality depends on the facts: public or login-gated access, site terms, copyright and personal data. Public visibility is not permission, and robots.txt is not a legal license either way.
What Is Web Scraping?
Web scraping is the automated collection of data from web pages. A program requests a page, reads its HTML, pulls out chosen fields, cleans them, and saves them as structured data such as CSV or JSON. Whether a given scrape is lawful depends on how the data is accessed, the site's terms, copyright, and privacy law.
Table of Contents
What Is Web Scraping?
Web scraping is the automated process of collecting information from web pages and converting it into structured data you can analyze. A scraper is a program that visits a page, reads the underlying HTML, picks out the pieces you care about and writes them to a file or database.
Here is a concrete example. The practice site Books to Scrape shows each book as a visual card with a title, a price such as £53.74 and a stock label. A scraper turns that card into one table row with three columns: title, price and in-stock status. Repeat that for every card and you have a dataset.
Structured vs. unstructured data
A web page is mostly unstructured: text and layout designed for people. Structured data has labeled fields in a fixed shape, such as a CSV row or a JSON object. Scraping is the bridge between the two, and the quality of that bridge decides whether the result is usable. See our guide to data quality for how teams measure that.
Getting data is not the same as having the right to use it
Being able to fetch a page does not settle whether you may copy, store, republish or sell what is on it. Terms of service, copyright, database rights and privacy law can all apply after the download. We return to this in the legal sections below.
How Does Web Scraping Work? Step by Step
Every scraper, from a ten-line script to an enterprise platform, follows the same chain: source → request or render → parse → extract → clean and validate → store → use. The steps below expand that chain.
Define the scope. Decide exactly which fields, pages and refresh rate you need. A narrow scope lowers cost, risk and breakage.
Check access, permissions and APIs. Look for an official API or data export first, read the site's terms, and read its robots.txt file, which follows the Robots Exclusion Protocol (RFC 9309).
Fetch or render the page. A plain HTTP request returns the raw HTML. Pages that build their content with JavaScript may need a browser engine to render first.
Locate the fields. Use CSS selectors or XPath, two query languages for pointing at elements in the page structure (the DOM).
Parse and extract. A parser such as Beautiful Soup turns HTML into a searchable tree so you can pull out text and attributes.
Normalize. Convert prices, dates and units into one consistent format.
Deduplicate and validate. Remove repeated records and reject rows with missing or impossible values.
Store and export. Write to CSV, JSON or a database. For larger projects, a database management system keeps the data queryable.
Schedule. Run the job daily, weekly or on demand, for example with a cron job or a serverless function (see Function as a Service).
Monitor. Track success rates, field-fill rates and layout changes. Good monitoring borrows ideas from cloud telemetry.
Skipping steps 7 and 10 is the most common reason scraped datasets quietly rot: the job keeps running while the data goes wrong.
Web Scraping vs. Web Crawling vs. APIs
Scraping extracts specific data from pages, crawling discovers and follows links across a site, and an API hands you structured data through an interface the owner designed. Real projects often combine all three.
Aspect | Web scraping | Web crawling | Official API |
Purpose | Extract chosen fields | Discover and index pages | Exchange data on the owner's terms |
Access method | HTTP requests or a browser | Follows links from seed URLs | Authenticated endpoints with documented limits |
Typical output | CSV, JSON, database rows | URL lists, page copies | Structured JSON or XML |
Strength | Works where no API exists | Finds pages at scale | Stable, documented, usually permitted |
Constraints | Breaks when layout changes; legal review needed | Needs politeness and rate control | Quotas, fees, limited fields |
When an API is better: if the site offers one that covers your fields, use it. It is more stable, it comes with explicit terms, and it avoids most layout-change maintenance. Our explainer on the application programming interface (API) covers how they work. Search engines are the classic crawlers; RFC 9309 defines how sites give crawlers instructions.
What Is Web Scraping Used For?
Scraping serves very different goals. Each of these uses has its own safeguards.
Ecommerce: tracking competitor prices, stock and product descriptions. See how retailers apply data in AI in retail.
Market research: gathering reviews, product features and public announcements to compare brands.
Jobs: collecting public job postings to study demand for skills. Candidate-level data is personal data, so handle it carefully, as discussed in AI in hiring.
Property: comparing listings and asking prices, a pattern also seen in machine learning in real estate sales.
Public-interest and academic research: archiving public records, tracking public statements or building research datasets, usually with ethics review.
News and media monitoring: following headlines and mentions across publishers.
Business intelligence: feeding dashboards, lead research (sales intelligence software) and augmented analytics.
Safeguard for personal data: when pages contain names, contact details or photos of people, collect only what you need, document your legal basis, and set a deletion date. The privacy section below explains why.
Types and Methods of Web Scraping
The right method depends on how the target page delivers its data.
Static HTML scraping: the data is in the page source. A simple HTTP request plus a parser is fast and cheap.
API-based extraction: many pages load data from a JSON endpoint behind the scenes. If that endpoint is public and permitted, reading it directly is cleaner than parsing HTML.
JavaScript-heavy pages: content built in the browser by frameworks like React or Vue.js does not appear in the raw HTML. You need rendering.
Headless browsers: tools such as Playwright or Selenium run a real browser without a visible window. They handle dynamic pages but use far more CPU and memory, so each page costs more and runs slower.
Manual and no-code: point-and-click tools let non-programmers build extractors visually.
Cloud-managed: a vendor runs the infrastructure and sometimes the proxies and rendering for you.
One-time vs. scheduled: a one-off export is simple. Recurring jobs need monitoring, retries and storage planning, a pattern close to hyperautomation.
Large-scale extraction: millions of pages demand queues, politeness controls, deduplication and cost tracking.
The trade-off is always reliability versus cost: plain requests are cheapest but fail on dynamic sites, while browser rendering is robust but expensive per page.
How to Build a Simple Web Scraper With Python
This tutorial uses Python with two libraries: Requests to download pages and Beautiful Soup to read them. The practice target is the sandbox store Books to Scrape, which exists for learning. To keep the example reliable, the script runs offline on a built-in sample that copies that site's markup, so no one's server is touched while you learn.
Step 1: Install the packages
pip install requests beautifulsoup4Step 2: Save and run the script
import csv
import re
import time
import requests
from bs4 import BeautifulSoup
SAMPLE_HTML = """
<html><body>
<article class="product_pod">
<h3><a href="/item-1" title="A Light in the Attic">A Light in the ...</a></h3>
<p class="price_color">£51.77</p>
<p class="instock availability">In stock</p>
</article>
<article class="product_pod">
<h3><a href="/item-2" title="Tipping the Velvet">Tipping the Velvet</a></h3>
<p class="price_color">£53.74</p>
<p class="instock availability">In stock</p>
</article>
<article class="product_pod">
<h3><a href="/item-3" title="Soumission">Soumission</a></h3>
<p class="instock availability">Out of stock</p>
</article>
</body></html>
"""
def fetch_html(url):
"""Download one page politely: identify yourself, set a timeout, check the status."""
headers = {"User-Agent": "learning-scraper/0.1 (contact: you@example.com)"}
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()
response.encoding = "utf-8"
time.sleep(1) # pause between requests
return response.text
def parse_products(html):
soup = BeautifulSoup(html, "html.parser")
records = []
for card in soup.select("article.product_pod"):
link = card.select_one("h3 a")
price_tag = card.select_one("p.price_color")
stock_tag = card.select_one("p.availability")
price = None
if price_tag:
match = re.search(r"\d+(?:\.\d+)?", price_tag.get_text())
price = float(match.group()) if match else None
records.append({
"title": link["title"] if link else None,
"price_gbp": price,
"in_stock": "in stock" in stock_tag.get_text().lower() if stock_tag else None,
})
return records
def save_csv(records, path="products.csv"):
valid = [r for r in records if r["title"] and r["price_gbp"] is not None]
with open(path, "w", newline="", encoding="utf-8") as handle:
writer = csv.DictWriter(handle, fieldnames=["title", "price_gbp", "in_stock"])
writer.writeheader()
writer.writerows(valid)
return len(valid), len(records) - len(valid)
if __name__ == "__main__":
html = SAMPLE_HTML # for the practice site: html = fetch_html("https://books.toscrape.com/")
kept, dropped = save_csv(parse_products(html))
print(f"Saved {kept} records, dropped {dropped} incomplete record(s).")Running it prints Saved 2 records, dropped 1 incomplete record(s). and creates a two-row products.csv file.
Step 3: Understand each part
fetch_html sends a request with a clear User-Agent, a 10-second timeout and raise_for_status(), which stops the script on 4xx or 5xx errors. The one-second sleep spaces out requests.
parse_products uses CSS selectors (article.product_pod, h3 a, p.price_color) to find each card, then a regular expression to turn "£53.74" into the number 53.74. Missing fields become None instead of crashing the run.
save_csv drops incomplete records and writes the rest with the csv module. It reports how many were kept and dropped, which is a first data-quality check.
Step 4: Point it at a live practice page
Replace the sample line with the commented fetch_html call to read the first page of Books to Scrape. Check any site's terms and robots.txt before you point a scraper at it, and never use this pattern to bypass logins, CAPTCHAs or blocks. For tables and numeric work afterwards, see NumPy. AI assistants can draft scrapers too (AI code-generation software), but you must still test the selectors and review the legal side yourself.
When to move up to Scrapy or Playwright
Use Scrapy when you need to crawl many pages with queues, retries and pipelines. Use Playwright when the data only appears after JavaScript runs.
Common Web Scraping Challenges and Limitations
Layout changes: a redesign renames a class and your selector returns nothing. Alerts on empty fields catch this early.
Dynamic content: data loaded after the first response needs rendering or a permitted data endpoint.
Pagination: "next page" links, infinite scroll and cursors each need their own logic, plus a stop condition.
Rate limits and 429 errors: the HTTP 429 Too Many Requests status means you are sending too fast. Slow down, back off, and honor any Retry-After header.
CAPTCHAs: these tests signal that a site wants proof of a human visitor. The compliant response is to stop, use an official API, or ask for permission, not to defeat the check.
Sessions and logins: content behind a login raises contract and authorization questions that public pages do not.
Duplicates, missing fields and stale data: these quietly degrade quality. Track freshness and field-fill rates, and watch for data drift.
Maintenance: scrapers are never "done". Budget for them as you would for any software (software maintenance cost).
Best Free and Open-Source Web Scraping Tools
These five tools cover most do-it-yourself projects. Each differs in purpose, so pick by job rather than popularity.
[[Requests|https://requests.readthedocs.io/en/latest/]]: an HTTP client library. Skill: beginner Python. Best for fetching static pages and API responses. Limit: it does not parse HTML or run JavaScript.
[[Beautiful Soup|https://beautiful-soup-4.readthedocs.io/en/latest/]]: an HTML and XML parser. Skill: beginner. Best for extracting fields from downloaded pages. Limit: it only parses what you give it.
[[Scrapy|https://docs.scrapy.org/]]: a full crawling framework with spiders, pipelines and scheduling. Skill: intermediate. Best for large, repeatable crawls. Limit: a steeper learning curve, and JavaScript needs add-ons.
[[Playwright|https://playwright.dev/python/docs/intro]]: browser automation for Chromium, Firefox and WebKit. Skill: intermediate. Best for dynamic pages. Limit: heavy on memory and CPU.
[[Selenium|https://www.selenium.dev/documentation/]]: a long-standing browser automation project built on WebDriver. Skill: intermediate. Best for teams with existing test automation. Limit: more setup than newer tools.
Free code is not free operation. You still pay for servers, bandwidth, proxies where permitted, storage, monitoring and, above all, developer hours. Teams that run pipelines like these often borrow practices from DevOps.
Best No-Code and Commercial Web Scraping Tools
These products differ in kind: some are visual apps, some are APIs that return pages, and some are infrastructure. Prices below come from vendor pages checked on 2026-11-11 and can change.
[[Octoparse|https://www.octoparse.com/pricing]] (no-code app): visual point-and-click extraction for non-programmers. It lists a free plan with 10 tasks and paid plans from $69 per month on Standard, billed annually. Limit: templates and tasks, not full code control.
[[ParseHub|https://www.parsehub.com/pricing]] (no-code app): a visual scraper with free and paid tiers. We could not verify its current prices for this guide, so check its pricing page.
[[Apify|https://apify.com/pricing]] (platform): runs scrapers called Actors in the cloud. Plans list a free tier with $5 of monthly prepaid usage, Starter at $19 per month plus pay-as-you-go, and $0.20 per compute unit (1 GB of RAM for one hour).
[[Zyte API|https://docs.zyte.com/zyte-api/pricing.html]] (managed extraction API): it charges only for successful responses, with price tiers set by the target site and request type, and a $5 initial free credit.
[[ScrapingBee|https://www.scrapingbee.com/pricing/]] (managed API): credit-based. Its Freelance plan lists 250,000 credits for $49 per month; a request with JavaScript rendering costs 5 credits.
[[Bright Data|https://brightdata.com/pricing/web-scraper]] (infrastructure and data): proxies, unblocker, scraper APIs and ready-made datasets, billed by request, gigabyte or record. Datasets list from $250 per 100,000 records.
[[ScraperAPI|https://www.scraperapi.com/pricing/]] (managed API): credit-based. Hobby lists $49 per month for 100,000 credits, and a standard page costs 1 credit.
Most of these are SaaS products, so the usual build-versus-buy logic applies (see custom software vs. SaaS).
Web Scraping Tools Compared: Features, Pricing, and Best Use Cases
Billing units differ, so do not compare the "Pricing basis" column row by row. A credit, a compute unit and a record are not the same thing. Prices checked 2026-11-11.
Tool | Category | Best for | Pricing basis | Strength | Limitation |
Requests | Library | Fetching static pages | Free, open source | Simple, widely used | No parsing or rendering |
Beautiful Soup | Library | Parsing HTML | Free, open source | Forgiving parser | Needs a fetcher |
Scrapy | Framework | Large repeatable crawls | Free, open source | Built-in queues and pipelines | Learning curve |
Playwright | Browser automation | JavaScript-heavy pages | Free, open source | Modern multi-browser control | High CPU and memory |
Selenium | Browser automation | Teams with test suites | Free, open source | Mature ecosystem | More setup |
Octoparse | No-code app | Non-programmers | Plan: Standard from $69/mo, billed annually | Visual builder | Less code control |
ParseHub | No-code app | Visual extraction | Free and paid tiers; not verified | Point-and-click | Price unverified here |
Apify | Platform | Cloud-run scrapers | Plan + usage: $19/mo Starter, $0.20 per compute unit | Large Actor store | Usage costs vary |
Zyte API | Managed API | Hard-to-fetch sites | Per successful response, tiered | Pays only for success | Tiered, variable cost |
ScrapingBee | Managed API | Developers needing rendering | Credits: $49/mo for 250,000 | JS rendering option | Rendering uses 5 credits |
Bright Data | Infrastructure and data | Scale, proxies, datasets | Per request, GB or record | Broad product range | Complex pricing |
ScraperAPI | Managed API | Simple API fetching | Credits: $49/mo for 100,000 | Easy integration | Credit costs vary by site |
Choose by profile:
Student or hobbyist: Requests + Beautiful Soup.
Developer with many pages: Scrapy, plus Playwright only where rendering is needed.
Analyst with no code: Octoparse or ParseHub.
Small team that wants less maintenance: a managed API such as Zyte, ScrapingBee or ScraperAPI.
Enterprise needing scale or ready datasets: Apify, Bright Data or a licensed dataset.
How Much Does Web Scraping Cost?
The libraries are free, but a working scraping operation never is. The honest question is total cost of ownership, meaning every cost needed to turn pages into trusted data.
DIY code: no license fee, but you pay for build time, servers, storage, monitoring and ongoing fixes.
No-code app: a monthly subscription, often limited by tasks, rows or cloud runtime.
Managed API: metered by credits or successful requests; the vendor absorbs rendering and blocking work, and you still pay for integration and validation.
Managed service or licensed dataset: the vendor delivers finished data, usually priced per record or by contract.
Cost drivers to model: compute and browser rendering, bandwidth and proxies (where permitted), retries and failure rates, storage, developer and analyst time, data validation, security, and legal and compliance review. Cloud bills for the compute portion are covered in cloud cost benchmarks and what Amazon Web Services costs. Published plan prices are facts from vendor pages; labor hours and failure rates are your estimates. Always divide total cost by usable records, not by requests sent.
Worked Web Scraping Cost Example
This is a hypothetical workload, not a benchmark. Assumptions: 50,000 product pages refreshed 4 times a month, so 200,000 page fetches monthly. After validation, 5% of fetches are unusable, leaving 190,000 usable records. Labor is valued at an assumed $50 per hour, and build time is spread over 12 months. Vendor prices are from pages checked on 2026-11-11.
Cost line (per month) | A: DIY Requests script | B: ScrapingBee, JS rendering on | C: ScrapingBee, no rendering |
Platform or server | $20 server + $5 storage = $25 | $99 tier (1,000,000 credits; 200,000 × 5 credits) | $49 tier (250,000 credits; 200,000 × 1 credit) |
Maintenance | 4 hours × $50 = $200 | 2 hours × $50 = $100 | 2 hours × $50 = $100 |
Build, amortized | 20 hours × $50 ÷ 12 = $83.33 | 8 hours × $50 ÷ 12 = $33.33 | 8 hours × $50 ÷ 12 = $33.33 |
Monthly total | $308.33 | $232.33 | $182.33 |
Cost per 1,000 usable records | $308.33 ÷ 190 = $1.62 | $232.33 ÷ 190 = $1.22 | $182.33 ÷ 190 = $0.96 |
What the numbers show: the managed API wins here mainly because it cuts maintenance hours, not because requests are cheap. The DIY server cost is only 8% of option A's monthly total, while labor is 92%. If success falls from 95% to 80%, usable records drop to 160,000 and option A rises to $1.93 per 1,000.
A licensed dataset for comparison: Bright Data lists datasets from $250 per 100,000 records, so 190,000 records would start near $475, or $2.50 per 1,000. That only applies if a dataset covering your sources exists and its license fits your use. Not included: proxies, legal review, security, analyst time and downstream use. Add them before you decide.
Is Web Scraping Legal?
Sometimes. No single rule makes all scraping legal or illegal. The answer depends on what you access, how, what you take, what you do with it, and where you and the data subjects are. This section is general information, not legal advice. Ask a qualified lawyer about your specific project.
The US: CFAA, Van Buren and hiQ
The Computer Fraud and Abuse Act (CFAA) bars accessing a computer "without authorization" or exceeding authorized access. In Van Buren v. United States (June 3, 2021), the Supreme Court said liability turns on a "gates-up-or-down" inquiry: either you can access an area of a system or you cannot. It left open whether limits written in contracts or policies count as gates.
In hiQ Labs v. LinkedIn (Ninth Circuit, April 18, 2022), the court affirmed a preliminary injunction for hiQ and reasoned that the CFAA likely does not reach public profiles that are "presumptively open to all comers." It did not declare scraping lawful. The court noted that other claims, including breach of contract, copyright and trespass to chattels, might still apply. The case then moved on: in November 2022 a federal district court found hiQ breached LinkedIn's user agreement, and the parties later agreed to a consent judgment and permanent injunction.
Access restrictions, terms, copyright and reuse
Authentication and circumvention: logging in, using someone else's credentials or defeating technical blocks creates much greater exposure than reading open pages.
Terms of service: breaching them can lead to contract claims, as the hiQ outcome shows.
Copyright and database rights: facts are generally treated differently from creative expression, but copying text, images or substantial parts of a database can infringe in some places.
Collect vs. republish vs. commercialize: private analysis is lower risk than republishing a site's content or selling it. Each step needs its own check.
Public visibility alone does not resolve legality. Treat the fact that a page loads in your browser as the start of the analysis.
Web Scraping Privacy, GDPR, and Other Legal Risks
If scraped data identifies a living person, privacy law applies, even when the data is public. Names, photos, emails, usernames and profile details are all personal data.
GDPR and UK GDPR basics
Lawful basis: you need one. For scraping, controllers usually rely on legitimate interests under Article 6(1)(f) of the GDPR.
The three-part test: the UK ICO explains that you must show a legitimate purpose, that scraping is necessary for it, and that individuals' interests do not override yours.
Reasonable expectations and transparency: people rarely expect strangers to collect their public posts. You may owe them notice even though you did not collect the data from them directly.
Minimization, sensitive data, security and retention: take only what you need, treat special-category data with extra care, protect it, and delete it on a schedule. Tools like AI data loss prevention and cloud compliance programs help.
What regulators have said
France's CNIL published a focus sheet on legitimate interest and web scraping on January 5, 2026. It says scraping personal data generally rests on legitimate interest, needs extra safeguards, and "is not prohibited per se" but must be analysed case by case. Its context is training AI systems, so apply it to other projects with care.
The European Data Protection Board published Guidelines 03/2026 on web scraping in the context of generative AI on July 8, 2026. As of November 11, 2026, its page still showed the public consultation as ongoing, so treat it as a draft, not final law. The ICO has also issued generative AI guidance on web-scraped data.
In the US, there is no single federal privacy law covering scraping. Rules vary by state and sector, so check the laws where your data subjects live. Related risk topics include website visitor identification and AI risk management.
Ethical Web Scraping Best Practices
Prefer an official API or ask for permission.
Read the terms and the robots.txt file. Note that RFC 9309 states the rules "are not a form of access authorization": robots.txt guides crawlers, and it is not a legal safe harbor either way.
Respect rate limits and use a clear, honest User-Agent with contact details.
Never defeat logins, paywalls, CAPTCHAs or blocks.
Collect the minimum fields, and secure what you store.
Record provenance: source URL, fetch time and terms version.
Validate quality before anyone uses the data.
Set retention limits and delete on schedule.
Review compliance regularly. Policies like AI guardrails and cloud governance offer useful models.
How to Choose the Right Web Scraping Tool
Start with access, then skill, then scale. Score each criterion from 1 to 5 for every option, weight the ones that matter most to you, and compare totals.
Criterion | Question to ask |
Access and authorization | Is there an API, license or permission? Is any login involved? |
Expertise | Can your team write and maintain code? |
Site complexity | Is the content static or JavaScript-rendered? |
Volume and freshness | How many pages, and how often? |
Data quality and SLA | Who guarantees accuracy and uptime? |
Support and integrations | Does it plug into your storage and BI tools? |
Privacy and total cost | Is personal data involved, and what is the cost per usable record? |
Small script (Requests + Beautiful Soup): a few static pages, one developer.
Scrapy or Playwright: recurring, larger or dynamic jobs with engineering capacity. For one-off needs, see internal tool development.
No-code tools: analysts who need results without code.
Managed APIs: teams that would rather pay for fewer blocks and less maintenance.
Licensed datasets: when you need finished data with clear rights. The same buy-versus-build logic appears in AI agents vs. SaaS.
When Should You Avoid Web Scraping?
Scraping is the wrong choice when a better route exists or the risk outweighs the value.
An official API or direct export already covers your fields.
A paid or open dataset exists with a clear license.
A data partnership or permission request is realistic.
The project needs logins, bypassing blocks or collecting sensitive personal data.
Maintenance costs would exceed the value of the data.
The Future of Web Scraping
Three trends are visible now. First, more sites render content with JavaScript, which pushes teams toward browser-based tools and managed APIs. Second, scraping for AI training has drawn regulator attention, as the CNIL, ICO and EDPB documents above show, so expect firmer privacy expectations. Third, demand for dependable pipelines is growing as AI agents and machine learning systems need fresh data. Platforms are also adding restrictions and licensing options, so permission-based data access will likely matter more. Publishers who want to be found by AI search should read AI search content optimization, and builders should note generative AI security risks. Treat all of this as direction, not prediction.
Frequently Asked Questions About Web Scraping
What is web scraping in simple terms?
Web scraping is using software to copy specific information from web pages into a structured file, such as a spreadsheet. Instead of copying and pasting by hand, a program reads the page's HTML and saves the fields you chose, like product names and prices.
What is an example of web scraping?
A common example is tracking prices. A script visits product pages on a practice store such as Books to Scrape, extracts each title and price, and saves them to a CSV file so you can compare changes over time.
How does web scraping work?
A scraper requests a page, receives its HTML, and uses selectors to find the fields you want. It then cleans and validates the values, stores them in a file or database, and repeats on a schedule while monitoring for failures and layout changes.
Is web scraping legal?
Sometimes. There is no single rule. Legality depends on whether the data is public or login-protected, the site's terms, copyright, privacy law and how you use the data. The hiQ v. LinkedIn case shows that winning on one claim does not clear all others. Ask a lawyer about your project.
Is it legal to scrape publicly available data?
Not automatically. In hiQ v. LinkedIn, the Ninth Circuit reasoned that the CFAA likely does not cover public profiles, but contract, copyright and privacy claims can still apply. Personal data stays regulated under GDPR even when it is public.
What is the difference between web scraping and web crawling?
Crawling discovers pages by following links, like a search engine does. Scraping extracts specific fields from pages. Many projects crawl to find pages and then scrape each one, so the two often work together.
Should I use an API instead of web scraping?
Yes, when an official API covers the fields you need. APIs are more stable, come with explicit terms and need less maintenance. Scrape only when no suitable API or licensed dataset exists, and after checking the site's terms.
Can I scrape a website with Python?
Yes. Requests downloads pages and Beautiful Soup parses them, as the tutorial above shows. For large crawls use Scrapy, and for pages that need JavaScript use Playwright or Selenium.
Is web scraping good for beginners?
Yes, if you start small. A few lines of Python with Requests and Beautiful Soup can extract data from a practice site. Beginners should practice on sites built for learning and learn the legal basics before scraping real businesses.
What are the best free web scraping tools?
The main free options are Requests, Beautiful Soup, Scrapy, Playwright and Selenium. Choose by job: Requests and Beautiful Soup for simple pages, Scrapy for large crawls, Playwright or Selenium for dynamic pages. They are free to license, not free to run.
What is the best web scraping tool for non-programmers?
No-code apps such as Octoparse and ParseHub let you click on page elements to build an extractor. Octoparse lists a free plan with 10 tasks. Check each tool's current limits, and confirm the site allows scraping.
How much does web scraping cost per month?
It ranges from near zero for a small script to hundreds of dollars or more. In our hypothetical example, 190,000 usable records cost about $0.96 to $1.62 per 1,000 once labor is included. Your costs depend on volume, rendering, failures and maintenance.
Why do websites block scrapers?
Sites block automated traffic to protect servers, prevent data resale, enforce terms and stop abuse. Fast request rates trigger limits such as HTTP 429. The compliant response is to slow down, use an API or ask permission, not to evade the block.
Does scraping personal data break GDPR?
Not by itself, but GDPR applies to personal data even when it is public. You need a lawful basis, usually legitimate interests, plus transparency, data minimization, security and a retention limit. Regulators such as the CNIL and ICO expect a documented balancing test.
What are alternatives to web scraping?
Alternatives include official APIs, licensed or open datasets, direct data exports, partner feeds and asking the site owner for permission. These routes usually come with clear rights and less maintenance than a custom scraper.
Does robots.txt make scraping legal?
No. RFC 9309 says robots.txt rules are not a form of access authorization. The file tells compliant crawlers what the owner prefers, but it neither grants a legal right to scrape nor, on its own, makes scraping illegal.
Key Takeaways
Web scraping is automated extraction of web page data into structured form. Crawling and APIs are different tools.
Every scraper follows the same pipeline: fetch, parse, extract, clean, validate, store, schedule and monitor.
Requests plus Beautiful Soup suit simple jobs, Scrapy suits large crawls, and Playwright or Selenium suit dynamic pages.
Free tools still cost time, servers and maintenance. Judge any option by cost per usable record.
Paid tools use different units, so convert prices into one comparable measure before choosing.
Legality depends on access method, terms, copyright, privacy law and jurisdiction. hiQ v. LinkedIn is not blanket permission.
Personal data stays regulated when public. EDPB Guidelines 03/2026 are still a consultation draft.
Prefer official APIs or licensed data, and never bypass logins, CAPTCHAs or blocks.
Actionable Next Steps
Write down the exact fields, pages and refresh rate you need.
Check for an official API, export or licensed dataset first.
Read the site's terms and robots.txt, and decide whether personal data is involved.
Pick an approach using the scorecard above: script, framework, no-code tool or managed API.
Run a small pilot with polite request rates and a clear User-Agent.
Validate records for missing fields, duplicates and odd values.
Model total monthly cost and divide by usable records.
Set monitoring, retention and a regular compliance review.
Glossary
Scraper: A program that extracts data from web pages.
Crawler: A program that discovers pages by following links.
HTML: The markup language that structures a web page.
DOM: Document Object Model, the tree of elements a browser builds from HTML.
HTTP: The protocol browsers and scrapers use to request and receive pages.
CSS selector: A pattern that points at elements on a page, such as p.price_color.
XPath: A query language for locating elements in HTML or XML.
API: An interface a site offers for requesting data in a structured way.
JSON: A lightweight text format for structured data.
Parsing: Reading raw HTML and turning it into a searchable structure.
Headless browser: A real browser that runs without a visible window.
robots.txt: A file where a site tells crawlers which paths to avoid or visit.
Pagination: Splitting results across multiple numbered or linked pages.
Rate limit: A cap on how many requests you can send in a time window.
CAPTCHA: A test meant to tell humans and automated programs apart.
Proxy: An intermediary server that forwards requests from a different IP address.
Normalization: Converting values, such as prices and dates, into one consistent format.
Structured data: Data organized in labeled fields, like rows and columns.
Deduplication: Removing repeated records from a dataset.
Compute unit: Apify's billing unit: 1 GB of memory used for one hour.
Sources & References
Vendor prices and plan details were checked on 2026-11-11 and can change. Court and regulator documents are cited as published.
RFC 9309: Robots Exclusion Protocol. IETF (RFC Editor), September 2022.
Requests: HTTP for Humans documentation. Requests project, n.d..
Beautiful Soup Documentation. Leonard Richardson, n.d..
Scrapy documentation. Scrapy project, n.d..
Playwright for Python: Installation. Microsoft, n.d..
The Selenium Browser Automation Project: Documentation. Selenium project, n.d..
Apify pricing. Apify, accessed 2026-11-11.
Octoparse pricing. Octoparse, accessed 2026-11-11.
ParseHub pricing (prices not verified for this guide). ParseHub, accessed 2026-11-11.
Zyte API pricing. Zyte documentation, accessed 2026-11-11.
ScrapingBee pricing. ScrapingBee, accessed 2026-11-11.
Bright Data pricing: Web Scraper. Bright Data, accessed 2026-11-11.
ScraperAPI pricing. ScraperAPI, accessed 2026-11-11.
Van Buren v. United States, No. 19-783. Supreme Court of the United States, June 3, 2021.
hiQ Labs, Inc. v. LinkedIn Corp., No. 17-16783. U.S. Court of Appeals for the Ninth Circuit, April 18, 2022.
Court Finds hiQ Breached LinkedIn's Terms Prohibiting Scraping. Proskauer Rose LLP, November 11, 2022.
Stipulation and Proposed Consent Judgment and Permanent Injunction, hiQ v. LinkedIn. Court filing via DocumentCloud, December 2022.
Regulation (EU) 2016/679 (General Data Protection Regulation). European Union, EUR-Lex, April 27, 2016.
Legitimate interests. UK Information Commissioner's Office, n.d..
Lawful basis for web scraping to train generative AI models. UK Information Commissioner's Office, n.d..
The legal basis of legitimate interest: focus sheet on web scraping. CNIL, January 5, 2026.
Guidelines 03/2026 on web scraping in the context of generative AI. European Data Protection Board, July 8, 2026 (public consultation draft).
429 Too Many Requests. MDN Web Docs, Mozilla, n.d..
Books to Scrape (practice site). Sandbox site for learning, n.d..
