403 Forbidden When Web Scraping: Causes and Fixes | FetchLayer

Guide · checked

Why your scraper gets a 403, and what to do about it

A 403 means the server understood the request and refused it. There are five common reasons, and each has a different fix. To show the difference, we sent the same request from six clients to three sites, then once through our own API.

Sent as books.toscrape.comdemo shopA demo shop built for practiceen.wikipedia.orgwikipediaAn encyclopedia articlenpmjs.comnpmA package page
No User-Agent(blank)
Python urllibPython-urllib/3.13
Python requestspython-requests/2.32.3
curlcurl/8.9.1
A named bot with a contactprice-research-bot/1.0 (contact: ops@example.com)
A desktop Chrome stringMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36
FetchLayer Sesameone API call per URL

Request

GET www.npmjs.com/package/zodUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36
Status
403
Body
5,661 B

What came back

<title>Just a moment...</title>
…
Enable JavaScript and cookies to continue
Recorded on 10 November 2026. Each plain row is one GET from one machine on an ordinary connection, and the rows differ only in the User-Agent header. Tap any cell to see the response.

The table shows three different situations. The demo shop answers everyone. Wikipedia refuses clients that do not say who they are, and a name with a contact fixes it. npmjs.com refuses every one of them with the same challenge page, so no header is going to help.

1Read the body before you change anything

Most scrapers log the status code and throw away the body. Keep the first few kilobytes. These are the two 403 bodies from the table. They need opposite fixes.

GET en.wikipedia.org/wiki/Markdown403 · 126 B · text/plain

Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also https://phabricator.wikimedia.org/T400119.
A policy. It tells you what it wants: a User-Agent, and to follow the linked robot policy.

GET npmjs.com/package/zod403 · ~5 KB · text/html

<title>Just a moment...</title>
…
Enable JavaScript and cookies to continue
A challenge. It expects a browser that runs its script and keeps a cookie, so a different header will not get you past it.

2Name the kind of 403

Work down this list. Each entry has a test that confirms it before you spend time on the fix.

  1. The site refuses default clients

    What you see
    A short plain-text body that mentions a user agent or a bot policy. It fails from a script and works from curl or a browser.
    Confirm it
    Send the same request with a User-Agent that names your tool and gives a contact. If that returns 200, this was it.
    Fix
    Keep that header. Read the policy the body links to; it usually asks for a request rate as well.

    Wikipedia, in the table above: No User-Agent, Python urllib, Python requests got a 403; curl and a named bot got the page.

  2. You asked too fast

    What you see
    The first requests work, then 403s start, and they clear after you stop for a while. Some sites send 429 for this; many send 403.
    Confirm it
    Pause for a few minutes and send one request. If it works, the count was the problem, not the request.
    Fix
    Slow down and add backoff that doubles on each refusal. Honour Retry-After when it is sent. Cache what you have already fetched so a re-run does not ask again.
  3. Your network is refused

    What you see
    The script works on your laptop and gets a 403 on a server, in CI or in a serverless function, with the same code and headers.
    Confirm it
    Run the identical request from both places. Many sites refuse whole address ranges that belong to hosting providers, whatever the request looks like.
    Fix
    Run the job from a network the site accepts, use the site's official API if it has one, or use a service that returns the page for you.
  4. A bot challenge

    What you see
    An HTML page titled "Just a moment...", "Attention Required", "Access denied" or "Verify you are human", often saying "Enable JavaScript and cookies to continue".
    Confirm it
    Change the User-Agent to a desktop browser's. If it is still a 403 with the same page, headers are not what the site checks.
    Fix
    Use an official API or data feed if one exists. Otherwise run a real browser yourself and maintain it as the site changes, or use a service that returns the page.

    npmjs.com, in the table above: every client got the challenge page, including the desktop Chrome string (5,661 bytes, status 403).

  5. The page is not public

    What you see
    A login form, a paywall, or a message that the content is not available in your country. It fails in a private browser window too.
    Confirm it
    Open the URL in a private window with no cookies. If you cannot see it there, neither can a scraper.
    Fix
    Use an account or API access you are entitled to, or leave it. Getting around a login or a paywall is not a scraping problem to solve.

3A 200 can be a 403 in disguise

Some sites answer a blocked request with status 200 and a "please verify" page. A scraper that checks only the status saves that page as data. Nothing errors, and nobody notices until the data is used.

The fix is to check for content you know must be there: a price, a product name, a heading. If it is missing, treat the response as a failure and retry later, even though it says 200. Checking the page title against the challenge phrases above catches most block pages too.

4Classify a 403 in code

This sorts a response into the kinds above, so a pipeline can retry the rate limits, alert on the challenges and stop on the rest. It needs requests.

import re
import requests

CHALLENGE = ("just a moment", "enable javascript and cookies",
             "verify you are human", "access denied", "captcha")

def classify(url, user_agent="my-tool/1.0 (contact: you@example.com)"):
    r = requests.get(url, headers={"User-Agent": user_agent}, timeout=30)
    head = r.text[:5000].lower()
    title = re.search(r"<title[^>]*>(.*?)</title>", head, re.S)
    title = title.group(1).strip() if title else ""
    if r.status_code == 200:
        # A real page can mention "captcha"; a block page is titled like one.
        if any(m in title for m in CHALLENGE):
            return "block page served as 200"
        return "ok"
    if r.status_code == 403:
        if "user-agent" in head or "robot policy" in head:
            return "policy: send a User-Agent that names your tool"
        if any(m in head for m in CHALLENGE):
            return "challenge: headers will not fix this"
        return "forbidden: check login, network and request rate"
    if r.status_code == 429:
        return "rate limited: wait " + r.headers.get("Retry-After", "and retry")
    return f"HTTP {r.status_code}"

What it printed on 10 November 2026

en.wikipedia.org, default requests UA
policy: send a User-Agent that names your tool
en.wikipedia.org, named UA
ok
npmjs.com, either UA
challenge: headers will not fix this
books.toscrape.com, either UA
ok

5Or hand the hard ones off

For a policy or a rate limit, the fixes above are enough, and you should not pay anyone for them. A challenge is different: you either keep a browser setup working as each site changes, or you send the URL to a service that returns the page. FetchLayer Sesame is ours. Send it the URL and it returns the page as readable Markdown, text, HTML or its links.

curl -X POST https://api.fetchlayer.dev/web-unblocker/fetch \
  -H "Authorization: Bearer $FETCHLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.npmjs.com/package/zod",
       "output": "readable",
       "expectText": "TypeScript-first"}'

Same npm page, through Sesame200

zod - npm

TypeScript-first schema declaration and validation library with static type inference. Latest version: 4.6.7, last published: a month ago.

1 credit1,331 tokens

Recorded on 10 November 2026. This was a hard page, and it cost the same 1 credit as a normal page. Hard pages take longer to come back than normal ones. expectText makes sure the page you get is the package page and not a challenge: if the text never appears, the request is reported as failed instead of handing you the wrong page.

See Sesame Read the docs 30 free requests, no card.

Questions

What does 403 Forbidden mean when scraping?

The server understood the request and refused to answer it. Unlike a 404, the page exists. Unlike a 401, logging in is usually not the fix. The body of the response says which kind of refusal it is, so read it before changing anything.

What is the difference between 403, 401 and 429?

401 means the request needs credentials it did not send. 429 means too many requests, and often carries a Retry-After header. 403 means the server will not answer this request, and sites use it for all sorts of reasons, including rate limits and bot challenges that should have been a 429 or a 401.

Will rotating User-Agent strings fix a 403?

Only when the site checks the User-Agent and nothing else. In our test on 10 November 2026, a descriptive User-Agent fixed Wikipedia's 403, and a desktop Chrome string did not change npmjs.com's 403 at all. Random strings are worse than one honest one: a sudden mix of browsers from one address looks less like a person, not more.

Why does the page load in my browser but not in my script?

A browser runs the page's JavaScript, keeps cookies between requests and sends many headers in a familiar order. A challenge page checks for those things and gives a script nothing until they are there. That is why a challenge 403 survives any header change.

Does a 403 mean my IP address is banned?

Not necessarily. If a private browser window on the same connection loads the page, the address is fine and the request is what is refused. If the browser is refused too, the address or its whole range is, which is common for cloud servers.

Is it OK to get past a 403?

A 403 on a public page is often a site protecting itself from load, which is why pacing requests and identifying your tool fix so many of them. Respect robots.txt and the site's terms, collect public data only, and never work around a login or a paywall. If a site offers an API, use it.

What does Sesame cost for a page like this?

From 1 credit. The npm page in this guide is a hard page: 1 credit. A request that comes back without a page is not billed. There are 30 free requests to try it, no card needed.