The table shows three different situations. The demo shop answers everyone. Wikipedia refuses clients that do not say who they are, and a name with a contact fixes it. npmjs.com refuses every one of them with the same challenge page, so no header is going to help.
1Read the body before you change anything
Most scrapers log the status code and throw away the body. Keep the first few kilobytes. These are the two 403 bodies from the table. They need opposite fixes.
Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also https://phabricator.wikimedia.org/T400119. <title>Just a moment...</title>
…
Enable JavaScript and cookies to continue 2Name the kind of 403
Work down this list. Each entry has a test that confirms it before you spend time on the fix.
-
The site refuses default clients
- What you see
- A short plain-text body that mentions a user agent or a bot policy. It fails from a script and works from curl or a browser.
- Confirm it
- Send the same request with a User-Agent that names your tool and gives a contact. If that returns 200, this was it.
- Fix
- Keep that header. Read the policy the body links to; it usually asks for a request rate as well.
Wikipedia, in the table above: No User-Agent, Python urllib, Python requests got a 403; curl and a named bot got the page.
-
You asked too fast
- What you see
- The first requests work, then 403s start, and they clear after you stop for a while. Some sites send 429 for this; many send 403.
- Confirm it
- Pause for a few minutes and send one request. If it works, the count was the problem, not the request.
- Fix
- Slow down and add backoff that doubles on each refusal. Honour Retry-After when it is sent. Cache what you have already fetched so a re-run does not ask again.
-
Your network is refused
- What you see
- The script works on your laptop and gets a 403 on a server, in CI or in a serverless function, with the same code and headers.
- Confirm it
- Run the identical request from both places. Many sites refuse whole address ranges that belong to hosting providers, whatever the request looks like.
- Fix
- Run the job from a network the site accepts, use the site's official API if it has one, or use a service that returns the page for you.
-
A bot challenge
- What you see
- An HTML page titled "Just a moment...", "Attention Required", "Access denied" or "Verify you are human", often saying "Enable JavaScript and cookies to continue".
- Confirm it
- Change the User-Agent to a desktop browser's. If it is still a 403 with the same page, headers are not what the site checks.
- Fix
- Use an official API or data feed if one exists. Otherwise run a real browser yourself and maintain it as the site changes, or use a service that returns the page.
npmjs.com, in the table above: every client got the challenge page, including the desktop Chrome string (5,661 bytes, status 403).
-
The page is not public
- What you see
- A login form, a paywall, or a message that the content is not available in your country. It fails in a private browser window too.
- Confirm it
- Open the URL in a private window with no cookies. If you cannot see it there, neither can a scraper.
- Fix
- Use an account or API access you are entitled to, or leave it. Getting around a login or a paywall is not a scraping problem to solve.
3A 200 can be a 403 in disguise
Some sites answer a blocked request with status 200 and a "please verify" page. A scraper that checks only the status saves that page as data. Nothing errors, and nobody notices until the data is used.
The fix is to check for content you know must be there: a price, a product name, a heading. If it is missing, treat the response as a failure and retry later, even though it says 200. Checking the page title against the challenge phrases above catches most block pages too.
4Classify a 403 in code
This sorts a response into the kinds above, so a pipeline can retry the rate limits, alert on the challenges and stop on the rest. It needs requests.
import re
import requests
CHALLENGE = ("just a moment", "enable javascript and cookies",
"verify you are human", "access denied", "captcha")
def classify(url, user_agent="my-tool/1.0 (contact: you@example.com)"):
r = requests.get(url, headers={"User-Agent": user_agent}, timeout=30)
head = r.text[:5000].lower()
title = re.search(r"<title[^>]*>(.*?)</title>", head, re.S)
title = title.group(1).strip() if title else ""
if r.status_code == 200:
# A real page can mention "captcha"; a block page is titled like one.
if any(m in title for m in CHALLENGE):
return "block page served as 200"
return "ok"
if r.status_code == 403:
if "user-agent" in head or "robot policy" in head:
return "policy: send a User-Agent that names your tool"
if any(m in head for m in CHALLENGE):
return "challenge: headers will not fix this"
return "forbidden: check login, network and request rate"
if r.status_code == 429:
return "rate limited: wait " + r.headers.get("Retry-After", "and retry")
return f"HTTP {r.status_code}" What it printed on 10 November 2026
- en.wikipedia.org, default requests UA
- policy: send a User-Agent that names your tool
- en.wikipedia.org, named UA
- ok
- npmjs.com, either UA
- challenge: headers will not fix this
- books.toscrape.com, either UA
- ok
5Or hand the hard ones off
For a policy or a rate limit, the fixes above are enough, and you should not pay anyone for them. A challenge is different: you either keep a browser setup working as each site changes, or you send the URL to a service that returns the page. FetchLayer Sesame is ours. Send it the URL and it returns the page as readable Markdown, text, HTML or its links.
curl -X POST https://api.fetchlayer.dev/web-unblocker/fetch \
-H "Authorization: Bearer $FETCHLAYER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://www.npmjs.com/package/zod",
"output": "readable",
"expectText": "TypeScript-first"}' zod - npm
TypeScript-first schema declaration and validation library with static type inference. Latest version: 4.6.7, last published: a month ago.
Recorded on 10 November 2026. This was a hard page, and it cost the same 1 credit as a normal page. Hard pages take longer to come back than normal ones. expectText makes sure the page you get is the package page and not a challenge: if the text never appears, the request is reported as failed instead of handing you the wrong page.
Questions
What does 403 Forbidden mean when scraping?
The server understood the request and refused to answer it. Unlike a 404, the page exists. Unlike a 401, logging in is usually not the fix. The body of the response says which kind of refusal it is, so read it before changing anything.
What is the difference between 403, 401 and 429?
401 means the request needs credentials it did not send. 429 means too many requests, and often carries a Retry-After header. 403 means the server will not answer this request, and sites use it for all sorts of reasons, including rate limits and bot challenges that should have been a 429 or a 401.
Will rotating User-Agent strings fix a 403?
Only when the site checks the User-Agent and nothing else. In our test on 10 November 2026, a descriptive User-Agent fixed Wikipedia's 403, and a desktop Chrome string did not change npmjs.com's 403 at all. Random strings are worse than one honest one: a sudden mix of browsers from one address looks less like a person, not more.
Why does the page load in my browser but not in my script?
A browser runs the page's JavaScript, keeps cookies between requests and sends many headers in a familiar order. A challenge page checks for those things and gives a script nothing until they are there. That is why a challenge 403 survives any header change.
Does a 403 mean my IP address is banned?
Not necessarily. If a private browser window on the same connection loads the page, the address is fine and the request is what is refused. If the browser is refused too, the address or its whole range is, which is common for cloud servers.
Is it OK to get past a 403?
A 403 on a public page is often a site protecting itself from load, which is why pacing requests and identifying your tool fix so many of them. Respect robots.txt and the site's terms, collect public data only, and never work around a login or a paywall. If a site offers an API, use it.
What does Sesame cost for a page like this?
From 1 credit. The npm page in this guide is a hard page: 1 credit. A request that comes back without a page is not billed. There are 30 free requests to try it, no card needed.