Fetch a web page
Fetch any public web page and get it back as HTML, visible text or Markdown — including the pages an ordinary request comes back empty from, and the pages that turn an ordinary request away. A page is always served the cheapest way that works, and costs more only when an easier attempt has genuinely failed — so an easy page never subsidises a hard one. `pagesFetched` and `usage.billableUnits` report what this particular request actually cost, in units, at three units to the credit: **1 for a simple page, 3 for a harder one, 6 for a hard one, 9 for the hardest**. The whole request runs inside a 120-second budget. Work that cannot finish in what is left of that budget is **skipped and reported as skipped**, never started and cut off halfway — which would bill for work that produced nothing. A 404 returns immediately: the page is not there, and working harder cannot change that. A 503 means the page could not be read at all, and it is not billed. The hardest pages are also the slowest. In a large burst some will exhaust the 120-second budget and come back unread rather than wrong, so expect a slow tail and retry it rather than sizing every timeout for the worst case. Use `output: "markdown"` when the page is going to a model: it keeps the structure, drops the markup, and is far smaller than the HTML. `expectText` and `waitForSelector` are how you tell a real page from a block page — if what you asked for never appears, the request is reported unsuccessful rather than answered with the wrong content. Billing: a simple page costs 1 unit, a harder one 3, a hard one 6, and the hardest 9 — at three units to the credit, so a simple page is a third of a credit. The cost is on every response as `pagesFetched`, and explained in `usage.billing`. A 503 is not billed.
/fetch Pages https://api.fetchlayer.dev/web-unblocker/fetch Headers
Authorization string required Bearer token — your FetchLayer API key.
Parameters
url string required The page to fetch. Must be an http or https address.
render string optional Whether to run the page in a browser: auto, always, never. auto only runs one when a direct read is not enough. (default: auto)
maxEffort string optional Your ceiling on how hard this request may try, cheapest first: basic, standard, advanced, maximum. It sets a ceiling on cost rather than choosing a tier — the cheapest that works is always used regardless. Set it to plain when you would rather a page failed than cost more than the cheapest tier. (default: solve)
country string optional Two-letter country code to read the page from, when the page differs by country.
session string optional An arbitrary label. Requests sharing a label are read over the same connection where possible, so a multi-step flow looks like one visitor.
assets string optional How much of the page to load: minimal, standard, full. minimal is the document only; standard loads what is needed to render, without images or video; full loads everything. Lower settings are faster and cheaper. (default: standard)
waitUntil string optional When to consider the page ready: domcontentloaded, load, networkidle. Only applies when the page runs in a browser. (default: domcontentloaded)
waitForSelector string optional A CSS selector to wait for before reading the page. If it never appears the request is reported unsuccessful rather than returning a partial page.
waitMs integer optional Extra time to let the page settle, in milliseconds, up to 15000.
expectText string optional Text that must appear on the page. If it does not, the request is reported unsuccessful rather than handing back the wrong page — a block page instead of the article you asked for.
device string optional Read the page as a desktop or mobile visitor. (default: desktop)
headers object optional Request headers to send to the page, such as Accept-Language or Referer. Headers that could impersonate another caller are not forwarded, and any that are dropped are listed in notes.
maxRedirects integer optional Redirects to follow before giving up, up to 20. Every hop is re-validated against the same address rules as the first.
format string optional Envelope format: json or markdown. (default: json)
output string optional The form the page comes back in: html, text, markdown. html is the page as rendered; text is visible text only; markdown keeps the structure and drops the markup — much smaller, and the best choice for feeding to a model. (default: html)
maxBytes integer optional Largest page to read, in bytes (1024 to 16777216). A larger page comes back cut short, with truncated true and a note saying so.
Response Fields
fetch
Fetch a web page
transport string The source. Always "web".
notes string[] Caveats you can act on: a page that carried no readable text, a body cut short at maxBytes, request headers that were not forwarded, or an escalation ceiling that was applied.
pagesFetched number What this request cost, in units — the same number as usage.billableUnits, under the field name billing reads. It is a cost, not a page count. A unit is a third of a credit, and the figure reflects how hard the page was to get: a simple page is 1 unit, a harder one 3 (one credit), a hard one 6, and the hardest 9 (three credits). A 503 is not billed.
url string The address that was requested.
finalUrl string The address the request ended at, after redirects.
status number The HTTP status the page answered with.
redirects object[] Redirects that were followed, in order.
title string The page title, when it has one.
contentType string The content type the page declared.
content string The page, in the form asked for by output.
contentFormat string Which form content is in: html, text or markdown.
textLength number Characters of visible text the page carried. 0 with a note is how a page that renders nothing announces itself.
truncated boolean True when the page exceeded maxBytes and was cut short.
usage object What the request cost, how long it took, and how much came back.
usage.bytesTransferred number Bytes transferred across every attempt. A lower bound when a browser was used, because some responses do not declare their length.
usage.totalMs number Wall-clock time for the whole request, every attempt included.
usage.step string The effort level that served the page — basic, standard, advanced or maximum, the same vocabulary as the maxEffort ceiling. This is what the charge is derived from, so a request that cost more than expected says which tier it landed in.
usage.billableUnits number What this request cost, in units, at three units to the credit. The same number as pagesFetched. Charged requests run on a per-account counter rather than a fraction taken off each response, so one response shows the units and the credit falls on every third one — it averages to a third of a credit per unit over any real volume.
Request
curl -X POST "https://api.fetchlayer.dev/web-unblocker/fetch" \
-H "Authorization: Bearer ss-your-key" \
-H "Content-Type: application/json" \
-d '{"url":"https://news.ycombinator.com/","output":"markdown"}'Response
{
"transport": "web",
"notes": [],
"pagesFetched": 1,
"url": "https://news.ycombinator.com/",
"finalUrl": "https://news.ycombinator.com/",
"status": 200,
"redirects": [],
"contentType": "text/html; charset=utf-8",
"title": "Hacker News",
"content": "# Hacker News\n\n1. …",
"contentFormat": "markdown",
"textLength": 3923,
"truncated": false,
"usage": {
"step": "basic",
"bytesTransferred": 34097,
"totalMs": 598,
"billableUnits": 1
}
}