How to Scrape Twitter/X in 2026
A practical guide to getting data out of X/Twitter in 2026. Covers the official API, Tweepy, DIY scraping, scraping APIs like FetchLayer, and MCP for AI agents.
Twitter/X is one of the richest sources of real-time public conversation on the internet. Developers, marketers, researchers, and AI agents all want access to it — but getting data out has become increasingly expensive and restricted.
In 2023, X eliminated its free API tier. Today the X API is pay-per-usage only, with no subscriptions (X’s pricing page, checked 12 November 2026):
- Post reads: $0.005 per post
- User and follower reads: $0.010 per account
- Creating posts: $0.015 per post ($0.200 with a URL)
- Ceiling: 3 million post reads a month, then a custom Enterprise plan
The gap between the old free access and metered pricing that scales linearly with every read and post is where most developers get stuck. For anyone building something real — monitoring, research, content tools, AI pipelines — X’s official API pricing makes it impractical.
So what actually works in 2026? Here are four methods, with real code and honest trade-offs for each.
Method 1: X’s Official API + Tweepy (Expensive, Rate-Limited)
Tweepy is the most popular Python wrapper for X’s official API. It handles OAuth and rate limiting, but you’re still subject to X’s pricing and constraints.
import tweepy
client = tweepy.Client(bearer_token="YOUR_BEARER_TOKEN")
# Search recent tweets
tweets = client.search_recent_tweets(query="python", max_results=10)
for tweet in tweets.data:
print(tweet.text)Pros:
- Well-documented, battle-tested library
- Direct access to X’s API
- Can write tweets, like, retweet
- Official support from X
Cons:
- Metered pay-per-usage pricing — $0.005 per post read, $0.010 per user or follower read, no bundled allowance ($20 of credit after your first purchase)
- A custom Enterprise plan required past 3 million post reads a month
- Complex OAuth setup — 4 keys (API key, API secret, access token, token secret)
- Rate limits — as low as 1 req/15s on some endpoints
- Python-only wrapper — other languages need their own OAuth implementation
- X can reject or suspend your developer account
Tweepy is a solid library for Python developers who already pay for X API access. For everyone else, the pricing makes it a non-starter.
Method 2: DIY Scraping (Fragile, Expensive to Maintain)
You can scrape X’s web interface with a headless browser. This avoids API pricing but introduces a cascade of other problems:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://x.com/search?q=python&src=typed_query&f=live")
# X uses a React frontend with dynamic selectors
page.wait_for_selector('[data-testid="tweet"]')
tweets = page.query_selector_all('[data-testid="tweet"]')
for tweet in tweets[:10]:
text = tweet.query_selector('[data-testid="tweetText"]')
if text:
print(text.inner_text())
browser.close()Pros:
- No API pricing
- Full control over what you extract
- No developer account needed
Cons:
- Fragile — X changes their DOM and anti-bot measures frequently
- Proxy costs — $200-500/month for residential proxies to avoid blocks
- Headless browsers are slow and resource-heavy
- Anti-bot detection — X is one of the most aggressive platforms at detecting automation
- You build and maintain the parser yourself
- Account suspensions — scraping accounts get banned regularly
- No structured data — you’re parsing HTML
This approach makes sense only if you need very specific data that no API provides. For standard tweet/profile/follower data, it’s overkill and expensive to maintain.
Method 3: Twitter/X Scraping API (Recommended)
A scraping API like FetchLayer gives you clean JSON through a simple REST endpoint.
// Search Twitter by keyword
const res = await fetch('https://api.fetchlayer.dev/twitter/search', {
method: 'POST',
headers: {
'Authorization': 'Bearer ss-your-fetchlayer-key',
'Content-Type': 'application/json'
},
body: JSON.stringify({
query: 'best CRM for startups',
product: 'Top',
count: 10
})
});
const data = await res.json();
for (const tweet of data.tweets) {
console.log(`@${tweet.author.handle}: ${tweet.text}`);
console.log(` ${tweet.counts.likes} likes · ${tweet.counts.reposts} retweets`);
}JavaScript and TypeScript developers can also use the official SDK: @fetchlayer/twitter with source on GitHub.
FetchLayer gives you 10 endpoints covering everything:
| Endpoint | What it returns |
|---|---|
search | Tweets matching a keyword (Top, Latest, People, Media, Lists) |
tweet-detail | Full tweet metadata by ID |
tweet-replies | Reply thread for any tweet |
user-profile-details | Profile, bio, follower count, join date |
about-profile | Extended profile metadata (category, business type) |
user-tweets | Recent tweets from a user |
user-replies | Recent replies from a user |
followers | Accounts following a user |
following | Accounts a user follows |
verified-followers | Verified accounts following a user |
Pros:
- Clean JSON response, no parsing needed
- Works from any language (it’s just HTTP)
- No X account or API credentials required
- No proxy infrastructure to manage
- 30 free requests, no credit card
- MCP server for AI agents (Cursor, Windsurf, Claude)
Cons:
- Third-party dependency (like any API)
- Paid for higher volumes
For most developers building something that needs Twitter data — monitoring, research, content tools, AI pipelines — this is the fastest path from zero to working product.
Method 4: MCP Server (For AI Agents & IDEs)
If you work in an AI-powered IDE like Cursor, Windsurf, Claude Desktop, or VS Code with Copilot, you can connect Twitter/X data directly through a Model Context Protocol (MCP) server.
FetchLayer runs an MCP server at mcp.fetchlayer.dev. Add this to your IDE config:
{
"mcpServers": {
"fetchlayer": {
"url": "https://mcp.fetchlayer.dev",
"headers": {
"Authorization": "Bearer ss-your-fetchlayer-key"
}
}
}
}Once connected, your AI agent can search Twitter, pull profiles, and analyze follower graphs inline — without you writing any API code.
Example prompt in Cursor:
Search X/Twitter for “best developer tools 2026” and find the top 5 most engaging tweets. For each tweet author, check their follower count and verification status.
The agent will chain multiple FetchLayer tools: search → tweet detail → user profile — all in one conversation turn.
Why this changes the game for AI agents:
- No API code to write — your agent calls the tools directly
- Multi-step analysis in one prompt — search, enrich, compare, synthesize
- Works with Cursor, Claude Desktop, Claude Code, VS Code, Windsurf, and more
- Same API key you already use for REST calls
Full Method Comparison
| Method | Setup Time | Monthly Cost | Reliability | Best For |
|---|---|---|---|---|
| X API + Tweepy | Hours (OAuth) | $0.005 per post read, $0.010 per user or follower read | High | Write access, or acting for users |
| DIY scraping | Days | $200-500 (proxies) | Low | Unique data needs |
| Scraping API (FetchLayer) | 2 min | 30 free requests, then $1.99 per 1,000 | High | Most developers |
| MCP (FetchLayer) | 2 min | Same as the API | High | AI agent users |
Which Method Should You Pick?
You’re building a product → Use a scraping API like FetchLayer. Clean JSON, no OAuth, free to start.
You use AI coding tools → Set up the MCP server. Your agent handles everything.
You already pay for X API Pro → Use Tweepy directly. You’ve already crossed the pricing hurdle.
You’re a hobbyist testing ideas → Start with FetchLayer’s 30 free requests. No commitment, workable immediately.
You need authenticated write actions → You need X’s official API. Tweepy is the way.
If you want to go deeper on the options, read these next:
- Twitter API Alternatives in 2026 — breakdown of every option with pricing
- Best Twitter Scraper APIs Compared — direct head-to-head comparison
- Twitter API with Python — complete Python integration guide
- FetchLayer API Reference — all 10 endpoints documented