How to Scrape Twitter/X in 2026: 4 Methods That Actually Work

Twitter/X is one of the richest sources of real-time public conversation on the internet. Developers, marketers, researchers, and AI agents all want access to it — but getting data out has become increasingly expensive and restricted.

In 2023, X eliminated their free API tier and introduced paid pricing. What they offer now:

• Free tier: 500 posts/month — functionally useless for data access
• Basic tier ($100/month): 10K posts/month, 1 req/15s on some endpoints
• Pro tier ($5,000/month): 1M posts/month, more endpoints, higher limits
• Enterprise: Custom pricing, fully negotiated

The gap between “free but useless” and “functional but $5K/month” is where most developers get stuck. For anyone building something real — monitoring, research, content tools, AI pipelines — X’s official API pricing makes it impractical.

So what actually works in 2026? Here are four methods, with real code and honest trade-offs for each.

Method 1: X’s Official API + Tweepy (Expensive, Rate-Limited)

Tweepy is the most popular Python wrapper for X’s official API. It handles OAuth and rate limiting, but you’re still subject to X’s pricing and constraints.

import tweepy

client = tweepy.Client(bearer_token="YOUR_BEARER_TOKEN")

# Search recent tweets
tweets = client.search_recent_tweets(query="python", max_results=10)

for tweet in tweets.data:
    print(tweet.text)

Pros:

• Well-documented, battle-tested library
• Direct access to X’s API
• Can write tweets, like, retweet
• Official support from X

Cons:

• $100/month minimum for meaningful use (Basic tier, 10K posts/mo)
• $5,000/month for Pro tier with follower endpoints
• Complex OAuth setup — 4 keys (API key, API secret, access token, token secret)
• Rate limits per tier — as low as 1 req/15s on Basic
• No follower graph on Basic tier — followers/following are Pro only
• Python-only wrapper — other languages need their own OAuth implementation
• Search is recent-only (last 7 days) on lower tiers
• X can reject or suspend your developer account

Tweepy is a solid library for Python developers who already pay for X API access. For everyone else, the pricing makes it a non-starter.

Method 2: DIY Scraping (Fragile, Expensive to Maintain)

You can scrape X’s web interface with a headless browser. This avoids API pricing but introduces a cascade of other problems:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://x.com/search?q=python&src=typed_query&f=live")

    # X uses a React frontend with dynamic selectors
    page.wait_for_selector('[data-testid="tweet"]')
    tweets = page.query_selector_all('[data-testid="tweet"]')

    for tweet in tweets[:10]:
        text = tweet.query_selector('[data-testid="tweetText"]')
        if text:
            print(text.inner_text())

    browser.close()

Pros:

• No API pricing
• Full control over what you extract
• No developer account needed

Cons:

• Fragile — X changes their DOM and anti-bot measures frequently
• Proxy costs — $200-500/month for residential proxies to avoid blocks
• Headless browsers are slow and resource-heavy
• Anti-bot detection — X is one of the most aggressive platforms at detecting automation
• You build and maintain the parser yourself
• Account suspensions — scraping accounts get banned regularly
• No structured data — you’re parsing HTML

This approach makes sense only if you need very specific data that no API provides. For standard tweet/profile/follower data, it’s overkill and expensive to maintain.

Method 3: Twitter/X Scraping API (Recommended)

A scraping API like FetchLayer handles all the infrastructure — proxies, parsing, anti-bot — and gives you clean JSON through a simple REST endpoint.

// Search Twitter by keyword
const res = await fetch('https://api.fetchlayer.dev/twitter/search', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer sk-your-api-key',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    query: 'best CRM for startups',
    product: 'Top',
    count: 10
  })
});

const data = await res.json();
for (const tweet of data.results) {
  console.log(`@${tweet.author.handle}: ${tweet.text}`);
  console.log(`  ${tweet.likeCount} likes · ${tweet.retweetCount} retweets`);
}

JavaScript and TypeScript developers can also use the official SDK: @fetchlayer/twitter with source on GitHub.

FetchLayer gives you 10 endpoints covering everything:

Pros:

• Clean JSON response, no parsing needed
• Works from any language (it’s just HTTP)
• No X account or API credentials required
• No proxy infrastructure to manage
• Free tier included, no credit card
• MCP server for AI agents (Cursor, Windsurf, Claude)

Cons:

• Third-party dependency (like any API)
• Paid for higher volumes

For most developers building something that needs Twitter data — monitoring, research, content tools, AI pipelines — this is the fastest path from zero to working product.

Method 4: MCP Server (For AI Agents & IDEs)

If you work in an AI-powered IDE like Cursor, Windsurf, Claude Desktop, or VS Code with Copilot, you can connect Twitter/X data directly through a Model Context Protocol (MCP) server.

FetchLayer runs an MCP server at mcp.fetchlayer.dev. Add this to your IDE config:

{
  "mcpServers": {
    "fetchlayer": {
      "url": "https://mcp.fetchlayer.dev",
      "headers": {
        "Authorization": "Bearer sk-your-api-key"
      }
    }
  }
}

Once connected, your AI agent can search Twitter, pull profiles, and analyze follower graphs inline — without you writing any API code.

Example prompt in Cursor:

Search X/Twitter for “best developer tools 2026” and find the top 5 most engaging tweets. For each tweet author, check their follower count and verification status.

The agent will chain multiple FetchLayer tools: search → tweet detail → user profile — all in one conversation turn.

Why this changes the game for AI agents:

• No API code to write — your agent calls the tools directly
• Multi-step analysis in one prompt — search, enrich, compare, synthesize
• Works with Cursor, Claude Desktop, Claude Code, VS Code, Windsurf, and more
• Same API key you already use for REST calls

Full Method Comparison

Which Method Should You Pick?

You’re building a product → Use a scraping API like FetchLayer. Clean JSON, no OAuth, free to start.

You use AI coding tools → Set up the MCP server. Your agent handles everything.

You already pay for X API Pro → Use Tweepy directly. You’ve already crossed the pricing hurdle.

You’re a hobbyist testing ideas → Start with FetchLayer’s free tier. No commitment, workable immediately.

You need authenticated write actions → You need X’s official API. Tweepy is the way.

If you want to go deeper on the options, read these next:

• Twitter API Alternatives in 2026 — breakdown of every option with pricing
• Best Twitter Scraper APIs Compared — direct head-to-head comparison
• Twitter API with Python — complete Python integration guide
• FetchLayer API Reference — all 10 endpoints documented