Automating X (Twitter) in 2026: shadow DOM, 429 lockouts, and the Premium gate

I spent a few weeks building a lead-search tool for X. Not a scraper library - a thing that runs in a real browser, signed in as a real account, and tries to behave like a person while it looks for accounts worth contacting.

Almost everything I assumed was wrong. This is the list of things that broke, and what actually worked.

1. Every profile visit is a full app boot

X's people search gives you handles and almost nothing else. Follower counts, bios, join dates - all of it lives on the profile page, and every profile page load is:

• a GraphQL Viewer request,
• the full React app booting,
• a pile of static assets.

Do that 30-40 times in a row and X answers HTTP 429. After that, x.com stops booting at all - in every browser on that IP, logged in or not - for somewhere between 30 and 60 minutes. Not a soft limit on one endpoint. The whole site.

The only reliable early signal I found was the #ScriptLoadFailure element, plus the 1003 code in the Viewer response body. Once you see it, you're done for the hour.

The important part is what you do next: nothing. Retrying is the failure mode. A naive for attempt in range(3) turns a 20-minute lockout into an hour, because each retry is another full app boot against a counter that's already tripped.

if await page.locator("#ScriptLoadFailure").count():
    raise RateLimited("X is refusing to serve the app; back off, do not retry")

Failure fast beats retrying with backoff, because the cost isn't the failed request - it's the next 40 requests that now can't happen.

2. DMs moved into a separate app that renders in a shadow root

The old DM page is gone. /messages now hands you over to /i/chat, an embedded React app served from abs.twimg.com/x-web/xchat/embed/. It renders into shadow roots.

That breaks the two things every scraper does first:

document.querySelector('[data-testid="dm-composer"]') // null
document.body.innerText // "" for the chat panel

innerText on body genuinely returns nothing useful, because the chat UI is inside a shadow tree, and shadow content isn't part of the host's text. You can walk it by hand:

const texts = [];
const walk = (root, depth = 0) => {
  if (depth > 10) return;
  texts.push(root.textContent || "");
  for (const el of root.querySelectorAll("*")) {
    if (el.shadowRoot) walk(el.shadowRoot, depth + 1);
  }
};
walk(document);
const bodyText = texts.join("\n").replace(/\s+/g, " ");

But here's the thing that took me too long to check: Playwright's CSS engine pierces open shadow roots already. page.locator('[data-testid="..."]') finds elements inside shadow DOM without any traversal. My problem was never reachability - it was that I was using selectors that no longer existed.

So don't guess the names. Read them out of the bundle:

• Load /i/chat, capture the script responses from abs.twimg.com/x-web/xchat/embed/assets/.
• Grep the chunks for dm- strings.
• You get the real vocabulary: dm-composer-textarea, dm-composer-send-button, dm-message-scroller, dm-conversation-panel.

Mine shipped filenames in backticks, so a quote-only regex missed most of them - a plain grep -o 'dm-[a-z-]*' found ~40 testids, the quote-matched one found five.

One more trap: the composer is a controlled React textarea. locator.fill() sets the DOM value but doesn't fire the synthetic events React listens to, so the send button stays disabled and the message never sends. Real keystrokes work:

await composer.click()
await page.keyboard.type(message, delay=45)

3. The composer isn't always rendered - and there's no error

If the account isn't verified and the recipient doesn't follow it, X doesn't render the composer at all. What you get is a dialog: Get verified to message @user - Only verified users can send Direct Message requests to people that don't follow them.

Dismissing that dialog does not reveal a composer. There is no textarea, no contenteditable, no role="textbox" - nothing to type into, anywhere in the document or its shadow roots.

If your automation assumes the composer exists, it doesn't fail loudly: it fails per target, quietly, thirty times in a row, while burning the rate-limit budget from section 1. That's how I ended up locked out on the first real run.

Detect the condition once and stop the whole run:

if "get verified to message" in body_text.lower():
    raise AccountGated("X requires verification for cold DMs")

This isn't a bug you can code around. It's a product decision on X's side, and the only honest responses are "pay for Premium" or "find another channel".

4. The chat app asks for a PIN the first time

Open a conversation in a fresh browser profile and /i/chat asks for an X Chat passcode - a 4-digit PIN, presented as four inputs. They're shadow-DOM children, and Playwright's fill() on them is flaky because of how the component handles focus.

What worked: focus the first input from inside the page, then send real keystrokes.

const boxes = [];
const walk = (root) => {
  root.querySelectorAll('input[aria-label^="Digit "]').forEach(i => boxes.push(i));
  root.querySelectorAll("*").forEach(el => el.shadowRoot && walk(el.shadowRoot));
};
walk(document);
boxes[0]?.focus();

Then page.keyboard.type(pin, delay=180). The delay matters; the component advances focus per keystroke, and typing too fast drops digits.

5. Profile data is cleaner in the JSON-LD than in the DOM

Scraping the profile DOM gives you localized follower counts (12,4 тис.) and translated bios. There's a <script type="application/ld+json"> block instead, with a UserProfileSchema object containing:

• exact userInteractionCount for followers,
• the raw, untranslated bio,
• the numeric user id.

That last one is the useful part. With a numeric id you can deep-link straight to a conversation instead of hunting for a localized "Message" button.

6. Search results are mostly noise

The unexpected finding: X's people search returns a lot of accounts that have nothing to do with the query. Searching for a niche term gives you local trends, unrelated verified accounts, and people who mentioned the word once in 2021.

A cheap local relevance gate - the query term (or a synonym) must appear in the bio or display name - cut 85 raw handles down to 7 on-topic accounts. That ratio is the actual value of the tool; the scraping is the easy part.

What I'd tell someone starting this today

• Back off before you're told to. The 429 isn't per-request, it's per-session, and the recovery cost is measured in tens of minutes.
• Read the JS bundle before writing selectors. Test IDs are in there; guessing costs days.
• Assume the UI you need isn't rendered. Gate checks are cheaper than retries.
• Treat shadow DOM as a normal DOM. Playwright handles it; your old selectors don't.
• Filter locally. The search API won't do it for you, and relevance is the product.

If you've automated a shadow-DOM app like this, I'd like to know how you test it - replaying against a live account for every selector change doesn't scale, and I haven't found a satisfying answer.