Your Browser Automation Clicks Might Be Landing 25% Off - And Nothing Will Tell You

I spent a day pointing an AI agent at a browser to publish one product across four marketplaces. Most of it worked. The parts that didn't work failed in the worst possible way: silently, with no error, no exception, and no log line. Here are the five failure modes I hit, in the order I hit them, and the probe that turned the worst one from "this site is broken" into a two-line fix. Everything below is from one working session. No hypotheticals. The symptom: clicks that do nothing I asked the agent to click a menu item. The tool reported success: [computer:left_click] Clicked at (383, 734) Nothing happened. No menu opened, no network request fired, no console error. I re-read the DOM - the element was there, visible, not disabled, not covered by an overlay: const el = document.elementFromPoint(383, 734); // => 削除 ← the exact element I wanted So the coordinate was right, the element was right, and the click "succeeded". And nothing happened. I burned close to an hour assuming the site was blocking synthetic input. It wasn't. The probe: make the page answer for itself Instead of reasoning about it, I made the page report whether a click ever arrived. Inject a button at a known position with a counter: const b = document.createElement('button'); b.id = '__clicktest'; b.textContent = 'CLICK TEST'; b.style.cssText = 'position:fixed;top:200px;left:200px;width:220px;height:60px;' + 'z-index:2147483647;background:#f0f;'; b.addEventListener('click', (e) => { window.__clickOK = (window.__clickOK || 0) + 1; window.__trusted = e.isTrusted; }); document.body.appendChild(b); window.__clickOK = 0; const r = b.getBoundingClientRect(); center=${r.x + r.width / 2},${r.y + r.height / 2}; // => center=310,230 Then click that exact center and read the counter: window.__clickOK; // => 0 Zero. The click was not landing on a button that occupied 220×60 pixels at a coordinate I had just measured. That rules out "the site blocks synthetic events" - a blocked event still arrives; it just gets ignored by the handler. Mine never arrived. So the coordinate space was wrong. The math: two coordinate spaces, one number The automation tool takes coordinates in screenshot space. The page reports coordinates in CSS pixel space. If the screenshot is captured at a different scale than the viewport, every coordinate you read from the DOM is wrong by a constant factor - and the failure is invisible, because a click still happens, just somewhere else. I measured the factor by bisecting. Device-pixel space (×2 , since devicePixelRatio was 2) missed. Then: window.__clickOK; // after clicking (232, 172) instead of (310, 230) // => 1 That gives the ratio directly: k = 232 / 310 = 0.7484 Viewport was 1702 CSS px wide; the screenshots came back ~1274 px. 1274 / 1702 ≈ 0.7485 . It matched. From then on, every click went through one conversion: const K = 0.7484; // measure this yourself; do not copy mine function clickPoint(el) { const r = el.getBoundingClientRect(); return { x: Math.round((r.x + r.width / 2) * K), y: Math.round((r.y + r.height / 2) * K), }; } Every interaction that had failed for the previous hour started working on the first try - dropdowns, confirmation dialogs, menu items. The lesson isn't the number. k depends on your window size, DPI, and tool. The lesson is that you should never trust a coordinate you didn't verify with a probe, because this class of bug produces no error at any layer. Run the probe once at the start of a session. It costs three seconds. Failure mode 2: the menu that was below the fold With clicks fixed, one menu item still did nothing. Same probe logic applied - elementFromPoint returned null for it. const r = item.getBoundingClientRect(); r.y; // => 981 window.innerHeight; // => 876 The dropdown extended past the bottom of the viewport. getBoundingClientRect() happily returns coordinates for content that isn't on screen, and a click at y=981 in a 876px viewport goes nowhere. Scroll first, then re-measure. Never cache coordinates across a scroll: el.scrollIntoView({ block: 'center' }); await new Promise(r => setTimeout(r, 300)); const r2 = el.getBoundingClientRect(); // re-read, always Failure mode 3: file inputs that don't exist yet One site had an upload drop zone with no file input in the DOM: document.querySelectorAll('input[type=file]').length; // => 0 The input is created on demand when you click the zone. So: click the zone, then look. To catch the element before the app calls .click() on it (which opens a native dialog you can't drive), patch createElement first: const orig = document.createElement.bind(document); window.__created = []; document.createElement = function (tag, ...rest) { const el = orig(tag, ...rest); if (String(tag).toLowerCase() === 'input') { setTimeout(() => { if (el.type === 'file') window.__created.push(el); }, 0); } return el; }; Now click the zone and the input is waiting for you in window.__created . Populate it with a DataTransfer and dispatch change : const dt = new DataTransfer(); dt.items.add(file); input.files = dt.files; input.dispatchEvent(new Event('change', { bubbles: true })); This worked on two of the sites I tried. On the third it didn't - see below. Failure mode 4: synthetic drops that React ignores Some drop zones aren't inputs at all. The obvious move is to synthesize the drop: const dt = new DataTransfer(); dt.items.add(file); for (const type of ['dragenter', 'dragover', 'drop']) { const ev = new DragEvent(type, { bubbles: true, cancelable: true }); Object.defineProperty(ev, 'dataTransfer', { value: dt }); zone.dispatchEvent(ev); } I tried this against every plausible target - the drop zone, its ancestors, document , window . Nothing. The editor never registered a file. To be sure the file wasn't the problem, I generated one in-page so there was no transfer step at all: const c = document.createElement('canvas'); c.width = 1200; c.height = 800; c.getContext('2d').fillRect(0, 0, 1200, 800); const blob = await new Promise(r => c.toBlob(r, 'image/jpeg', 0.8)); const file = new File([blob], 'test.jpg', { type: 'image/jpeg' }); Same result. That's a real boundary: for that editor, drag-and-drop needs genuine OS-level input, and no amount of event synthesis substitutes for it. I stopped and did those uploads by hand. Knowing where the wall is is worth more than another hour of clever attempts. Failure mode 5: localhost is still mixed content To avoid shipping image bytes through a JS payload, I served them locally with CORS enabled and fetched them from the page: class H(http.server.SimpleHTTPRequestHandler): def end_headers(self): self.send_header('Access-Control-Allow-Origin', '*') super().end_headers() curl confirmed the header. From an https:// page, the fetch hung until it timed out: await fetch('http://127.0.0.1:8941/img1.png'); // never resolves An https page fetching http://127.0.0.1 is mixed content. The CORS header is irrelevant - the request never gets far enough to matter. Terminate that path early instead of debugging your server. The preflight I now run first Five minutes of setup that would have saved me most of a day: // 1. Coordinate probe - is my click space the page's space? // (inject the test button above, click its center, read window.__clickOK) // 2. Is the target actually on screen? const onScreen = (el) => { const r = el.getBoundingClientRect(); return r.top >= 0 && r.bottom 0; }; // 3. Does this page even have the input I assume it has? document.querySelectorAll('input[type=file]').length; // 4. Verify before every destructive click. // Hover first, then confirm what is under the cursor: document.elementFromPoint(x / K, y / K)?.textContent?.trim(); Step 4 matters more than it looks. In one menu, "Delete" sat directly below "Unpublish" - 24 pixels apart. With a 25% coordinate error, "unpublish" lands on "delete". Hovering and reading back the element text before committing turns a destructive misfire into a no-op. What I'd tell my past self The productive shift wasn't a better selector strategy. It was refusing to reason about why a click "didn't work" and instead making the page report what it actually received. Every one of these five failures is invisible from the outside: success-shaped tool output, no exception, no console error. The only reliable signal came from instrumenting the page and reading a counter. If your automation is mysteriously doing nothing, don't start with the selector. Start with the probe. All measurements here come from a single session driving Chrome against live marketplace sites. The scaling factor k is specific to that window and tool - measure your own. Top comments (0)