Blog · · 8 min read

Why my Puppeteer scraper kept getting blocked — and the 5 fixes that worked

Amazon India Analytics Dashboard — the system that depends on the scraper
The dashboard the scraper feeds. None of this works if Seller Central locks you out on the second run.

It was the second Monday morning. The cron log showed one line: Error: captcha encountered. The Seller Central account the client had given me access to was now locked for the next six hours. My first thought wasn't "how do I fix the bot detection" — it was "how do I explain to the client that I broke their login."

I'd written what I thought was a careful Puppeteer scraper. Headless Chrome, a normal User-Agent, modest delays, polite navigation. It ran perfectly the first Monday. By the second Monday it was hard-blocked. Amazon's bot detection is more sophisticated than I expected — and I expected a lot.

Here's what was wrong, what I tried, and the five fixes that finally made the scraper run every Monday at 06:00 IST without anyone intervening — reliably, for months now.

First, what Amazon is actually checking

Before the fixes, the diagnosis. A vanilla Puppeteer browser — even running headless with a "normal" User-Agent — leaks at least a dozen signals that say "I am a robot". Amazon checks most of them. The big ones:

Amazon almost certainly checks several more — but these are the ones that matter, in roughly the order they're checked. The fixes below address them in that order.

Fix #1 — puppeteer-extra-plugin-stealth

The big one. puppeteer-extra-plugin-stealth patches roughly seventeen of the above signals automatically. It rewrites navigator.webdriver, fakes a plausible navigator.plugins array, restores chrome.runtime, intercepts the WebGL call, and a dozen smaller things.

The setup:

// npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth

const puppeteer    = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({
  headless: 'new',
  args: [
    '--no-sandbox',
    '--disable-blink-features=AutomationControlled',  // critical, see below
    '--disable-features=IsolateOrigins,site-per-process'
  ]
});

Two things that surprised me about this plugin:

  1. The --disable-blink-features=AutomationControlled Chrome flag is critical and the plugin does not set it for you. Without it, Chrome itself advertises automation through a separate channel that the stealth plugin can't reach. You have to pass it explicitly in args.
  2. It is not magic. The plugin handles the obvious checks. Advanced fingerprinting — canvas fingerprints, audio fingerprints, font enumeration — passes through unmodified. For Seller Central it was enough. For some heavily-protected SaaS UIs (notably Cloudflare's "Pro" tier), stealth alone isn't.

After this fix, my scraper got further into Seller Central than before — but it still tripped a captcha on the more sensitive pages, particularly the Advertising Reports flow where the keyword data lives. That's where the next four fixes came in.

Fix #2 — Persistent browser context (don't log in fresh every time)

Every fresh login is suspicious. Cookies, localStorage, IndexedDB, cached service workers — when a browser arrives at Seller Central with none of these set, it looks like a brand-new device that somehow already has the password. That's an anomaly Amazon flags.

The fix is to reuse the same Chrome profile across every run, using Puppeteer's userDataDir option:

const browser = await puppeteer.launch({
  userDataDir: './chrome-profile-amazon',   // ← the key change
  headless: 'new',
  args: [/* ...same args as before */]
});

The workflow that comes with it:

  1. First login is interactive. Run once on your laptop with headless: false, log in by hand, solve the initial 2FA challenge, accept any cookie banners. Close the browser cleanly. The chrome-profile-amazon/ directory now contains every cookie and storage entry that real session has.
  2. Every cron run after that just reopens that profile. Amazon sees a "returning user" — same device fingerprint, same cookies, same recently-active session — not a "fresh login".
  3. The session does eventually expire and you'll need to re-do the interactive login. In my case it's roughly every couple of months — Amazon doesn't publish the policy, so it's empirical.

One thing to be careful about: that profile directory contains your active session token. Treat it like a secret. Do not commit it to git. Add chrome-profile-amazon/ to .gitignore immediately. Back it up to encrypted storage; restoring it is much faster than re-doing the interactive login.

Fix #3 — A real-looking User-Agent (the obvious one)

The stealth plugin removes "HeadlessChrome" from the User-Agent, but it picks a generic UA that may or may not match your scraper's actual environment. Set it explicitly to a current, real Chrome on the OS your VPS actually runs:

// VPS is Linux, so use a Linux Chrome UA — NOT Windows
await page.setUserAgent(
  'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' +
  '(KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36'
);

Three things that matter here:

Fix #4 — Human-paced timing (randomized delays beat fast loops)

Real users fidget. They hover for half a second before clicking. They take longer to type a long keyword than a short one. They occasionally pause mid-form for two seconds because something distracted them. Puppeteer's defaults do none of this.

const sleep = ms => new Promise(r => setTimeout(r, ms));
const human = (min, max) => sleep(min + Math.random() * (max - min));

// between major actions: 1.2-2.8 seconds, randomized
await page.click('a[href*="campaigns"]');
await human(1200, 2800);

// between keystrokes: 80-140ms, randomized
await page.type('input[name="search"]', keyword, {
  delay: 80 + Math.random() * 60
});
await human(800, 1600);

// occasionally, move the mouse to a "nothing" coordinate
if (Math.random() < 0.3) {
  const box = await page.$eval('main', el => {
    const r = el.getBoundingClientRect();
    return { x: r.left + Math.random() * r.width, y: r.top + Math.random() * r.height };
  });
  await page.mouse.move(box.x, box.y, { steps: 12 });
}

The total run takes longer than it could — around half an hour, instead of the raw few minutes it'd take at full speed. That's fine. The cron at 06:00 IST gives you a four-hour budget before anyone on the client's team wakes up. Use it.

One small piece of paranoia that helps: the page.mouse.move() with steps: 12. That makes the cursor travel along a curved path with intermediate positions, instead of teleporting from A to B. It sounds excessive. It costs nothing. Helps.

Fix #5 — Detect captcha, then stop. Don't retry.

This is the fix that everyone skips, and it's the one that bit me hardest the first time. When the scraper does get served a captcha — and it will, eventually — retrying immediately is the worst possible response. Each retry escalates the block. Three retries in a row turned my six-hour lockout into a twenty-four-hour one.

The right pattern: detect captcha, log it, exit cleanly, alert a human, do not retry until the next scheduled run.

async function isCaptchaPage(page) {
  return await page.evaluate(() => {
    return !!(
      document.querySelector('form[action*="captcha"]') ||
      document.querySelector('#captchacharacters') ||
      document.body.innerText.includes('Type the characters you see')
    );
  });
}

await page.goto(SELLER_CENTRAL_URL, { waitUntil: 'networkidle2' });

if (await isCaptchaPage(page)) {
  await logCaptchaEvent({ at: new Date(), url: page.url() });
  await sendAlert('Amazon scraper hit captcha — pausing');
  await browser.close();
  process.exit(1);   // cron will run next Monday — DO NOT loop
}

Three rules I follow now:

  1. Never auto-retry a captcha. Cron will rerun next Monday. That's the retry.
  2. Log every captcha event with timestamp, URL, and the user-agent in use. Patterns emerge. I noticed mine were clustering on long-weekend Mondays — Amazon presumably running tighter checks when traffic patterns shift.
  3. Alert a human. When captcha fires, it writes a row into the same audit-log table the dashboard reads from, with a distinct event type. The first thing I check on Monday morning is that table — if there's a captcha row, the latest data is from the previous week, not today. If two Mondays in a row hit captcha, that's the signal to do an interactive run with headless: false and see what changed.

The captcha selectors above cover most cases I've seen, but Seller Central's captcha page markup does drift over time — when you implement this, verify the selectors against whatever your actual scraper hits in DevTools and adjust accordingly.

What still occasionally breaks

None of this is bulletproof. Three things still bite, in order of frequency:

  1. Amazon UI changes break selectors. Twice in the last year, Amazon shipped a Seller Central UI change that moved or renamed an element my scraper depended on. Each time, the scraper ran, parsed nothing, and wrote zero rows. The fix was 30 minutes of DevTools and updating two CSS selectors. The lesson: log row counts on every run. A scrape that writes "0 rows" without throwing is a silent bug.
  2. Captcha still appears occasionally. Roughly once every six weeks, often clustering around long weekends or Amazon sales events when traffic patterns shift. With Fix #5 in place this is a non-event — the run aborts cleanly and next Monday's run usually succeeds.
  3. VPS IP reputation. If your VPS provider shares IPs across customers, eventually one of those customers does something Amazon doesn't like, and the whole IP range gets a tighter detection profile. I haven't needed residential proxies yet, but it's the next escalation if reputation becomes an issue.

I deliberately didn't reach for residential proxies. They cost meaningfully more, add a third-party dependency to a system I want to keep simple, and sit in a legal grey zone for commercial scraping. The five fixes above were enough.

The fix order, summarised

  1. puppeteer-extra-plugin-stealth + the --disable-blink-features=AutomationControlled Chrome flag
  2. Persistent userDataDir profile so you look like a returning user, not a fresh login
  3. A current, OS-matched User-Agent — set explicitly, not rotated
  4. Randomised delays between actions (1.2–2.8 s) and between keystrokes (80–140 ms)
  5. Detect captcha, stop, alert. Never auto-retry.

In that order. Doing 5 without 1 doesn't help. Doing 1 without 5 means the first captcha escalates into a multi-day lockout.

The full architecture this scraper feeds — the PostgreSQL schema, the rank-history window-function trick, the Next.js dashboard on top — is in the Amazon Ads analytics writeup, and the broader case study is on the Amazon Analytics work page.

Building something on Amazon Seller Central and stuck on detection? Gauravthakur1k@gmail.com · Services →