中
← Articles

August 10, 2026

A Rescue Crawl: Archiving xitek.com Before It Shuts Down

A rescue-archiving engineering log for a photography community about to shut down. Covers the anti-scraping battle, architecture, and implementation in depth.

Background: What “Rescue Crawling” Really Means

xitek.com is a veteran Chinese photography community with 20 years of deep gear reviews — camera head-to-heads, lens comparison tests, measured parameters — much of it first-hand content you can’t find anywhere else. When the site announced shutdown, all of it was about to disappear.

“Rescue crawling” differs from normal scraping in emphasis: not speed, but certainty — reliably capturing everything within a finite window without failures. It’s an engineering problem spanning anti-scraping, rate-limiting, resumable pipelines, and long-running reliability.

Tech Choice: Why Playwright, Not httrack

The instinct was “just mirror the site with httrack/wget”. Testing immediately killed that:

  • curl to homepage → blocked by WAF (403 interception page)
  • httrack / wget → same, instantly blocked
  • Even OpenClaw’s browser proxy (CDP channel) timed out on xitek

xitek deploys a WAF firewall + JS fingerprint check. Plain HTTP clients lack real browser rendering and TLS fingerprints, so they’re flagged as bots immediately.

Only a real Chromium (Playwright) with full JS execution can access it. The stack:

{
  "dependencies": {
    "crawlee": "^3.0.0",
    "playwright": "^1.40.0"
  }
}

Note: I originally planned better-sqlite3 for the queue, but it fails to compile natively on Node v26. I switched to Node’s built-in node:sqlite (DatabaseSync) — zero native deps, no compile headaches.

The Hard Part: Reverse-Engineering the WAF Rate Limit

This is the crux. I mapped the WAF’s behavior through a series of controlled experiments.

Experiment 1: Can new browser fingerprints bypass it?

// Fresh browser context + random UA + canvas tweak to fake a "new fingerprint"
for (let i = 0; i < 3; i++) {
  const ctx = await browser.newContext({
    userAgent: UAS[i % UAS.length],
    colorScheme: i % 2 ? 'dark' : 'light',
  });
  // ... fetch one article
}

Result: of 3 fresh fingerprints, the first 2 were still WAF-blocked; only the 3rd succeeded.

Experiment 2: Can a proxy IP bypass it?

Routed through Shadowrocket’s HTTP proxy (port 1082):

chromium.launch({ proxy: { server: 'http://127.0.0.1:1082' } });

Result: blocked on the very first request (proxy IP pools often have worse reputation).

Conclusion

The WAF does IP-based frequency rate-limiting: roughly 3-4 requests per window triggers interception, and cooldown is ~4-5 minutes to recover. Neither new fingerprints nor proxy IPs truly bypass it — the only reliable defense is strict rate-limiting + enforced cooldown.

So the cadence became: batch of 3 pages → 7-minute cooldown → next batch, ~30 pages/hour.

Architecture: A Resumable Task Queue

A SQLite-backed task state machine — the lifeline for rescue scenarios. Regain progress anytime, never re-crawl, never miss.

// queue.js — crawl queue on node:sqlite
import { DatabaseSync } from 'node:sqlite';

export const STATUS = {
  PENDING: 'pending', RUNNING: 'running', DONE: 'done',
  FAILED: 'failed',   SKIPPED: 'skipped',
};

export class CrawlQueue {
  constructor(dbPath) {
    this.db = new DatabaseSync(dbPath);
    this.db.exec(`
      CREATE TABLE IF NOT EXISTS queue (
        url TEXT PRIMARY KEY,
        type TEXT NOT NULL,          -- list | article | image
        status TEXT NOT NULL,
        priority INTEGER DEFAULT 100,
        attempts INTEGER DEFAULT 0,
        next_retry_at INTEGER DEFAULT 0,
        error TEXT,
        created_at INTEGER, updated_at INTEGER
      );
    `);
  }

  next() {
    const row = this.db.prepare(`
      SELECT * FROM queue
      WHERE status = 'pending' AND next_retry_at <= ?
      ORDER BY priority ASC, created_at ASC LIMIT 1
    `).get(Date.now());
    if (!row) return null;
    this.db.prepare(`UPDATE queue SET status='running' WHERE url=?`).run(row.url);
    return { ...row, extra: row.extra ? JSON.parse(row.extra) : null };
  }

  fail(url, error, { permanent = false } = {}) {
    // exponential backoff: 30s → 2m → 10m → 1h
    const backoff = [30_000, 120_000, 600_000, 3_600_000][attempts - 1] || 3_600_000;
    // ... update status & next_retry_at
  }

  done(url) { this.db.prepare(`UPDATE queue SET status='done' WHERE url=?`).run(url); }
}

Three task types, prioritized via priority:

  • list (listing pages, priority 0-10): parse pagination, enqueue all detail URLs
  • article (detail pages, priority 20): fetch body, persist to disk
  • image (images, priority 40): download in-article images

Cracking the Pagination Structure

After fetching the homepage with a real browser, I extracted the pagination pattern from the raw DOM:

Total reviews ≈ 1,642
Listing:  https://info.xitek.com/reviews/1642-{n}.html   (n = 2..152)
Detail:   https://info.xitek.com/reviews/YYYYMM/DD-ID.html
Images:   https://cmscdn.xitek.com/uploads/allimg/YYYYMMDD/xxx.jpg
Body:     .sam_body
Title:    h1
Date:     /\d{4}-\d{2}-\d{2}/

Body extraction uses regex rather than a DOM library to avoid extra dependencies:

// article.js — extract structured article from detail HTML
export function parseArticle(html, url) {
  const extract = (re, s = html) => {
    const m = s.match(re);
    return m ? m[1].trim() : null;
  };

  const title = stripTags(extract(/<h1[^>]*>([\s\S]*?)<\/h1>/i) || '').trim();
  const date  = extract(/(\d{4}-\d{2}-\d{2})/);
  const author = extract(/来源[：:]\s*([^<]{1,20})/i) || extract(/作者[：:]\s*([^<]{1,20})/i);

  // body HTML inside .sam_body
  let bodyHtml = extract(/<div[^>]*class="[^"]*sam_body[^"]*"[^>]*>([\s\S]*?)<\/div>/i);
  // collect cmscdn images
  const images = [];
  const imgRe = /src="([^"]+\.jpg)"/gi; let m;
  while ((m = imgRe.exec(html))) {
    const src = m[1];
    if (src.startsWith('data:')) continue;
    if (!src.includes('cmscdn')) continue;
    images.push(src);
    if (images.length >= 60) break;
  }
  return { title, author, date, bodyHtml, images, url, id };
}

Discovering Details from Listing Pages

export function parseListUrls(html) {
  const urls = new Set();
  const re = /href="(\/\/(?:info\.xitek\.com)?\/reviews\/\d{6}\/\d+-\d+\.html)"/g;
  let m;
  while ((m = re.exec(html))) {
    const abs = m[1].startsWith('//') ? 'https:' + m[1] : 'https://info.xitek.com' + m[1];
    urls.add(abs);
  }
  return [...urls];
}

Each listing page yields 15-16 detail URLs which are deduped and enqueued. Once all 152 listing pages are done, the detail queue naturally reaches ~2,000+ (about 1,642 unique after dedup).

Anti-Scraping Browser Wrapper

Key pieces: hiding automation signals + rate limiting + WAF detection:

// browser.js — anti-scraping fingerprint + rate-limit + WAF detection
import { chromium } from 'playwright';

export class BrowserClient {
  constructor({ delayMs = 9000, jitterMs = 5000 } = {}) {
    this.delayMs = delayMs; this.jitterMs = jitterMs;
  }

  async start() {
    this.browser = await chromium.launch({
      headless: true,
      args: ['--disable-blink-features=AutomationControlled', '--no-sandbox'],
    });
    const ctx = await this.browser.newContext({
      locale: 'zh-CN',
      timezoneId: 'Asia/Shanghai',
      userAgent: 'Mozilla/5.0 (...) Chrome/126.0.0.0 Safari/537.36',
    });
    // hide webdriver, masquerade as real Chrome
    await ctx.addInitScript(() => {
      Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
    });
    this.page = await ctx.newPage();
  }

  // rate limit: ensure at least delayMs + jitter between requests
  async throttle() {
    const elapsed = Date.now() - this.lastRequest;
    const wait = this.delayMs + Math.floor(Math.random() * this.jitterMs);
    if (elapsed < wait) await sleep(wait - elapsed);
    this.lastRequest = Date.now();
  }

  // goto with an absolute timeout to avoid hanging on WAF challenge JS
  async gotoWithTimeout(url, timeout = 65000) {
    return await Promise.race([
      this.page.goto(url, { waitUntil: 'domcontentloaded', timeout }),
      new Promise((_, rej) => setTimeout(() => rej(new Error('goto timeout')), timeout)),
    ]);
  }

  async fetch(url) {
    await this.throttle();
    const resp = await this.gotoWithTimeout(url);
    const title = await this.page.title();
    if (resp?.status() === 403 || /WAF|拦截/.test(title)) {
      return { html: null, waf: true };
    }
    return { html: await this.page.content(), waf: false };
  }
}

Main Loop: Batch + Cooldown + Resume

while (true) {
  if (wafCooldown && Date.now() < wafCooldown) {
    console.log('[cooldown] waiting'); break;  // hand off to crontab
  }
  const task = q.next();
  if (!task) break;

  const { html, waf } = await client.fetch(task.url);
  if (waf) { q.fail(task.url, 'WAF blocked'); wafCooldown = Date.now() + 3*60*1000; continue; }

  if (task.type === 'list') {
    const urls = parseListUrls(html);
    q.enqueueMany(urls.map(u => ({ url: u, type: 'article', priority: 20 })));
    q.done(task.url);
  } else if (task.type === 'article') {
    const art = parseArticle(html, task.url);
    saveArchive(art, html);         // persist raw HTML
    saveKnowledge(art);             // persist cleaned Markdown
    enqueueArticleImages(q, art);   // enqueue images
    q.done(task.url);
  }
}

Reliability Engineering: From Daemon Script to crontab

Pitfall 1: A bash daemon script deadlocking

Initially a crawl-loop.sh long-running process handled “batch → cooldown → batch” in an inner loop. Hours in, a batch’s node process hung, and the daemon’s cleanup logic cascade-deadlocked — zero output for 6 hours.

Root cause: pkill -f "src/index.js" inside the daemon collided with the batch-stat call node src/index.js --stats, corrupting state.

Fix: switch to crontab-driven batches. Each run is an independent short-lived process (max 240s, force-killed on timeout), triggered by system crontab every 8 minutes. Processes are naturally isolated; a single batch crash doesn’t affect the next:

# crawl-batch.sh — single batch, PID-lock to prevent re-entry, 240s timeout guard
./crawl-batch.sh   # fetch 3 pages, then exit
PATH=/opt/homebrew/bin:/usr/bin:/bin
*/8 * * * * /Users/zhangshengrui/projects/xitek-save/crawler/crawl-batch.sh

Pitfall 2: An LLM-driven cron guardian is unreliable

I initially used an OpenClaw cron (agentTurn type) to guard the process. Once the LLM call timed out and the guardian didn’t run, the process died overnight with no one to restart it. Switching to system crontab calling raw shell — no LLM, no OpenClaw dependency — gave the highest reliability.

Three-Layer Storage

LayerLocationContentPurpose
Raw archivearchive/Full HTML + imagesOffline browsing, faithful backup
Structured DBdata/xitek.dbQueue + metadataResume, retrieval index
Knowledge baseknowledge/reviews/<year>/<id>.mdCleaned bodyFeed to RAG
// persist cleaned body as Markdown for RAG
function saveKnowledge(art) {
  const y = (art.date || 'unknown').slice(0, 4);
  const file = `knowledge/reviews/${y}/${art.id || Date.now()}.md`;
  const md = [
    `# ${art.title}`,
    `- URL: ${art.url}`,
    `- 日期: ${art.date}`,
    `- 作者: ${art.author}`,
    ``,
    `## 正文`, ``,
    art.bodyText,
    ``, `## 图片`,
    ...(art.images || []).map(i => `- ${i}`),
  ].join('\n');
  writeFileSync(file, md);
}

Current Status & Next Steps

The pipeline is in stable production crawling. All 152 listing pages are done; 100+ article details are persisted (both raw HTML and knowledge Markdown), and the queue auto-advances from listing → detail → image, with resume + crontab keeping it running non-stop.

Planned next:

  1. Full reviews: persist all ~1,642 details + images
  2. Expand to P1/P2: other info columns, vision/she/travel channels
  3. RAG build: chunk knowledge/*.md → vectorize → feed the existing rag-service → searchable photography knowledge base
  4. Offline browsing: the archive/ structure can serve as a static site for offline reading

Full code and run config live in ~/projects/xitek-save/, with the plan in docs/PLAN.md.

The most valuable thing in rescue crawling isn’t the code — it’s an attitude of respecting content. When a platform carrying twenty years of community memory is about to vanish, saving it completely, reliably, and searchably is itself meaningful long-term work.

AI
👋 Hi, I'm Xiaorui. Feel free to chat with me!
Chat with me →