xitek.com Rescue Archive Project

Before a 20-year-old Chinese photography community went offline, I built a resumable Playwright crawler that defeats WAF rate-limiting to rescue-archive over a thousand gear reviews — laying the groundwork for a personal RAG knowledge base.

This project grew out of a reflection on the lifecycle of digital content.

I’ve worked in the photography field, and xitek.com is a platform I know well. When I learned it was shutting down, my first thought wasn’t sentimentality — it was recognizing that two decades of deep gear reviews were first-hand material you can’t find anywhere else, about to vanish with the site forever. Rescue archiving began from there.

Project Reflections

Why it’s worth saving

Photography reviews differ from news — they live on measured data, head-to-head comparisons, and historical lineage: how a lens stacks up against its contemporaries, how a brand’s product line evolved. The information density is high, and much of it was never syndicated anywhere else. Once the site goes offline, this content disappears from the internet permanently. That’s the textbook rescue scenario: content that is valuable, durable, and irreplaceable.

The hard part isn’t scraping — it’s the anti-scraping

I assumed mirroring a site was a matter of “downloading HTML.” In reality I hit three layers of resistance:

1. WAF fingerprint check — plain HTTP clients are blocked outright; only a real browser passes
2. IP frequency rate-limiting — roughly 3-4 requests trigger interception
3. Neither new IPs nor new fingerprints bypass it — verified repeatedly via controlled experiments

This reframed my understanding of anti-scraping: it doesn’t aim to block all crawlers, it aims to push the cost of crawling above a sustainable threshold. The way to win isn’t clever tricks — it’s to accept the cadence: strict rate-limiting, enforced cooldown, and long, low-intensity operation. A mindset of trading speed for stability.

Engineering principles for rescue crawling

This project distilled several reusable principles:

• Resumability is the lifeline: interrupt at any moment and resume exactly, never re-crawl, never miss. A SQLite queue + state machine is the core.
• Short-lived processes beat long-running daemons: a bash daemon deadlocks and crashes hard to recover; crontab firing an independent short batch every 8 minutes means a single batch failure can’t disrupt the next — the most fault-tolerant design.
• Never depend on unreliable orchestration: an LLM-driven cron guardian once timed out and nobody was home. Switching to system crontab calling raw shell removed the LLM dependency entirely.

What’s Next

This project doesn’t stop at “done archiving” — it’s the start of a content-asset pipeline:

• Personal photography knowledge base (RAG): chunk the archived Markdown, vectorize it, and feed the existing rag-service — so questions like “how does lens X perform” or “compare these two bodies” get direct answers.
• Reviews database: structure out “brand / category / year / tested products” dimensions from the text for gear-trend analysis and brand comparisons.
• Offline archive: the archive/ tree is naturally a static-site structure, deployable as a browsable offline collection.

Long-term Value

The thing that struck me most about rescue crawling is how it turns “storage” from a technical act into an act of respect for content. When a digital platform’s memory faces extinction, whether we can save it completely, reliably, and searchably tests not just engineering skill but judgment about long-term content value. The “anti-scraping + resumable + layered storage” architecture I built here becomes a reusable foundation for all my future data-archiving projects.

Highlights

• Reverse-engineered the WAF rate-limit mechanism (new IP / new fingerprint both useless)
• Three-layer storage: raw HTML + SQLite queue + cleaned Markdown
• crontab-driven + resumable
• stays stable under aggressive anti-scraping
• Long-term value judgment behind preserving content