lncrawl-scraper 1.1.0 · 2026-07-30 16:45 +04
Every code path in the scraper, exercised against production Cloudflare deployments rather than fixtures — the transport group, the diagnosis classifier, the escalation ladder, a real browser solve, the archive tier, and a live tor-pool. 28 of 31 scenarios pass. A stubbed transport can confirm that the code does what it says; only this can confirm that what it says is true of a real server.
Every base URL in lncrawl's source index was probed twice — once with a
plain requests session, once with the impersonating transport — and each
response classified by the library's own diagnosis. That pairing is the measurement: the
only difference between the two requests is the network signature.
| Edge | Hosts |
|---|---|
| Cloudflare | 159 |
| nginx | 46 |
| cowboy | 30 |
| unreachable | 20 |
| apache | 15 |
| DDoS-Guard | 6 |
| unknown | 6 |
| tengine | 5 |
| litespeed | 4 |
| caddy | 4 |
Of 45 Cloudflare hosts reached through the local tor-pool: 33 served outright, 8 challenged, 0 reputation-blocked.
A negative result worth keeping: the reputation layer is not uniformly hostile to Tor on this corpus. The model treats a Tor exit as clearing nothing at layer 1, which is conservative rather than measured — and the consequence is that a browser solver buys more here than a residential proxy would.
requests
impersonated transport
Reading the chart: the impersonated client meets the managed challenge on far fewer hosts, and meets the lighter scoring tiers less often — those are the layers a network signature answers. It meets the behavioural layer more often, which is not a regression: that is this session's own request volume against the same hosts being rate limited, and it is the one layer no client-side change addresses.
Each one names the layer it exercises and what a pass actually proves — "it returned 200" is not evidence about a detection layer on its own. Expand any row for the individual assertions and the values observed.
What a pass proves. Layers 2-5 are one barrier: the only difference between these two requests is the network signature, and it decides the outcome.
looked up from the last probeWhat a pass proves. A stale impersonation profile predates the post-quantum key share current builds send, so pinning one contradicts the User-Agent it claims.
What a pass proves. The baseline tier handles a real protected site end to end, including the soup, JSON and file helpers.
https://novelfull.net/What a pass proves. The status code says 'forbidden' and the correct remedy is a browser, not a new address. A conventional client rotates its proxy here and gets nowhere.
https://centralnovel.com/What a pass proves. The failure is actionable: it names the layer and the capability that would reach it, instead of exhausting retries on a 403.
https://centralnovel.com/What a pass proves. Nothing distinguishes the scoring tiers from outside, so the diagnosis names the strictest emit-only one and asks for a stronger tier — not a new address.
https://novelmania.com.br/What a pass proves. The layer reads a secret. Retrying is an infinite loop against a wall, so a single request is spent and the message names the only legitimate route.
no host in the last probe is behind an identity-provider gate
What a pass proves. The classifier is exercised against responses a real server produced, not fixtures.
https://httpbingo.orgWhat a pass proves. The site's answer about a path says nothing about the client. Attributing it to a layer would retire a healthy address over a typo in a URL.
https://novelfull.net/What a pass proves. The population-level result: what 501 real hosts do today, and what the transport profile is worth measured across all of them rather than anecdotally.
501 hosts from lncrawl's source indexWhat a pass proves. The classifier knows the rest of the mitigation market, and identification alone never turns a host that serves content into a refusal.
What a pass proves. A deep page arriving with no referrer and no prior history is a navigation pattern no person produces; both mechanisms are observable in the requests actually sent.
https://httpbingo.orgWhat a pass proves. The rule that matters most: a 429 says the address works and is being asked for too much, so the remedy is arithmetic — not a new exit, which would reset the history the layer measures.
https://httpbingo.orgWhat a pass proves. The possess-side veto holds even when a rotation is cheaply available: discarding the address would reset the accumulated history the layer reads.
https://httpbingo.orgWhat a pass proves. A process that forgets cannot accumulate, and the behavioural layer reads exactly what accumulates. The binding layer is the most valuable thing to keep.
https://centralnovel.com/What a pass proves. Separate state would present as two visitors who contradict each other, arriving in bursts, one of them always cold.
https://httpbingo.orgWhat a pass proves. The pool is wired end to end: the session key becomes the SOCKS username, the credential travels as its password, and the egress IP is a Tor exit.
https://check.torproject.org/api/ipWhat a pass proves. Stickiness is what makes a clearance reusable and lets per-zone history accrue. A rotating address would invalidate both.
https://check.torproject.org/api/ipWhat a pass proves. Rotation is a reassignment inside the pool: the endpoint URL is unchanged, but the address behind it — and therefore the identity — is not.
https://check.torproject.org/api/ipWhat a pass proves. This is the only signal that catches a soft block — a proxy relaying bytes cannot see a 403 or a captcha inside an HTTPS tunnel — and the pool weighs a report by its kind.
http://127.0.0.1:8080What a pass proves. THE headline rule. Every Tor exit is on the same published lists, so a reputation block cannot be fixed by another one — and the decision is driven by what the running pool actually offers, not by a hardcoded assumption.
http://127.0.0.1:8080What a pass proves. A negative result worth recording: assuming every Cloudflare host refuses Tor would make the address strategy look more important than it is on this corpus.
45 Cloudflare hosts via tor-poolWhat a pass proves. The naturally-occurring case: this host bans the machine's whole ASN, so no client change of any kind can help. With no alternative address the stop is immediate rather than after spending the rotation budget on the same address.
https://www.readwn.com/the host is not banning this network right now: accept
What a pass proves. The cheapest way past a protected site is not to touch it. The response must carry the real URL, or relative links redirect the whole crawl into the snapshot.
https://centralnovel.com/What a pass proves. The planner walks up only as far as required, and the archive rescues a host the live stack refuses.
https://centralnovel.com/What a pass proves. solve-once-and-reuse: the expensive tier runs once and the clearance is replayed on the identity that earned it, so later pages cost one HTTP request.
https://centralnovel.com/What a pass proves. the one capability HTTP fidelity cannot substitute for, and the one failure that is silent: a shell parses to nothing and raises nothing.
https://www.wuxiaworld.com/What a pass proves. The only layer that returns no error. Following a decoy link poisons the store and flags the session, and nothing in the response says so.
https://novelfull.net/What a pass proves. The classic rotating-proxy failure, made structurally impossible: the cookie is bound to the address, User-Agent and TLS profile together.
What a pass proves. The one layer with no bypass. Deployed fail-open today, so a valid signature is a positive identification that skips the challenge machinery entirely.
https://httpbingo.orgWhat a pass proves. A failure on our side of the proxy must not become a detection story. The SOCKS5 handshake has no status code, so this arrives as a bare transport error — and attributing it to layer 1 caused a rotation, a false 'blocked' report against a healthy exit, and a persisted verdict that the site refuses our address.
tor-pool with a deliberately wrong tokentor-pool at http://127.0.0.1:8080 runs with AUTH_DISABLED, so any credential is accepted - restart it with AUTH_DISABLED=false to run this
| scraper | 1.1.0 |
| Python (harness) | Python 3.9.6 |
| Python (browser tier) | Python 3.12.13 |
| curl_cffi | 0.13.0 |
| impersonation profile | chrome136 |
| nodriver | 0.50.3 |
| browser | Google Chrome (headed) |
| tor-pool | ghcr.io/lncrawl/tor-pool:latest |
| tor-pool exits | 5 instances |
| local egress | 91.73.67.32 |
uv run poe live-probe # classify every host in lncrawl's source index (~4 min) uv run poe live-tor # classify Cloudflare hosts through tor-pool uv run poe live # run every scenario uv run poe live-report # rebuild this page # the clearance tier needs its own interpreter (nodriver: Python 3.10-3.13) and Chrome /tmp/scr312/bin/python livetest/clearance.py
Requirements and the politeness rules are in
livetest/README.md. Targets are looked up from the probe rather than
hardcoded, because site configuration moves: one host in this corpus switched from
Turnstile to plain scoring between two runs an hour apart.