Issue with anti-bot detection #1905

thomas-doerr-lexware · 2026-04-07T19:00:34Z

thomas-doerr-lexware
Apr 7, 2026

Hello,

I updated today from version 0.6.2 to the newest 0.8.5. A lot of my test pages are working as before, but I noticed that for some URLs I get the error message "Blocked by anti-bot protection: Structural: no tag".
Pages that from my perspective are not running an anti-bot protection.

I noticed, that the problem started with version 0.7.0.
And after a few checks, my assumption is, that Wordpress pages building with the theme Avada are facing this problem.

Since there are a lot of changes beween 0.6.2 and 0.8.5, can someone with a running system, crawl4ai 0.8.5, double check if the following websites can be scraped without any anti-bot protection problem?

www.debeuf.de
www.timo-doerr.com
avada.com
knuddelkinder.de
engel-pfaffenweiler.de

I get for all of them the error message "Blocked by anti-bot protection: Structural: no tag" and I'm not sure, if I should create an issue since it is a bug or if I need to search in my code. I already tried to change my configuration, but without any change.

Thanks in advance!
Thomas Dörr

artur282 · 2026-09-11T20:24:05Z

artur282
Sep 11, 2026

The Structural: no <body> tag reason comes from the third tier of the detector, not from a real block response: crawl4ai/antibot_detector.py

def _structural_integrity_check(html: str) -> Tuple[bool, str]:
    ...
    html_len = len(html)

    # Skip large pages (unlikely to be block pages) and data responses
    if html_len > _STRUCTURAL_MAX_SIZE or _looks_like_data(html):
        return False, ""

    signals = []

    # Signal 1: No <body> tag - definitive structural failure
    if not _BODY_RE.search(html):
        return True, f"Structural: no <body> tag ({html_len} bytes)"

Two properties of that check matter for your case:

It only runs on responses under _STRUCTURAL_MAX_SIZE (50 KB) that do not look like JSON/XML, and it is applied to crawl_result.html after the fetch/render (async_webcrawler.py, the "Mark blocked results as failed" block). So a page whose captured HTML is a short shell — e.g. the pre-hydration document, a redirect stub, or markup whose <body> disappeared in the captured fragment — is reported as blocked even though the request itself was fine. Avada/WordPress pages that load most of their content through their own scripts hit exactly this.
It is skipped when the result came from raw: URLs, from a successful fallback_fetch_function, when placeholder_html is set, or when the response is a binary download. That is the intended escape path.

How to tell whether it is a false positive: result.crawl_stats records the reason per proxy attempt (proxies_used[].reason) and resolved_by; if result.html does contain real markup and only the <body> token is missing, it is markup-shape, not a bot wall.

Workarounds, cheapest first:

Render longer before the check: CrawlerRunConfig(wait_for="css:body", page_timeout=60000, delay_before_return_html=1.0, scan_full_page=True), plus BrowserConfig(use_persistent_context=True, user_data_dir=..., enable_stealth=True, magic=True). Note magic=True and simulate_user=True/override_navigator=True live on CrawlerRunConfig, and enable_stealth cannot be combined with browser_mode="builtin" on a managed browser (it raises).
Provide your own HTML when the fetch is the problem: fallback_fetch_function (HTTP fetch + a real browser elsewhere) is treated as authoritative and skips the re-check.
If you need the previous behaviour while you investigate, pin crawl4ai==0.6.2.

The detector itself could be a bit less absolute here: _BODY_RE requires a literal <body> token, so a valid fragment or a shell that the browser would hydrate is classified as a block. Making tier 3 opt-out (or letting it fall back to a DOM-presence check on the rendered page) would turn these into recoverable cases — I would support that change if a maintainer wants it.

0 replies

Navigation Menu

Uh oh!

Issue with anti-bot detection #1905

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Issue with anti-bot detection #1905

Uh oh!

thomas-doerr-lexware Apr 7, 2026

Replies: 1 comment

Uh oh!

artur282 Sep 11, 2026

thomas-doerr-lexware
Apr 7, 2026

artur282
Sep 11, 2026