Skip to content

fix(scraping): run post-processing on the selected content, not body - #2247

Open
Yasser-Ameur wants to merge 1 commit into
unclecode:developfrom
Yasser-Ameur:fix/post-processing-skipped-with-selector
Open

fix(scraping): run post-processing on the selected content, not body#2247
Yasser-Ameur wants to merge 1 commit into
unclecode:developfrom
Yasser-Ameur:fix/post-processing-skipped-with-selector

Conversation

@Yasser-Ameur

Copy link
Copy Markdown

Summary

When css_selector or target_elements is set, the post-processing passes
silently stop running. only_text=True leaves <b> in place, and inline
style, onclick, data-* and whole base64 data: image payloads reach
cleaned_html and the markdown verbatim. Without a selector the same options
work, which is what makes it easy to miss.

In LXMLWebScrapingStrategy._scrap, content_element is a copy.deepcopy of
the selector match, so it is detached from body. The passes that follow kept
mutating body, while cleaned_html is serialised from content_element. Nine
of them were affected: the only_text unwrap, base64 img src truncation,
empty-element removal, attribute stripping, mermaid SVG replacement,
style/link/meta/noscript removal, script removal and remove_forms.

_process_element and table extraction deliberately stay on body, because
link and media collection is page-wide by design and an existing test asserts
that.

List of files changed and why

  • crawl4ai/content_scraping_strategy.py - point the post-processing passes at
    the element that is actually serialised.
  • tests/test_selector_post_processing.py - new, mirrors
    tests/test_issue_1484_css_selector.py.

How Has This Been Tested?

pytest tests/test_selector_post_processing.py is 27 passed on this branch and
4 failed, 23 passed with the source reverted, for instance
assert '<style>' not in cleaned finding <style>.job { color: red; }</style>.

Four selector shapes are parametrised, with the boundaries: no selector match,
a single <b> selected, only_text=False, keep_data_attributes=True, the
sidebar still excluded, and links still collected page-wide.

The wider sweep is 152 passed, 16 failed, the 16 being the pre-existing
test_pdf_download_limits.py failures on develop.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

With css_selector or target_elements set, cleaned_html kept <b> under
only_text, emitted inline style, onclick and data-* attributes, empty
elements and whole base64 image payloads, and let a <script>, <style>,
<noscript>, <meta>, <link>, mermaid SVG or <form> inside the selection
survive untouched.

_scrap() builds content_element from a deep copy of the selector match,
which detaches it from body. Nine passes mutated body after that copy:
the only_text unwrap, base64 src truncation, remove_empty_elements_fast,
remove_unwanted_attributes_fast, mermaid SVG replacement, style/link/
meta/noscript removal, script removal and remove_forms. All nine fed a
tree that was never the one lxml.tostring() serialised into
cleaned_html. excluded_tags and excluded_selector run before the copy,
which is why they kept working and masked this.

Point all nine passes at content_element, which is body itself when no
selector is given, so the no-selector path is unchanged. Link, media
and table extraction stay on body so whole-page collection there is
unaffected.
@Yasser-Ameur
Yasser-Ameur marked this pull request as ready for review September 9, 2026 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant