[Bug]: BFSDeepCrawlStrategy re-scans the full level for parent lookup (O(n²)); BestFirstCrawlingStrategy can enqueue the same URL twice

[yashikam19]

crawl4ai version

0.9.3

Expected Behavior

• Per-level bookkeeping in BFSDeepCrawlStrategy (matching a fetched result back to its parent URL) should scale linearly with the number of URLs in a level.
• A URL discovered by multiple pages in BestFirstCrawlingStrategy should be scored and queued once, not once per inbound link.

Current Behavior

1. O(n²) parent lookup - deep_crawling/bfs_strategy.py, _arun_batch and _arun_stream:

parent_url = next((parent for (u, parent) in current_level if u == url), None)

This linearly re-scans the entire current_level list once per fetched result. For a level of N URLs this is O(N²) pure-Python work with no I/O involved - a page with high fan-out (hundreds/thousands of links) makes this bookkeeping step dominate.

2. Duplicate enqueue - deep_crawling/bff_strategy.py, link_discovery/ _arun_best_first:
   visited is only populated when an item is dequeued, not when it's discovered, so link_discovery's dedup check (if base_url in visited: continue) doesn't stop the same URL being pushed onto the priority queue twice if two different pages link to it before it's first processed. The duplicate is silently dropped later at dequeue time, so output is still correct, but it was scored and queued for nothing.

Is this reproducible?

Yes

Inputs Causing the Bug

- URL(s): Any site with high link fan-out per page, or shared links between sibling pages
- Settings used: BFSDeepCrawlStrategy(max_depth=2, max_pages=1000+) for issue 1; BestFirstCrawlingStrategy(...) with a url_scorer for issue 2

Steps to Reproduce

1. Run `BFSDeepCrawlStrategy` against a page with hundreds+ of internal links at depth 1; profile `_arun_batch`'s per-result loop - the parent lookup dominates as level size grows.
2. Run `BestFirstCrawlingStrategy` against a small site where ≥2 pages link to the same third page before it's first crawled; log `queue.qsize()` or inspect enqueued items - the shared URL appears twice.

Code snippets

OS

macOS 26.6.2 (Darwin 25.6.0)

Python version

3.13.12

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[yashikam19]

Opened a fix in #2244