fix(crawler): skip body visibility check when ignore_body_visibility=True (#2129) - #2192
Open
Battleplus wants to merge 2 commits into
Open
fix(crawler): skip body visibility check when ignore_body_visibility=True (#2129)#2192Battleplus wants to merge 2 commits into
Battleplus wants to merge 2 commits into
Conversation
…True (unclecode#2129) When ignore_body_visibility=True (the default), the crawler still waited for body to become attached (30s timeout) and ran a visibility check, only to discard the result. This added a hardcoded 30s penalty on every page where body is never visible (e.g., AngularJS ng-cloak, Vue v-cloak). Wrap the entire body visibility block in `if not config.ignore_body_visibility` so the default path skips it entirely. Remove redundant inner guards. Closes unclecode#2129
…ity-delay Resolve crawl4ai/async_crawler_strategy.py: keep main's configurable body_visibility_timeout for the visibility wait, and apply this PR's skip of both body waits when ignore_body_visibility=True (unclecode#2129). The unclecode#2144 wasted-wait warning is superseded by the skip: with the waits skipped there is no budget-burning wait left to warn about. Signed-off-by: Battleplus <3559424769@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2129 — When ignore_body_visibility=True (the default), the crawler still waited for body to become attached (30s timeout) and ran a visibility check, only to discard the result. This added a hardcoded 30s penalty on every page where body is never visible.
Root cause
The body visibility check block (wait_for_selector + csp_compliant_wait) ran unconditionally, then the result was checked against config.ignore_body_visibility. On pages with ng-cloak/v-cloak where body never becomes visible, this wasted 30 seconds every crawl.
Fix
Wrap the entire body visibility block in
if not config.ignore_body_visibilityso the default path skips it entirely. Remove redundant inner guards that are now unreachable.Files changed