[Bug]: md-fit strips meaningful content metadata (usernames, attribution) not just boilerplate

[mattheworiordan]

crawl4ai version

0.8.6

Expected Behavior

md-fit should strip page chrome (navigation, footers, sidebars, cookie banners) but preserve meaningful content metadata like author names and attribution on user-generated content. Comment attribution (who said what) is semantic content, not boilerplate.

Current Behavior

md-fit strips comment author usernames and profile links from GitHub pages while keeping the comment body and date. The output shows commented Apr 6, 2026 with no indication of who wrote it.

With -o md, you get:

**[mattheworiordan](https://gist.github.com/mattheworiordan)** commented Apr 6, 2026

With -o md-fit, you get:

commented Apr 6, 2026

The username and profile link are gone. On pages with multiple commenters, it's impossible to tell who said what.

Is this reproducible?

Yes

Inputs Causing the Bug

- URL: https://gist.github.com/mattheworiordan/99ba717f7a8ca5a7f838913722cfe7ac
- Settings: `-o md-fit` (default fit mode, no other flags)

Steps to Reproduce

# 1. Crawl the gist with full markdown - username preserved
crwl crawl https://gist.github.com/mattheworiordan/99ba717f7a8ca5a7f838913722cfe7ac -o md

# Look for: **[mattheworiordan](...)** commented Apr 6, 2026

# 2. Same URL with md-fit - username stripped
crwl crawl https://gist.github.com/mattheworiordan/99ba717f7a8ca5a7f838913722cfe7ac -o md-fit

# Look for: commented Apr 6, 2026  (no username)

Code snippets

# Not a code bug - this is CLI / markdown generation behaviour.
# Reproduces purely via crwl CLI as shown above.

OS

macOS (Darwin 25.2.0)

Python version

3.14.3

Browser

(default Chromium managed by crawl4ai)

Browser version

(crawl4ai default)

Error logs & Screenshots (if applicable)

n/a

[hafezparast]

This is expected behavior of the PruningContentFilter — it scores each HTML block on text density, link ratio, tag weight, and class patterns. Short metadata elements (usernames, timestamps) score low on density and get pruned alongside actual boilerplate.

On your GitHub gist page, the author <span class="author"> is short text, heavily linked, and the class matches the "comment" negative pattern — all of which push the score below the 0.48 threshold.

PR #1904 adds preserve_classes and preserve_tags params to PruningContentFilter so you can whitelist elements that should always be kept:

from crawl4ai import CrawlerRunConfig, DefaultMarkdownGenerator
from crawl4ai.content_filter_strategy import PruningContentFilter

config = CrawlerRunConfig(
    markdown_generator=DefaultMarkdownGenerator(
        content_filter=PruningContentFilter(
            preserve_classes=["author", "comment-header"],
            preserve_tags=["time"],
        )
    )
)

Immediate workarounds (before the PR merges):

1. Lower the threshold: PruningContentFilter(threshold=0.35) — keeps more content but also more noise
2. Use BM25ContentFilter(user_query="author comments") instead — relevance-based, not density-based

To test the whitelist before merge:

pip install git+https://github.com/hafezparast/crawl4ai.git@fix/maysam-pruning-preserve-whitelist-1900