[Bug]: total_score, head_data is not always populated for LinkPreviewConfig

[bkennedy-improving]

crawl4ai version

0.7.8

Expected Behavior

Per Documentation:

3. Total Score - Smart Combination

Intelligently combines intrinsic and contextual scores with fallbacks:

• When both scores available: (intrinsic * 0.3) + (contextual * 0.7)
• When only intrinsic: uses intrinsic score
• When only contextual: uses contextual score
• When neither: not calculated

---

head_data is returned.

Current Behavior

Below is an example output:

As you can see there is no intelligent fallback even though an intrinsic score is available.

Related, but it seems like I'm not getting head_data back either. That could be the pdf files I'm looking at.

{
  "https://www.pa.gov/en/grants/search/grant-details/dced/9": [
    {
      "url": "https://dced.pa.gov/programs/ben-franklin-technology-development-authority-venture-investment-program",
      "text": "Program Page",
      "intrinsic_score": 5.0,
      "contextual_score": null,
      "total_score": null,
      "head_data": null
    },
    {
      "url": "https://grants.pa.gov/",
      "text": "Go to Application  (opens in a new tab)",
      "intrinsic_score": 4.0,
      "contextual_score": null,
      "total_score": null,
      "head_data": null
    },
    {
      "url": "https://dced.pa.gov/download/bftda-venture-investment-program-guidelines?wpdmdl=87903",
      "text": "Program Guidelines",
      "intrinsic_score": 4.0,
      "contextual_score": null,
      "total_score": null,
      "head_data": null
    },
    {
      "url": "https://www.pa.gov/privacy-policy",
      "text": "Privacy Policy(opens in a new tab)",
      "intrinsic_score": 3.5,
      "contextual_score": null,
      "total_score": null,
      "head_data": null
    },
    {
      "url": "https://www.pa.gov/en/agencies/dced.html",
      "text": "Visit the DCED Website  (opens in a new tab)",
      "intrinsic_score": 2.7857142857142856,
      "contextual_score": null,
      "total_score": null,
      "head_data": null
    }
  ]
}

Additionally, from an 'apples to apples' comparison, total_score shouldn't just be the intrinsic value if contextual isn't available, but rather the weighted intrinsic value. It seemed like from the documentation total_score would simply represent the raw intrinsic score (if 5, then return 5, not 5 x .3 = 1.5)

Scenario	Apples-to-Apples Approach	Resulting Formula / Score
Both available	Weighted average	$(intrinsic \times 0.3) + (contextual \times 0.7)$
Only intrinsic	Uses weighted intrinsic score	$intrinsic \times 0.3$
Only contextual	Uses weighted contextual score	$contextual \times 0.7$

Is this reproducible?

Yes

Inputs Causing the Bug

See example above.

Steps to Reproduce

See Code Snippet

Code snippets

The relevant code looks something like this:


        md_generator = DefaultMarkdownGenerator()
        config = CrawlerRunConfig(
            url_matcher=str(t.url),
            markdown_generator=md_generator,
            excluded_tags=['nav', 'footer', 'header'],
            extraction_strategy=llm_strategy,
            cache_mode=CacheMode.BYPASS,
            stream=True,
            score_links=True,
            exclude_all_images=True,
            link_preview_config=LinkPreviewConfig(
                include_internal=True,
                include_external=True,
                max_links=20,
                concurrency=5,
                timeout=10,
                query='my query here'
                score_threshold=0.2,
            ),
        )


Happy to provide more details privately.

OS

Linux

Python version

3.12.3

Browser

Chrome

Browser version

144.0.7559.59

Error logs & Screenshots (if applicable)

No response

[Ahmed-Tawfik94]

Root Cause Analysis

• Primary cause:
  The _merge_head_data() function in crawl4ai/link_preview.py only calculates total_score for links that are successfully processed and present in url_to_head_data. Links that fail head extraction (e.g., PDFs, timeout errors, non-HTML content) are not in this dictionary and therefore never get their total_score calculated, even when they have a valid intrinsic_score.

• Contributing factors:

  1. Selective score calculation: Lines 330-335 and 368-373 only calculate total_score inside the if link.href in url_to_head_data: block
  2. Missing else branch: The else clauses (lines 338-340 and 376-378) simply append the original link without calculating total_score
  3. Documentation ambiguity: The documentation states "uses intrinsic score" but doesn't clarify if this means raw or weighted

• Evidence:

  • Line 330-335 (internal links): total_score calculated only when link is in url_to_head_data
  • Line 338-340 (internal links): else branch appends link unchanged - no total_score calculation
  • Line 368-373 (external links): Same pattern - only calculated for successful head extractions
  • Line 376-378 (external links): else branch appends link unchanged
  • The calculate_total_score() function correctly handles the "intrinsic only" case (line 3528-3530 in utils.py), but it's never called for links without head data