Skip to content

Fix false positive copyright detection from Unicode surrogates#4664

Open
Tarun-goswamii wants to merge 2 commits intoaboutcode-org:developfrom
Tarun-goswamii:fix-surrogate-false-positive-copyright
Open

Fix false positive copyright detection from Unicode surrogates#4664
Tarun-goswamii wants to merge 2 commits intoaboutcode-org:developfrom
Tarun-goswamii:fix-surrogate-false-positive-copyright

Conversation

@Tarun-goswamii
Copy link

@Tarun-goswamii Tarun-goswamii commented Jan 4, 2026 •
edited
Loading

Fixes #4381

Tasks

  • Reviewed contribution guidelines
  • PR is descriptively titled 📑 and links the original issue above 🔗
  • Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
    Run tests locally to check for errors.
  • Commits are in uniquely-named feature branch and has no merge conflicts 📁
  • Updated documentation pages (if applicable)
  • Updated CHANGELOG.rst (if applicable)

Tarun-goswamii pushed a commit to Tarun-goswamii/scancode-toolkit that referenced this pull request Jan 4, 2026
@tarun111111
Fix false positive copyright detection from Unicode surrogates
54df6f8 
Remove Unicode surrogate characters (U+D800-U+DFFF) from text before
copyright detection to prevent false positives like '(c) \Truei (c) Y'
that occur when surrogate bytes are misinterpreted.

This fixes issue aboutcode-org#4664 where files containing surrogate character
ranges (like busybox-1.37.0/docs/unicode_full-bmp.txt) were incorrectly
detected as having copyright content.

Changes:
- Add SURROGATE_PATTERN regex constant to match U+D800-U+DFFF range
- Add sanitize_line_for_detection() function to strip surrogates
- Integrate sanitization in detect_copyrights_from_lines()
- Add test suite for surrogate handling

Fixes: aboutcode-org#4664
@Tarun-goswamii Tarun-goswamii force-pushed the fix-surrogate-false-positive-copyright branch from 6d1df73 to 54df6f8 Compare January 4, 2026 14:20
Tarun-goswamii pushed a commit to Tarun-goswamii/scancode-toolkit that referenced this pull request Jan 4, 2026
@tarun111111
Fix false positive copyright detection from Unicode surrogates
e4ff487 
Remove Unicode surrogate characters (U+D800-U+DFFF) from text before
copyright detection to prevent false positives like '(c) \Truei (c) Y'
that occur when surrogate bytes are misinterpreted.

This fixes issue aboutcode-org#4664 where files containing surrogate character
ranges (like busybox-1.37.0/docs/unicode_full-bmp.txt) were incorrectly
detected as having copyright content.

Changes:
- Add SURROGATE_PATTERN regex constant to match U+D800-U+DFFF range
- Add sanitize_line_for_detection() function to strip surrogates
- Integrate sanitization in detect_copyrights_from_lines()
- Add test suite for surrogate handling

Fixes: aboutcode-org#4664
Signed-off-by: tarun111111 <tarunpuri2544@gmail.com>
@Tarun-goswamii Tarun-goswamii force-pushed the fix-surrogate-false-positive-copyright branch from 54df6f8 to e4ff487 Compare January 4, 2026 14:23
Tarun-goswamii pushed a commit to Tarun-goswamii/scancode-toolkit that referenced this pull request Jan 4, 2026
@tarun111111
Fix false positive copyright detection from Unicode surrogates
1861e13 
Remove Unicode surrogate characters (U+D800-U+DFFF) from text before
copyright detection to prevent false positives like '(c) \Truei (c) Y'
that occur when surrogate bytes are misinterpreted.

This fixes issue aboutcode-org#4664 where files containing surrogate character
ranges (like busybox-1.37.0/docs/unicode_full-bmp.txt) were incorrectly
detected as having copyright content.

Changes:
- Add SURROGATE_PATTERN regex constant to match U+D800-U+DFFF range
- Add sanitize_line_for_detection() function to strip surrogates
- Integrate sanitization in detect_copyrights_from_lines()
- Add test suite for surrogate handling

Fixes: aboutcode-org#4664
Signed-off-by: tarun111111 <tarunpuri2544@gmail.com>
@Tarun-goswamii Tarun-goswamii force-pushed the fix-surrogate-false-positive-copyright branch from e4ff487 to 1861e13 Compare January 4, 2026 14:28
Tarun-goswamii pushed a commit to Tarun-goswamii/scancode-toolkit that referenced this pull request Jan 4, 2026
@tarun111111
Fix false positive copyright detection from Unicode surrogates
08b4010 
Remove Unicode surrogate characters (U+D800-U+DFFF) from text before
copyright detection to prevent false positives like '(c) \Truei (c) Y'
that occur when surrogate bytes are misinterpreted.

This fixes issue aboutcode-org#4664 where files containing surrogate character
ranges (like busybox-1.37.0/docs/unicode_full-bmp.txt) were incorrectly
detected as having copyright content.

Changes:
- Add SURROGATE_PATTERN regex constant to match U+D800-U+DFFF range
- Add sanitize_line_for_detection() function to strip surrogates
- Integrate sanitization in detect_copyrights_from_lines()
- Add test suite for surrogate handling

Fixes: aboutcode-org#4664
Signed-off-by: tarun111111 <tarunpuri2544@gmail.com>
@Tarun-goswamii Tarun-goswamii force-pushed the fix-surrogate-false-positive-copyright branch from 1861e13 to 08b4010 Compare January 4, 2026 14:33
@Tarun-goswamii Tarun-goswamii changed the title Fix false positive copyright detection from Unicode surrogates (#4381) Fix false positive copyright detection from Unicode surrogates (#4664) Jan 4, 2026
@tarun111111
Fix false positive copyright detection from Unicode surrogates
18cc347 
Remove Unicode surrogate characters (U+D800-U+DFFF) from text before
copyright detection to prevent false positives like '(c) \Truei (c) Y'
that occur when surrogate bytes are misinterpreted.

This fixes issue aboutcode-org#4381 where files containing surrogate character
ranges (like busybox-1.37.0/docs/unicode_full-bmp.txt) were incorrectly
detected as having copyright content.

Changes:
- Add SURROGATE_PATTERN regex constant to match U+D800-U+DFFF range
- Add sanitize_line_for_detection() function to strip surrogates
- Integrate sanitization in detect_copyrights_from_lines()
- Add test suite for surrogate handling

Fixes: aboutcode-org#4381
Signed-off-by: tarun111111 <tarunpuri2544@gmail.com>
@Tarun-goswamii Tarun-goswamii force-pushed the fix-surrogate-false-positive-copyright branch from 08b4010 to 18cc347 Compare January 4, 2026 15:33
@Tarun-goswamii Tarun-goswamii changed the title Fix false positive copyright detection from Unicode surrogates (#4664) Fix false positive copyright detection from Unicode surrogates Jan 4, 2026
@Tarun-goswamii
Copy link
Author

Tarun-goswamii commented Jan 4, 2026

Regarding the failing license_validate_ignorables tests
The 5 failing tests (license_validate_ignorables_1 through _5) are unrelated to this PR.

This PR only modifies:
copyrights.py - adds surrogate character sanitization for copyright detection
test_copyrights_surrogate.py - new test file for the fix
The license_validate_ignorables tests validate the license database/index and have no connection to copyright detection code.

Tests relevant to this PR have passed:
✅ core_tests cluecode_license_base (includes copyright tests)
✅ All platform tests (ubuntu22, ubuntu24, win2022, win2025, macos14)
✅ DCO check

@chinyeungli Could you please confirm if the license_validate_ignorables failures are a known/pre-existing issue on the develop branch?

@chinyeungli chinyeungli mentioned this pull request Jan 5, 2026
Open
@Tarun-goswamii
Copy link
Author

Tarun-goswamii commented Jan 10, 2026

@chinyeungli , @AyanSinhaMahapatra
Hello Sir, I wanted to kindly follow up on this PR. I’ve already shared details
via email, but I understand everyone may be busy. Since this fix addresses
false positives in copyright detection from Unicode surrogates, I’d really
appreciate a review when you have the time.

Please let me know if there are any adjustments or clarifications needed
from my side to help move this forward. Thank you for your guidance and support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AyanSinhaMahapatra AyanSinhaMahapatra Awaiting requested review from AyanSinhaMahapatra

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Potential noise from copyright detection

2 participants

@Tarun-goswamii @tarun111111