Skip to content

Fix UTF-8 offsets in sequence patterns - #873

Open
yangfan-yf-yf wants to merge 1 commit into
python-rope:masterfrom
yangfan-yf-yf:fix/match-sequence-utf8-offsets
Open

Fix UTF-8 offsets in sequence patterns#873
yangfan-yf-yf wants to merge 1 commit into
python-rope:masterfrom
yangfan-yf-yf:fix/match-sequence-utf8-offsets

Conversation

@yangfan-yf-yf

Copy link
Copy Markdown
Contributor

Description

I reproduced a crash on master (f005ac786da7d556fdf5733e9b9ddae1eaf27242) when patching a sequence pattern containing non-ASCII text:

from rope.refactor import patchedast

source = "match value:\n    case [é, y]:\n        pass\n"
patchedast.get_patched_ast(source, True)

The walker reads : as the closing delimiter and raises MismatchedTokenError. Nested and empty sequence patterns after non-ASCII text are also affected, and some inputs produce incorrect source regions without raising.

AST columns are UTF-8 byte offsets, while the walker slices a Python string using character offsets. I convert the AST coordinates at the sequence-pattern slicing sites, including the empty-pattern path. The generic SourceLinesAdapter coordinate behavior is preserved.

The regression test covers seven patterns, including non-ASCII captures and literals, nested and empty sequences, bare sequences, and multiline input. It checks both source round-tripping and each sequence node's source region against ast.get_source_segment.

Related to #623; follows the sequence delimiter handling introduced in #863. Delimiter warnings for adjacent comments remain outside this change.

Validation on Windows / Python 3.12.14:

  • All seven new regression scenarios fail before the fix and pass after it.
  • Related modules: 311 passed, 10 skipped; seven subtests passed.
  • Full suite: 2117 passed, 12 skipped, 5 xfailed; seven subtests passed.
  • Target-file pre-commit checks and git diff --check pass.

Checklist

  • I have added tests that prove my fix is effective.
  • I have updated CHANGELOG.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant