gh-156047: Rank tokens before probing for a keyword typo by johnslavik · Pull Request #156087 · python/cpython

johnslavik · 2026-08-20T09:14:57Z

@pablogsal Look at this! Amazing!

>>> a=b=c=d=e=f=g=h=i=j=k=l=m=0;tpye x = 1;n=o=p=q=r=s=t=u=v=w=x=y=z=2
  File "<python-input-0>", line 1
    a=b=c=d=e=f=g=h=i=j=k=l=m=0;tpye x = 1;n=o=p=q=r=s=t=u=v=w=x=y=z=2
                                ^^^^
SyntaxError: invalid syntax. Did you mean 'type'?

cc @m-aciek

Issue: Keyword typo suggestions differ between file and string tokenizers #156047

johnslavik · 2026-08-20T09:25:36Z

FYI here's logs of what tokens are examined in the typo cases from our tests:

Details

============='with block ad something:\n  pass'=============
rank=1 token.string='ad'
SyntaxError: invalid syntax. Did you mean 'and'?

==================='fur a in b:\n  pass'====================
rank=1 token.string='a'
rank=3 token.string='fur'
SyntaxError: invalid syntax. Did you mean 'for'?

============'for a in b:\n  pass\nelso:\n  pass'============
rank=3 token.string='elso'
SyntaxError: invalid syntax. Did you mean 'else'?

==================='whille True:\n  pass'===================
rank=6 token.string='whille'
SyntaxError: invalid syntax. Did you mean 'while'?

===================='iff x > 5:\n  pass'====================
rank=1 token.string='x'
rank=3 token.string='iff'
SyntaxError: invalid syntax. Did you mean 'if'?

============='if x:\n  pass\nelseif y:\n  pass'=============
rank=1 token.string='y'
rank=3 token.string='elseif'
SyntaxError: invalid syntax. Did you mean 'elif'?

============='tyo:\n  pass\nexcept y:\n  pass'==============
rank=3 token.string='tyo'
SyntaxError: invalid syntax. Did you mean 'try'?

================='classe MyClass:\n  pass'==================
rank=1 token.string='MyClass'
rank=9 token.string='classe'
SyntaxError: invalid syntax. Did you mean 'class'?

========================'impor math'========================
rank=1 token.string='math'
rank=6 token.string='impor'
SyntaxError: invalid syntax. Did you mean 'import'?

====================='form x import y'======================
rank=1 token.string='x'
rank=3 token.string='form'
SyntaxError: invalid syntax. Did you mean 'from'?

========'defn calculate_sum(a, b):\n  return a + b'=========
rank=1 token.string='a'
rank=1 token.string='calculate_sum'
rank=4 token.string='b'
rank=15 token.string='defn'
SyntaxError: invalid syntax. Did you mean 'def'?

==============='def foo():\n  returm result'================
rank=1 token.string='result'
rank=8 token.string='returm'
SyntaxError: invalid syntax. Did you mean 'return'?

====================='lamda x: x ** 2'======================
rank=1 token.string='x'
rank=2 token.string='x'
rank=3 token.string='lamda'
SyntaxError: invalid syntax. Did you mean 'lambda'?

==================='def foo():\n  yeld i'===================
rank=1 token.string='i'
rank=3 token.string='yeld'
SyntaxError: invalid syntax. Did you mean 'yield'?

==============='def foo():\n  globel counter'===============
rank=1 token.string='counter'
rank=9 token.string='globel'
SyntaxError: invalid syntax. Did you mean 'global'?

=================='frum math import sqrt'===================
rank=1 token.string='math'
rank=6 token.string='frum'
SyntaxError: invalid syntax. Did you mean 'from'?

============='asynch def fetch_data():\n  pass'=============
rank=5 token.string='asynch'
SyntaxError: invalid syntax. Did you mean 'async'?

=========='async def foo():\n  awaid fetch_data()'==========
rank=1 token.string='fetch_data'
rank=12 token.string='awaid'
SyntaxError: invalid syntax. Did you mean 'await'?

================'raisee ValueError("Error")'================
rank=1 token.string='ValueError'
rank=12 token.string='raisee'
SyntaxError: invalid syntax. Did you mean 'raise'?

==============='[x for x\nin range(3)\nof x]'===============
rank=1 token.string='of'
SyntaxError: invalid syntax. Did you mean 'if'?

=============='[123 fur x\nin range(3)\nif x]'==============
rank=1 token.string='fur'
SyntaxError: invalid syntax. Did you mean 'for'?

==================='for x im n:\n  pass'====================
rank=1 token.string='im'
SyntaxError: invalid syntax. Did you mean 'in'?

========================='mach x:'==========================
rank=1 token.string='x'
rank=3 token.string='mach'
SyntaxError: invalid syntax. Did you mean 'match'?

========================='math x:'==========================
rank=1 token.string='x'
rank=3 token.string='math'
SyntaxError: invalid syntax. Did you mean 'match'?

===================='match 1:\n  cse 1:'====================
rank=1 token.string='cse'
SyntaxError: invalid syntax. Did you mean 'case'?

======================='typ x = int'========================
rank=1 token.string='x'
rank=3 token.string='typ'
SyntaxError: invalid syntax. Did you mean 'type'?

======================'typed x = int'=======================
rank=1 token.string='x'
rank=3 token.string='typed'
SyntaxError: invalid syntax. Did you mean 'type'?

======================'lazi import x'=======================
rank=1 token.string='x'
rank=8 token.string='lazi'
SyntaxError: invalid syntax. Did you mean 'lazy'?

======================'lezi import x'=======================
rank=1 token.string='x'
rank=8 token.string='lezi'
SyntaxError: invalid syntax. Did you mean 'lazy'?

===================='switch x:\n case:'=====================
rank=1 token.string='x'
rank=3 token.string='switch'
SyntaxError: invalid syntax. Did you mean 'match'?

========================='delete x'=========================
rank=1 token.string='x'
rank=3 token.string='delete'
SyntaxError: invalid syntax. Did you mean 'del'?

======================'function f():'=======================
rank=1 token.string='f'
rank=3 token.string='function'
SyntaxError: invalid syntax. Did you mean 'def'?

========================'func f():'=========================
rank=1 token.string='f'
rank=3 token.string='func'
SyntaxError: invalid syntax. Did you mean 'def'?

========================'void f():'=========================
rank=1 token.string='f'
rank=3 token.string='void'
SyntaxError: invalid syntax. Did you mean 'def'?

='a=a=a=a=a=a=a=a=a=a=0;tpye x = int;z=z=z=z=z=z=z=z=z=z=1'=
rank=1 token.string='x'
rank=3 token.string='tpye'
SyntaxError: invalid syntax. Did you mean 'type'?

We can tweak the rank calculation a bit to look for a token on the left first but I don't think it's a huge deal in this context.

pablogsal

Hummmm…. I am not very convinced about this method. It’s true that’s a bit better but it is still not without inconsistencies

pablogsal · 2026-08-20T09:56:20Z

+                the_end = end_line if line == 0 else end_line + 1
+                if token.start[0] + line != the_end:
+                    continue
+                rank = abs(offset - token.end[1])


I think the rank anchor is a bit off here: self.end_offset is 1-based and relative to the original line, but token.end[1] is 0-based and relative to the dedented snippet, so for indented code the anchor drifts right by the dedent width and the neighbours get probed before the typo (you can see it in your traces, where x ranks above iff). Not incorrect because every candidate is validated by recompiling, but can we translate end_offset to the snippet coordinates before ranking?

I'll also add a case where dedent() makes a difference. For the TYPO_CASES we had, dedent() was a no-op so this was harder to spot.

pablogsal · 2026-08-20T09:56:20Z

+                the_end = end_line if line == 0 else end_line + 1
+                if token.start[0] + line != the_end:
+                    continue
+                rank = abs(offset - token.end[1])


This can prefer the token at the parser caret over the actual typo. retrun a + b now suggests and, because replacing a makes the expression compile. We need a ranking that keeps retrun first and a test for this case.

Somewhat related fun fact, you can trigger this in main too:

def outer(): if True: pass retrun a+b

This is because last_stmt_location in SyntaxError metadata changes to the if True: pass location, making def outer() being cut out and if True: pass\nreturn a+b being examined, which fails with 'return' outside function.

Some fix will be needed in _PyPegen_register_stmts that sets last_stmt_location.

The other issue I mentioned is tracked in #156092.

bedevere-app · 2026-08-20T09:56:27Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

m-aciek

I found one regression, and left suggestion for the exception catch. It's nice it covers more cases, I hope it is possible to make it work with multiline still?

johnslavik · 2026-08-22T11:13:07Z

I was poking at this for a longer while and found numerous improvements we could do without adding token ranking.
Let's defer this PR until those improvements are considered first.

Rank tokens before probing for a keyword typo

b9c5baa

bedevere-app Bot added the awaiting review label Aug 20, 2026

johnslavik changed the title ~~Rank tokens before probing for a keyword typo~~ gh-156047: Rank tokens before probing for a keyword typo Aug 20, 2026

bedevere-app Bot mentioned this pull request Aug 20, 2026

Keyword typo suggestions differ between file and string tokenizers #156047

Open

johnslavik requested a review from pablogsal August 20, 2026 09:15

johnslavik commented Aug 20, 2026

View reviewed changes

Comment thread Lib/traceback.py

Check token string first, then do the line math

fdcc3a0

pablogsal requested changes Aug 20, 2026

View reviewed changes

bedevere-app Bot added awaiting changes and removed awaiting review labels Aug 20, 2026

m-aciek reviewed Aug 20, 2026

View reviewed changes

Comment thread Lib/traceback.py

Comment thread Lib/traceback.py

johnslavik commented Aug 20, 2026

View reviewed changes

Comment thread Lib/traceback.py

Uh oh!

Uh oh!

Conversation

johnslavik commented Aug 20, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

johnslavik commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

pablogsal Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

johnslavik Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablogsal Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

johnslavik Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnslavik Aug 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnslavik Aug 22, 2026

Choose a reason for hiding this comment

Uh oh!

bedevere-app Bot commented Aug 20, 2026

Uh oh!

m-aciek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johnslavik commented Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

johnslavik commented Aug 20, 2026 •

edited by bedevere-app Bot

Loading

johnslavik commented Aug 20, 2026 •

edited

Loading

johnslavik Aug 20, 2026 •

edited

Loading

johnslavik Aug 20, 2026 •

edited

Loading

johnslavik Aug 22, 2026 •

edited

Loading