nameparser 2.0 implementation#288
Draft
derek73 wants to merge 202 commits into
Draft
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v2 core design uses enum.StrEnum (3.11+). Full #257 (dropping typing_extensions, CI matrix) remains tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Validate that all replace() field values are str (user error if None) - Append missing-role synthetics in canonical Role order, not kwargs order - Unquote return type annotation (postponed annotations in effect) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Casefolded seven-component tuple in canonical Role order for dedup, dict keys, and sorting. Semantic layer comparison; __eq__ remains strict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds tests/v2/test_layering.py to mechanically enforce the conventions
doc's import layering and the public v2 export surface. Appends the v2
core types (Span, Role, Token, Ambiguity, AmbiguityKind, ParsedName,
Lexicon, Policy, PolicyPatch, PatronymicRule, UNSET, GIVEN_FIRST,
FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, Locale) to nameparser/__init__.py
alongside the existing v1 HumanName export.
Turns on component-flag mypy strictness for the four v2 modules and
check_untyped_defs for tests/v2, then fixes the resulting test-only
type mismatches by using the precise constructed types (Span(...),
frozenset({...}), tuple-of-pairs) where the literal type didn't matter
to the test, and adding narrow # type: ignore[arg-type] comments only
where a test deliberately exercises runtime coercion/validation of an
intentionally mismatched static type (e.g. Token span/Ambiguity kind
coercion tests, PatronymicRule string coercion, dict aliasing test).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pickle.dumps(Lexicon.default()) raised TypeError because the default slots-dataclass pickle path serializes the _cap_map MappingProxyType. Ship every other slot and rebuild the proxy from the canonical capitalization_exceptions tuple on load. Parser (Plan 3) is picklable by construction per the core spec, and a Parser holds a Lexicon. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
requires-python is >=3.11 on this branch, so the 3.10 job can no longer install the package (uv sync either fails or silently substitutes a managed 3.11). The rest of #257 (typing_extensions, classifiers) stays tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With requires-python at >=3.11, Self imports directly from typing and the conditional dependency can never activate; ruff (UP035/UP036) flags the dead version block once the floor is raised. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v1 suite is fully annotated (since #250 put tests/ under mypy), so pyproject carries no per-file ANN ignores; the new tests/v2 modules must be annotated too or 'ruff check' fails in CI. Also drops an unused import ruff flagged (F401). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…MITERS
Routing parens to maiden used to require editing both delimiter
buckets in tandem, with the nickname default's contents discovered by
digging: a pair in both buckets silently parsed as nickname. Now
Policy canonicalizes the effective nickname set to
nickname_delimiters - maiden_delimiters (listing a pair in the
specific bucket IS the intent), so
Policy(maiden_delimiters={('(', ')')}) is the whole recipe -- same
canonicalization precedent as the name_order coercion, equal values
converge. The nickname default is now the public, documented
DEFAULT_NICKNAME_DELIMITERS constant for explicit set math.
The 1.x facade keeps v1's nickname-wins precedence (pinned by the v1
suite) via a shim-side pre-subtraction on the maiden bucket, pinned
by test_snapshot_overlap_keeps_v1_nickname_precedence. Case row
nickname_bucket_wins_when_shared becomes
maiden_delimiters_win_when_shared. Docs: customize one-liner example,
migrate precedence note, release-log entry, reference autodata.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chancellor is already in the default TITLES set, so the doctest's add() was a no-op demonstrating nothing. dean is genuinely absent (the old v1 docs' example word) -- and absent for a teachable reason, which the ambiguous-entries paragraph now uses: 'Dean' doubles as a given name, so a default that swallowed it would misparse 'Dean Martin'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The task pages delegate detail to the reference ('see modules for the
full field list'), but Token/Ambiguity/Lexicon/Policy/Locale had no
class docstrings (autodoc rendered bare constructor signatures), Role
inherited Enum's 'Create a collection of name/value pairs', and no
dataclass field carried a #: doc. Every public class now opens with
what it is / how you get one / how it relates to its neighbors, every
field has a one-line #: doc that renders in the reference, and
parse()/Parser.parse()'s first sentences speak to users instead of
citing spec sections. Maintainer notes stay, demoted below the
user-facing text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
autodata on a re-export can't see the defining module's #: doc
comments, so every module-level constant rendered its TYPE's
docstring -- GIVEN_FIRST et al. showed tuple's 'Built-in immutable
sequence', DEFAULT_NICKNAME_DELIMITERS showed frozenset's constructor
help, UNSET rendered blank. Replace them with manual py:data
directives carrying real descriptions, under two new reference
sub-headings ('Name-order constants' -- stating that name_order
accepts ONLY these three, arbitrary Role tuples raise -- and
'Delimiter defaults'). The customize table's name_order type cell now
says 'one of the three exported order constants' instead of the
tuple[Role, Role, Role] annotation that invited hand-rolled tuples.
The #: comments stay in _policy.py for source readers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Removes ... from the input' contradicted the spans-into-original
invariant: parse('John 😀 Smith').original keeps the emoji; the
tokenizer just never includes those characters in any token, so they
appear in no field or rendered view. Say that.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified: assign consults name_order only for the NO_COMMA structure; 'Yamamoto, Haruki' takes family from the comma format with name_order never read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
autodoc's default alphabetical member sort scrambled the deliberate field ordering (strip_bidi rendered before strip_emoji, orphaning its 'the same way' reference; Lexicon's vocabulary grouping interleaved). autodoc_member_order = 'bysource' matches the codebase's declaration-order-is-canonical rule and the customize.rst table; the strip_bidi doc is also now self-contained regardless of ordering. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified live: with ' - ' configured, comma-split suffix groups parse unchanged. The comma is structural (the same reading that parses 'Family, Given' input), not a configurable suffix delimiter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Enum members without doc comments are skipped by autodoc, so the 'stable vocabulary of kinds' was invisible on the very page that promised it. Each member now carries a #: doc -- honestly marking ORDER and SUFFIX_OR_NICKNAME as reserved (no emitter yet; planned 2.x) and describing the emitter behavior of the three live kinds. PatronymicRule's two members get the same treatment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Lenient initial-shaped test' was internal jargon. The flag decides 'John Smith, V': suffix (the fifth) by default, or given-initial V when False; multi-letter suffixes are unaffected. Verified live both ways before wording. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Single letter, bare or period-written; of the default suffix words that means exactly the roman numerals I and V. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Yamamoto, Haruki' only demonstrates the point if you already know which word is the surname. 'Thomas, John' is ambiguous to an English-reading audience in exactly the way the comma resolves. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Append ellipses to the field docs' example parentheticals and teach the inspection idiom once in the class docstring (Lexicon.default().<field> is the full list). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each field doc now ends with a same-page link to the config data constant it sources from (the constants render their complete contents in the reference's compat section). The renamed pairs (suffix_words -> SUFFIX_NOT_ACRONYMS, bound_given_names -> BOUND_FIRST_NAMES) double as visible v1<->v2 mapping documentation, and particles_ambiguous -- which has no constant -- documents its derivation from NON_FIRST_NAME_PREFIXES instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same enum-member invisibility as AmbiguityKind: without doc comments autodoc skipped them, so the class docstring promised seven fields the page never listed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fold example (verified live): 'Gabriel García Márquez' -> family 'García Márquez' for multi-part-surname data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review feedback (Derek): 'How the parser thinks' implied nondeterminism -> 'How the parser works'; the 'four short essays' opener and 'pipeline, in one breath' phrasing replaced with direct statements; 'From string to name' now walks the canonical example concretely (8 tokens, Dr. span (0,3), family as a view joining de/la/Vega) and defines span before explaining what it prevents; the three-container bullets lead with the container names; the Dean paragraph returns to the v1 docs' problem-first framing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DEFAULT_NICKNAME_DELIMITERS grows by the eight conventions the issue
proposed: smart quotes, low-high (de/pl/cs/hu), right-right (sv/fi),
guillemets both directions (« Petit » inner spacing tolerated), CJK
corner brackets, fullwidth parentheses. Curly single quotes stay
excluded (U+2019 is the apostrophe in O'Connor, pinned). Both APIs
get them: the shim's sentinel map gains named keys
(smart_double_quotes, guillemets, ...) so the v1 keyed idioms
(pop/move/del) work on the new pairs like the original trio.
The conventions share characters in opposite roles ('"' closes the
German pair but opens the English one; '»' closes guillemets and
opens the reversed pair), which made extraction emit spurious
unbalanced-delimiter ambiguities: extract now tracks unmatched-open
offsets and drops candidates inside a region another pair
successfully masked. Genuine unbalanced opens still flag.
TDD throughout (extract suppression -> defaults -> shim sentinels,
each red first); 9 case rows through both runners; differential
harness exit 0 unchanged (no typographic quotes in the corpus);
release-log feat entry; reference/customize/migrate docs updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BOUND_FIRST_NAMES had only the transliterations; the script originals now join too: عبد (one entry covers abdul/abdel/abdal -- script attaches the article to the following word), the kunya pair أبو/ابو, and أم/ام. Behavior mirrors the Latin twins, probed live before pinning: bound join fires with 3+ tokens (عبد الرحمن محمد -> given 'عبد الرحمن'), the two-token kunya stays split (أبو مازن unchanged), and non-leading أبو still prefix-chains onto family (أحمد أبو خليل unchanged -- both prior pins hold). TDD: 4 case rows red in both runners first. Differential exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Twelve given-name titles -- Arabic honorifics precede the given name,
so all join الشيخ in FIRST_NAME_TITLES: the article forms (never
given names) الدكتور/الدكتورة/الأستاذ/الأستاذة/الحاج/الحاجة/الشيخة
and the bare doctor/professor/engineer forms
دكتور/دكتورة/أستاذ/أستاذة/مهندس. Deferred under the collision rule
(recorded in the data-module comment): bare سيد/شيخ/أمير/سلطان (all
common given names), the د. abbreviation (bare د would swallow
single-letter initials, the bare-κ trap), and the Ottoman
post-nominals باشا/بك/أفندي (survive as family names).
و ('and') joins conjunctions: formal script attaches it to the next
word, but the standalone informal spacing is common in real data;
single-character like y/и, so the #11 carve-out protects short names.
TDD: 20 rows red first, incl. the الحاج + عبد الرحمن compound
(title + bound join + family) and the 6-piece و title chain.
Differential exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation branch for the 2.0 design — see the RFC in #285 and the umbrella issue #284 for the design discussion and feedback questions. Implementation learnings that change the design will land as amendment commits on #285, per the plan described there.
Progress
Span,Role,Token,Ambiguity/AmbiguityKind,ParsedName;Lexicon,Policy/PolicyPatch/apply_patch,Locale. Frozen, slotted, hashable, picklable dataclasses with eager fail-loud validation; ~110 dedicated tests undertests/v2/.render()/initials()/capitalized()/__str__)Parser/parse()/parser_for()/matches()+ shared case tableHumanNamefacade +CONSTANTSshim (existing test corpus as regression harness) — full v1 suite (1,240+ tests) reconciled and passing against the facade; differential harness (tools/differential/) verifies 1.4-on-PyPI vs the facade over 486 corpus names with one classified diffru,tr_az) — shipped with the non-interference gate, Provide constants in non-Latin scripts (Cyrillic, Greek, Arabic, Hebrew) #269 default vocabulary, and CLI--localeNotes on what's here so far
TypeError, well-typed bad value →ValueError, enum lookups stayValueErrorper stdlib precedent.tests/v2/test_layering.py), and the seven-field canonical order derives fromRoledeclaration order everywhere — no ordering is ever stated twice.nameparser/configdata modules through 2.x, including the newmaiden_markers.py(Recognize maiden-name markers like "née" and "geb." (Jane Smith née Jones) #274 — marker words like "née"/"geb."; decisions recorded on the issue).🤖 Generated with Claude Code