Control token swapping fail for transformers 5.9.0 for user message activations

[antonpibm]

TL;DR

The aLoRA user-message injection in configure_chat_template (Pass 2) drops the
first character of the invocation text and relies on that being equivalent to
dropping the first token. For the <context> invocation on the
Granite 4.1 tokenizer this equivalence holds under transformers 5.6.2 but
breaks under transformers 5.9.0 — the same tokenizer files tokenize
<context> differently because the two library versions instantiate a different
tokenizer class with a different pre-tokenizer. This is a Granite-4.1-only issue;
Granite 4.2 is unaffected (see the separate 4.2 note).

The invariant the code relies on

Pass 2 injects the control token and then re-emits invocation_text[1:] (the
invocation text minus its first character). At inference, token-exchange swaps
the control token's embedding to the adapter's first invocation token. For
the on-wire sequence to reconstruct the original invocation, dropping the first
character of the string must equal dropping the first token of its
tokenization — which is only true when the leading < tokenizes as its own,
standalone token.

• Holds: <context> → ['<', 'context', '>'] = [27, 2196, 29].
  Drop char 0 → context> → [2196, 29]. [27] + [2196, 29] == original. ✅
• Broken: <context> → ['<context', '>'] = [35628, 29].
  Drop char 0 → context> → [2196, 29]. [35628] + [2196, 29] =
  [35628, 2196, 29] = <context + context + > — a garbled duplicate,
  not [35628, 29]. ❌

Observed behavior across transformers versions

Test: tests/composer/test_chat_template.py::TestInvocationFirstCharDropProperty::test_first_char_drop_equals_first_token_drop
(a pure tokenizer-property test on ibm-granite/granite-4.1-3b, run with
-v -s --tb=short).

transformers 5.6.2 — PASSED

tests/composer/test_chat_template.py::TestInvocationFirstCharDropProperty::test_first_char_drop_equals_first_token_drop PASSED
1 passed in 2.22s

<context> → [27, 2196, 29] (leading < is a lone token) → invariant holds.

transformers 5.9.0 — FAILED

tests/composer/test_chat_template.py::TestInvocationFirstCharDropProperty::test_first_char_drop_equals_first_token_drop FAILED

E   AssertionError: invocation '<context>': dropping first char of the string
E   produced tokens [2196, 29] but the tail of the full tokenization is [29]
E   assert [29] == [2196, 29]
E     At index 0 diff: 29 != 2196
1 failed in 0.71s

<context> → [35628, 29] (<context is a single merged token) → invariant
broken.

Both runs use the same ibm-granite/granite-4.1-3b files (same Hub snapshot
c0650403…, same tokenizer.json, same vocab, same tokenizers 0.22.2). Only
the transformers version differs.

Root cause: why the same files tokenize differently

tokenizer_config.json declares "tokenizer_class": "GPT2Tokenizer", but
AutoTokenizer.from_pretrained resolves that to a different concrete class
between the two versions, and those classes apply different pre-tokenizers:

	transformers 5.6.2	transformers 5.9.0
Loaded class (MRO)	GPT2Tokenizer → TokenizersBackend → PreTrainedTokenizerBase	TokenizersBackend → PreTrainedTokenizerBase
isinstance(tok, GPT2Tokenizer)	True	False
Effective pre-tokenizer	ByteLevel — the GPT2Tokenizer subclass forces GPT2's canonical ByteLevel, overriding the file	the file's Sequence[Split(regex)], used verbatim
<context>	[27, 2196, 29] (< lone)	[35628, 29] (<context merged)

• The 4.1 tokenizer.json on disk has a Sequence[Split(regex)] pre-tokenizer.
  Its regex groups an optional leading non-letter with the following letters
  ([^\r\n\p{L}\p{N}]?\p{L}+), which allows <context to merge into one token.
• 5.6.2 honors tokenizer_class: GPT2Tokenizer and instantiates that
  subclass, which overrides the file's pre-tokenizer with ByteLevel.
  ByteLevel splits the leading < off, so the merge never happens and the bug is
  masked.
• 5.9.0 instantiates the generic TokenizersBackend directly (the
  GPT2Tokenizer subclass is not in the MRO), which uses the tokenizer.json
  pre-tokenizer as written — the Sequence[Split] — so <context merges and
  the bug surfaces.

Note: the transformers version does not change the vocab or merges. It
changes which tokenizer class is instantiated, and therefore whether the
file's Split pre-tokenizer is respected or replaced by ByteLevel.

The change in AutoTokenizer class resolution between 5.6.2 and 5.9.0 is a
transformers-internal change; the observable facts above (MRO, isinstance,
resulting token ids from identical files) are directly verified, but the exact
upstream commit/PR responsible was not traced.

Fix suggestion

Make the injection independent of the pre-tokenizer by dropping the first
token, not the first character.

Pass 2 currently does (conceptually):

{%- set content = _parts[0] + ns.adapter_token + ns.adapter_invocation_text[1:] + _parts[1] %}

The [1:] (first-character drop) is the fragile step. Options, preferred first:

1. Drop the first token at compose time, not the first char in Jinja.
   In configure_chat_template, precompute the invocation text with its first
   token removed — tokenize invocation_text, drop ids[0], decode the tail —
   and pass that decoded tail into the template instead of relying on [1:].
   This makes the emitted tail exactly tokenize(invocation)[1:] by
   construction, regardless of how the leading char pre-tokenizes.

   ids = tokenizer(invocation_text, add_special_tokens=False).input_ids
   invocation_tail = tokenizer.decode(ids[1:])   # tail after the first *token*
   # emit: adapter_token + invocation_tail   (no string [1:] slice in Jinja)

   The runtime swap already substitutes the control token's embedding with
   ids[0] (the first invocation token), so adapter_token + tail reconstructs
   the full invocation token sequence for any pre-tokenizer.

2. Guard + fail loudly. If keeping the char-drop, assert at compose time that
   tokenize(invocation)[0] decodes to a single character equal to
   invocation_text[0], and raise a clear error otherwise, so a mis-tokenizing
   base is caught at compose rather than silently corrupting inference.

Option 1 is the real fix; option 2 is a minimum safeguard.

Scope / who is affected

• Affected: Granite 4.1 (and any base whose tokenizer.json uses a
  Split-style pre-tokenizer that merges a leading < with following letters)
  when loaded under transformers 5.9+, for user-message aLoRA invocations
  whose first character does not tokenize standalone (e.g. <context>).
  <requirements>, <certainty>, <guardian> keep < lone and are unaffected.
• Not affected: Granite 4.2 (ByteLevel pre-tokenizer, < always lone);
  Granite 4.1 under transformers 5.6.2; adapters that activate at the
  assistant boundary (their first invocation token is a special token, e.g.
  <|im_start|>, not a <-prefixed word).

Reproduction

pytest "tests/composer/test_chat_template.py::TestInvocationFirstCharDropProperty::test_first_char_drop_equals_first_token_drop" -v -s --tb=short

pytest "tests/composer/test_chat_template.py::TestInvocationFirstCharDropProperty::test_first_token_is_single_character" -v -s --tb=short

Run under transformers 5.9.0 (fails) vs 5.6.2 (passes). An end-to-end
reproduction through configure_chat_template is in the same class:
test_context_invocation_injection_not_corrupted_granite_4_1 (marked xfail; it
skips where the leading < stays a lone token and xfails where it merges).