Skip to content

Honor native bridge weight initialization config - #1577

Open
happykawayigt wants to merge 3 commits into
TransformerLensOrg:dev-4.xfrom
happykawayigt:fix/native-init-weights
Open

Honor native bridge weight initialization config#1577
happykawayigt wants to merge 3 commits into
TransformerLensOrg:dev-4.xfrom
happykawayigt:fix/native-init-weights

Conversation

@happykawayigt

Copy link
Copy Markdown

Description

Honors cfg.init_weights=False in TransformerBridge.boot_native() and adds a public TransformerBridge.init_weights() method for deterministic, in-place reinitialization of TL-native bridges.

Bridge construction replaces native submodules with generalized component wrappers, so the native initializer now unwraps those components before resetting the original parameters. Calling init_weights() on a non-native bridge raises an actionable error pointing to boot_native().

Fixes #1566

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Validation

  • pytest tests/unit/model_bridge/test_boot_native.py tests/unit/model_bridge/test_build_bridge_from_module.py tests/unit/model_bridge/test_native_features.py -q: 78 passed
  • mypy .: Success, no issues found in 421 source files
  • Full pytest tests/unit -m "not slow" -q: 5142 passed; the only two failures were existing UTF-8 test files read with the Windows GBK locale, and both pass with PYTHONUTF8=1
  • Black, isort, pycln, and git diff --check

Development note: AI-assisted implementation; I manually traced bridge component replacement, reviewed the patch, and ran the focused and repository-wide validation above.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

@happykawayigt
happykawayigt marked this pull request as ready for review July 31, 2026 06:09
@emerardd

Copy link
Copy Markdown

Thanks for working on this. I tested the re-init path locally and found two edge cases worth checking:

UnembeddingBridge adds a zero bias to the native bias-free head during bridge construction. The current test zeroes all parameters before re-init, so it cannot detect that this bias is not reinitialized. Filling parameters with a non-zero value such as 42 shows that head.bias remains stale unless initialize_native_model() explicitly zeros it.

With init_weights=False, the current seeded branch still enters fork_rng, but cfg.seed is never applied to NativeModel construction. As a result, different non-null seed values produce identical default parameters and only affect whether global RNG state is restored. The issue wording may imply that the seeded fork_rng should only be used when custom initialization runs.

I also exercised re-init across plain/RMS/no-norm, gated MLP, attention-only, rotary, separate-output-vocab, and all five initialization modes; those paths otherwise worked correctly.

@jlarson4 jlarson4 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for closing this out @happykawayigt! Great resolution. Just one small note:

HookedTransformer.init_weights() now has a bridge equivalent, but the migration guide never mentions init_weights. There is a table for API table equivalents in Migrating to v3 (migrating_to_v3.md:223). Could you add that row for cfg.init_weights / bridge.init_weights()?

@happykawayigt
happykawayigt force-pushed the fix/native-init-weights branch from 3aef3ab to 3089115 Compare July 31, 2026 21:55
@happykawayigt

Copy link
Copy Markdown
Author

Added in 3089115: the migration table now maps cfg.init_weights to bridge.init_weights() and notes that boot_native() honors the config flag during construction. The updated checks are green. Thanks for catching this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Proposal] Honor cfg.init_weights=False in boot_native and add bridge.init_weights()

3 participants