Honor native bridge weight initialization config - #1577
Conversation
|
Thanks for working on this. I tested the re-init path locally and found two edge cases worth checking:
With I also exercised re-init across plain/RMS/no-norm, gated MLP, attention-only, rotary, separate-output-vocab, and all five initialization modes; those paths otherwise worked correctly. |
jlarson4
left a comment
There was a problem hiding this comment.
Thanks for closing this out @happykawayigt! Great resolution. Just one small note:
HookedTransformer.init_weights() now has a bridge equivalent, but the migration guide never mentions init_weights. There is a table for API table equivalents in Migrating to v3 (migrating_to_v3.md:223). Could you add that row for cfg.init_weights / bridge.init_weights()?
3aef3ab to
3089115
Compare
|
Added in 3089115: the migration table now maps |
Description
Honors
cfg.init_weights=FalseinTransformerBridge.boot_native()and adds a publicTransformerBridge.init_weights()method for deterministic, in-place reinitialization of TL-native bridges.Bridge construction replaces native submodules with generalized component wrappers, so the native initializer now unwraps those components before resetting the original parameters. Calling
init_weights()on a non-native bridge raises an actionable error pointing toboot_native().Fixes #1566
Type of change
Validation
pytest tests/unit/model_bridge/test_boot_native.py tests/unit/model_bridge/test_build_bridge_from_module.py tests/unit/model_bridge/test_native_features.py -q: 78 passedmypy .: Success, no issues found in 421 source filespytest tests/unit -m "not slow" -q: 5142 passed; the only two failures were existing UTF-8 test files read with the Windows GBK locale, and both pass withPYTHONUTF8=1git diff --checkDevelopment note: AI-assisted implementation; I manually traced bridge component replacement, reviewed the patch, and ran the focused and repository-wide validation above.
Checklist