Skip to content

docs: demonstrate W_pos migration - #1572

Open
lntutor wants to merge 4 commits into
TransformerLensOrg:dev-4.xfrom
lntutor:docs/w-pos-migration-1560
Open

docs: demonstrate W_pos migration#1572
lntutor wants to merge 4 commits into
TransformerLensOrg:dev-4.xfrom
lntutor:docs/w-pos-migration-1560

Conversation

@lntutor

@lntutor lntutor commented Jul 30, 2026

Copy link
Copy Markdown

Description

Adds a runnable GPT-2 migration example for the HookedTransformer.W_pos and HookedTransformer.W_E_pos mappings. The example:

  • reads bridge.pos_embed.W_pos
  • constructs W_E_pos from bridge.W_E and W_pos
  • verifies the expected tensor shapes
  • demonstrates equality with the legacy properties under matching compatibility-mode weight processing

The accompanying explanation also points readers back to the existing numerical-matching guidance and mentions the raw/no-processing alternative.

Fixes #1560

Type of change

  • This change requires a documentation update

Verification

  • make format
  • Executed the documented GPT-2 snippet; both shape assertions and torch.testing.assert_close comparisons pass
  • uv run build-docs (build succeeded, 4 warnings; all four are pre-existing optional-import/duplicate-object warnings)
  • git diff --check

Checklist

  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Not applicable: code comments and new automated tests, because this is a documentation-only runnable example.

@lntutor

lntutor commented Jul 30, 2026

Copy link
Copy Markdown
Author

The full-coverage job's only failure is an unrelated flaky numerical-tolerance assertion in tests/unit/test_direct_path_patching.py after 6,597 tests passed. This PR only changes docs/source/content/migrating_to_v3.md. I don't have permission to rerun the workflow; could a maintainer please rerun the failed job?

@lntutor

lntutor commented Jul 31, 2026

Copy link
Copy Markdown
Author

The failing Full Code Coverage job appears unrelated to this docs-only change. It completed 6,597 tests and failed only tests/unit/test_direct_path_patching.py::TestCorrectness::test_correctness_against_actual_ln_forward on a numerical tolerance assertion.

I reproduced that exact test on the current PR head 086e4c1c with the CI Python version:

uv run --python 3.12 pytest tests/unit/test_direct_path_patching.py::TestCorrectness::test_correctness_against_actual_ln_forward -q
1 passed

It also passes under Python 3.13 locally. The PR changes only docs/source/content/migrating_to_v3.md; the failing module is untouched. The test's module-scoped random model is constructed without resetting its seed immediately before construction, so xdist scheduling can change the generated weights even though the session seed is fixed.

I do not have upstream workflow rerun permission. Could a maintainer rerun the failed Full Code Coverage job for this exact head?

@jlarson4

Copy link
Copy Markdown
Collaborator

@lntutor I am rerunning it now sorry just getting to your comments! Lots of contributions to go through this morning.

This specific CI test has been producing sporadic failures lately, it is a known issue that I have not gotten around to addressing. Thank you for the specific details about the failing test, I have noted this PR down in my notes for this problem and will come back here to reference it when I get around to fixing it!

@jlarson4

Copy link
Copy Markdown
Collaborator

Flaky test is still failing, so I am pushing a fix to dev-4.x. I will make sure to pul it in so this can pass CI as soon as its ready.

@jlarson4 jlarson4 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thanks @lntutor! Just one small comment

Comment thread docs/source/content/migrating_to_v3.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Proposal] Add a runnable W_pos / W_E_pos migration snippet to the migration guide

2 participants