feat: hybrid interop tests

[NoelStephensUnity]

Purpose of this PR

This PR adds 21 hybrid prefab test cases, fixes 9 issues the tests found (6 NGO issues in hybrid prefab mode, 2 NGO scene migration issues that affect every NGO project, and 1 test harness issue), and warns when a hybrid prefab instance uses NGO RPCs or NetworkVariables inside the N4E prediction loop, which is not supported. The results back the hybrid mode compatibility matrix (N4E prediction and unified remotes used with NGO RPCs, NetworkVariables and ownership).

PR Scope:

New tests (Tests/Runtime/Unified, UnifiedHost / UnifiedServer):

Combination	Result
Owner-predicted hybrid prefab	Predicted on the owning client.
NGO RPC sent, or NetworkVariable written, from PredictionUpdate	Not supported, gated on IsFirstTimeFullyPredictingTick or not. Logs a warning once per NetworkBehaviour (see below).
NGO RPC or NetworkVariable write made outside the prediction loop	No warning.
Unified remote to NGO RPC to unified remote, and NGO RPC to unified remote to NGO RPC	Both complete.
NGO ownership change	The ghost's owner follows, and the new owner predicts.

Reading a NetworkVariable in PredictionUpdate, including a tick-stamped value applied when the predicted tick reaches its stamp, is also not supported: NetworkVariables are not rolled back or tick aligned. Reads cannot be detected, so they do not warn.

CreateHybridPrefab takes an optional GhostMode so tests can create predicted and owner-predicted hybrid prefabs.

Prediction loop warning:

• RPCs and NetworkVariables are not part of N4E prediction, so a re-simulated tick sends or writes them again. A hybrid prefab instance now logs a warning, once per NetworkBehaviour, when it sends an RPC or writes a NetworkVariable while its ghost is in the N4E prediction loop (IsInPredictionLoop). It warns on clients, the server and the host. Non-hybrid instances only pay a bool check.

Fixes:

• UnifiedBootstrap re-registers the worlds it created for other NetworkManagers. Every ClientServerBootstrap constructor clears N4E's ServerWorlds and ClientWorlds, and each NetworkManager creates its own bootstrap. Remote methods send through those lists, so with several NetworkManagers in one process, server-to-client remotes were dropped.
• NetworkObjectBridge defaults a GhostObject to interpolation only when the bridge is first added (Reset). It no longer does it on every OnValidate, which reverted prediction set in the inspector.
• NGO spawn ownership and ownership changes set GhostObject.OwnerNetworkId. Before this, an owner-predicted ghost was never given an owner, so no client predicted it.
• NetworkManager shutdown disposes only its own world instead of calling World.DisposeAllWorlds(). This means one peer shutting down no longer breaks the others.
• A client that disconnects itself now receives its own ClientDisconnected event. UnifiedNetcodeTransport.DisconnectLocalClient only requested the N4E disconnect, which N4E reports after NGO's shutdown has stopped listening. It now notifies immediately, as UnityTransport does.
• A hybrid prefab instance that is part of a client's initial synchronization, but only spawns once its ghost arrives after the synchronization completed, is now moved into its server-side scene like the rest of the synchronized NetworkObjects. It stayed in the active scene.

NGO scene migration fixes (affect every NGO project, with CHANGELOG entries, and are also ported to develop-2.0.0):

• A scene migration (SceneEventType.ObjectSceneChanged) was sent to every connected client, including clients that did not observe the migrated NetworkObject and logged "Trying to synchronize NetworkObjectId but it was not spawned". It is now only sent to the clients that observe at least one migrated NetworkObject, and each client is only sent the NetworkObjects it observes. In distributed authority mode the owner applies the same filter; the CMB service is still sent every migrated NetworkObject, since it keeps the session state, and a migration the DAHost forwards is not filtered again.
• A NetworkObject shown with NetworkShow after it migrated into another scene, while hidden from that client, was instantiated in the client's active scene, since the client was never sent that migration. The client now moves the spawned NetworkObject into its server-side scene (its owner's scene in distributed authority mode), as it already does during its initial synchronization.

Test harness fix:

• The integration test hook that registers hybrid instances as pending ghosts also registered the server's own instance. On a dedicated server, which is never a connected client, that moved the instance into the DontDestroyOnLoad scene, and the server then sent every client a scene migration for an object none of them had spawned ("Trying to synchronize NetworkObjectId but it was not spawned", SceneEventData.cs:1302). The hook now skips the server, as the runtime does.

NetworkObjectDontDestroyWithOwnerTests (6 cases) and NetworkSpawnManagerTests (4 cases) now run in hybrid prefab mode. Both failed before the shutdown fix. DisconnectTests (2 cases) and PeerDisconnectCallbackTests (12 cases) also run in hybrid prefab mode. Their client-initiated cases failed before the disconnect fix. DisconnectTests is now only a disconnect test, with one fixture per host mode and a NetworkObject spawned while the clients disconnect. The owner persistence cases it duplicated are covered by NetworkObjectDontDestroyWithOwnerTests.

New NetworkObjectSceneMigrationObserverTests (4 tests, Host / DAHost / Server / UnifiedHost / UnifiedServer) cover both scene migration fixes and the late synchronization fix.

The hybrid tests are adapted to the N4E 7.1.0 API renames ([RPC(SendDirection)], PredictionUpdate(PredictionUpdateContext), GhostObject.CanWriteState).

Out of this PR's scope:

• PredictedPhysicsUpdate and prediction switching combined with NGO features (not covered yet). NGO RPCs and NetworkVariables from GatherInput are not supported, by decision; GatherInput runs outside the prediction loop, so it does not warn.
• Unified parenting (GhostObject.ParentReplication is in N4E 7.1.0; NGO will defer parenting to it in a separate PR).
• InstantiateAndSpawn and hybrid player prefabs not selecting the world (resolved with GhostObject.DelaySpawning / Spawn() in a separate PR, in progress).

Jira ticket

MTT-16222

Changelog

• Fixed: Issue where moving a NetworkObject into another scene made the clients that did not observe it log "Trying to synchronize NetworkObjectId but it was not spawned". The scene migration is now only sent to the clients that observe the NetworkObject.
• Fixed: Issue where a NetworkObject that was moved into another scene while hidden from a client spawned in that client's active scene when it was shown with NetworkShow, instead of the scene it is in on the server.

Documentation

• No documentation changes or additions were necessary.

Testing & QA (How your changes can be verified during release Playtest)

Headless PlayMode runs on 6000.7.0b1 with the unified job's scripting defines, filtered to the hybrid fixtures from #4172 plus the fixtures in this PR. Rows marked N4E 7.0.0 were run before the develop-3.x.x merge that moved this branch to N4E 7.1.0.

Run	Passed	Failed
N4E 7.1.0: NetworkObjectSceneMigrationObserverTests, UNIFIED_TESTS unset (Host / DAHost / Server)	12	0 (DAHost failed 3 of 4 before the distributed authority part of the fixes)
N4E 7.1.0: NetworkObjectSceneMigrationObserverTests, UNIFIED_TESTS=true	8	0
N4E 7.1.0: UNIFIED_TESTS unset, all of Unity.Netcode.RuntimeTests	5646	96 (the same NetworkVariableTests local-config cases as below)
N4E 7.1.0: testproject scene management tests, UNIFIED_TESTS unset	606	0
N4E 7.1.0: HybridInteropTests, UNIFIED_TESTS=true	12	0
N4E 7.1.0: HybridInteropTests, prediction loop check forced off	8	4 (both warning tests, as expected)
N4E 7.1.0: HybridInteropTests, IsInPredictionLoop condition removed	10	2 (the no-warning test, as expected)
N4E 7.0.0: UNIFIED_TESTS=true	286	0
N4E 7.0.0: UNIFIED_TESTS unset, all of Unity.Netcode.RuntimeTests	5644	96 (all in NetworkVariableTests, which fails the same 96 cases before these changes, at d4fda26e8. It only fails with the unified scripting defines and UNIFIED_TESTS unset, a combination CI does not run)
N4E 7.0.0: testproject scene management tests (NetworkSceneManager*, NetworkObjectSceneMigrationTests, DontDestroyOnLoad, scene event tests), UNIFIED_TESTS unset	602	0
N4E 7.0.0: NetworkObjectSceneMigrationObserverTests, UNIFIED_TESTS=true and unset	16	0
N4E 7.0.0: Changed fixtures, UNIFIED_TESTS=true, 3 runs	44 each	0
N4E 7.0.0: NetworkObjectDontDestroyWithOwnerTests, UNIFIED_TESTS=true, 3 runs	6 each	0
N4E 7.0.0: Disconnect fixtures, UNIFIED_TESTS=true, 3 runs	16 each	0
N4E 7.0.0: Disconnect fixtures, UNIFIED_TESTS unset	16	0

The DisconnectTests cull and the removal of HybridPredictionTests were not rerun locally; CI covers them.

Functional Testing

Manual testing :

• Manual testing done

Automated tests:

• Covered by existing automated tests
• Covered by new automated tests

Does the change require QA team to:

• Review automated tests?
• Execute manual tests?
• Provide feedback about the PR?

Up-port

Not needed. Hybrid prefab mode only exists on develop-3.x.x. The two NGO scene migration fixes are ported to develop-2.0.0 in #4185.

Backports

Not needed.

[u-pr]

⚠️ Some settings were ignored in '.pr_agent.toml (repository root)' repo settings

Check the configuration's TOML syntax, permitted settings, and size limit. See the error below.

---

Error message:
Ignored prohibited settings: github_app.handle_pr_actions. Permitted settings were applied.

Configuration content:

# This is the configuration for the u-pr bot
# https://github.cds.internal.unity3d.com/unity/u-pr
# For configuration of this file:
# https://developer.portal.internal.unity.com/catalog/default/component/u-pr/docs/configuration/automatic_runs/

[github_app]
handle_pr_actions = ['opened', 'ready_for_review']  # PR events that auto-run pr_commands
pr_commands       = ["/harness-review"]             # what runs on those events
reviewer_commands = ["/harness-review"]             # what runs when @u-pr is added as reviewer

[u-pr]

💡 Harness Review

The change adds hybrid prediction and interop coverage and keeps ghost ownership and world disposal scoped to each network manager, but the spawn callback order exposes an inconsistent owner state and two test assertions leave intended behavior unprotected.

_{Reviewed commit f6fee1d}

_{🤖 Helpful? 👍/👎}

[NoelStephensUnity]

@u-pr please review again

[u-pr]

The changes addressing the previous review comments look good:

• Setting the ghost owner immediately following SetupOnSpawn in SpawnNetworkObjectLocallyCommon ensures OnNetworkSpawn callbacks see the correct ghost ownership.
• Asserting InconsistentRawTicks > 0 directly verifies that re-simulated ticks receive updated values when NetworkVariable updates arrive without tick alignment.
• Limiting the ignored test in NetworkObjectDontDestroyWithOwnerTests to !m_UseHost preserves coverage for the UnifiedHost test fixture.

Summary of New Observations

1. Disposed World Cleanup in UnifiedBootstrap:
   When a NetworkManager shuts down, NetcodeWorld.Dispose() is called, but the world reference remains in s_CreatedWorlds, ServerWorlds, and ClientWorlds until a subsequent bootstrap runs. In scenarios where a client shuts down while the server remains running (or in multi-client environments), remote invocations iterating over ClientWorlds may encounter disposed worlds and throw an ObjectDisposedException. Exposing an unregister method on UnifiedBootstrap to remove the world upon shutdown would keep these static lists consistent and avoid keeping disposed worlds referenced in memory.

2. Null Safety in NetworkObjectBridge.UpdateGhostOwner:
   Adding a null check for networkManager.NetcodeWorld before accessing .LocalConnection prevents potential NullReferenceExceptions if UpdateGhostOwner is called while the netcode world is not initialized.

3. Re-simulation Timing in WriteNetworkVariableFromPrediction:
   When gateOnFirstTimeTick is false, the wait condition on line 385 only checks the total number of value changes. If 20 first-time predictions complete before a snapshot triggers re-simulation, OwnerWrittenTickDecreases may still be 0 when WriteNetworkVariableFromPrediction finishes. Waiting for at least one decrease when ungated (analogous to checking ResimulatedTicks > 0 in SendRpcsFromPrediction) prevents intermittent test flakiness.

_{🤖 Helpful? 👍/👎}

[NoelStephensUnity]

@u-pr please review again

[u-pr]

The latest changes in commit d4fda26 look solid and address the previous comments:

• Host NetcodeWorld null guard: Guarding networkManager.NetcodeWorld != null in NetworkObjectBridge.UpdateGhostOwner safely prevents potential NullReferenceExceptions if ghost ownership updates occur before or during host world initialization.
• Re-simulation decrease wait condition: Waiting for clientBehaviour.OwnerWrittenTickDecreases > 0 when ungated in WriteNetworkVariableFromPrediction ensures a re-simulation has moved the value backward before concluding the write phase, eliminating intermittent timing flakes.

Review Summary

• Ghost ownership propagation: Moving UpdateGhostOwner to SpawnNetworkObjectLocallyCommon right after SetupOnSpawn ensures OnNetworkSpawn callbacks see the correct ghost owner network ID, and updating it during ChangeOwnership keeps N4E ghost prediction aligned with NGO object ownership.
• Prefab configuration: Moving default interpolation configuration to Reset() preserves prediction settings configured in the inspector or in test helpers.
• Bootstrap world restoration: Re-registering existing worlds in UnifiedBootstrap ensures multi-NetworkManager environments maintain valid lists in ServerWorlds and ClientWorlds. (As a minor note for the planned client disconnect follow-up PR, unregistering disposed worlds from UnifiedBootstrap on shutdown can be tackled there to prevent stale references).
• Test coverage: The new tests in HybridPredictionTests and HybridInteropTests thoroughly validate ghost prediction, RPC/Remote interop, and tick-stamped NetworkVariable consistency across re-simulations.

Overall, the changes are clean, well-tested, and ready to go.

_{🤖 Helpful? 👍/👎}

[codecov-github-com]

Codecov Report

Attention: Patch coverage is 90.54054% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...cts/Runtime/SceneManagement/NetworkSceneManager.cs	88.88%	5 Missing ⚠️
...eobjects/Runtime/SceneManagement/SceneEventData.cs	90.47%	2 Missing ⚠️

@@                Coverage Diff                @@
##           develop-3.x.x    #4176      +/-   ##
=================================================
+ Coverage          78.19%   78.23%   +0.04%     
=================================================
  Files                153      153              
  Lines              26272    26328      +56     
=================================================
+ Hits               20544    20599      +55     
- Misses              5728     5729       +1     

Flag	Coverage Δ
NGOv2_project_testproject_ubuntu	76.70% <90.54%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...tcode.gameobjects/Runtime/Core/NetworkBehaviour.cs	86.23% <ø> (ø)
...netcode.gameobjects/Runtime/Core/NetworkManager.cs	79.97% <ø> (ø)
.../Runtime/Messaging/Messages/CreateObjectMessage.cs	87.19% <100.00%> (+0.65%)	⬆️
...cts/Runtime/NetworkVariable/NetworkVariableBase.cs	87.17% <ø> (ø)
...ameobjects/Runtime/Spawning/NetworkSpawnManager.cs	73.22% <ø> (ø)
...eobjects/Runtime/SceneManagement/SceneEventData.cs	77.99% <90.47%> (+0.50%)	⬆️
...cts/Runtime/SceneManagement/NetworkSceneManager.cs	79.83% <88.88%> (+0.33%)	⬆️

Components	Coverage Δ
com.unity.netcode.gameobjects	78.23% <90.54%> (+0.04%)	⬆️

ℹ️ Need help interpreting these results?

[NoelStephensUnity]

@u-pr please review again

[u-pr]

Review Summary

The latest commits (5e8e7e7, b60f7a1, 2897835, and 423c9e2) make substantial improvements to scene migration handling and transport disconnect signaling:

• Observer-filtered scene migrations: Filtering ObjectSceneChanged targets and payload objects so only observing clients receive scene migrations prevents clients from logging unspawned-object warnings.
• Scene synchronization for late/shown objects: Calling SynchronizeNetworkObjectScene in CreateObjectMessage (when objects are revealed via NetworkShow) and in GhostSpawnManager (when hybrid instances complete ghost spawning) ensures objects land in their server-side scenes even if they missed the initial migration event.
• Dedicated server test registration: Skipping pending ghost registration for servers in NetcodeIntegrationTest.RegisterPendingGhost resolves unwanted DontDestroyOnLoad migrations on dedicated servers.
• Client disconnect handling: Notifying NetworkEvent.Disconnect immediately in UnifiedNetcodeTransport.DisconnectLocalClient aligns behavior with UnityTransport and ensures disconnect callbacks fire before shutdown tears down listeners.
• Integration test coverage: The new tests in NetworkObjectSceneMigrationObserverTests thoroughly validate observer filtering and scene synchronization across various show/join scenarios.

---

Findings

1. Scene Management Guard in GhostSpawnManager (see inline comment):
   In GhostSpawnManager.ProcessGhostPendingSynchronization, m_NetworkManager.SceneManager.SynchronizeNetworkObjectScene(networkObject) is called without verifying m_NetworkManager.NetworkConfig.EnableSceneManagement. In NGO, SceneManager is null when scene management is disabled. Adding the guard (consistent with CreateObjectMessage.cs) prevents a potential NullReferenceException.

2. Connection ID Cleanup on Server Disconnect (see inline comment):
   In UnifiedNetcodeTransport.OnClientDisconnectFromServer, m_Connections.Remove(connectionEvent.Id.Value) is invoked to prevent DisconnectLocalClient from re-notifying upon shutdown. Because DisconnectLocalClient checks (int)ServerClientId, also removing (int)ServerClientId ensures wasConnected evaluates to false even if connectionEvent.Id.Value differs from ServerClientId.

_{🤖 Helpful? 👍/👎}

[NoelStephensUnity]

@u-pr please review again and look at my comments for your last two suggested fixes.

[u-pr]

Feedback on Previous Suggestions

Your explanations on both points are completely sound:

1. Scene Management Guard in GhostSpawnManager:
   m_NetworkManager.SceneManager is instantiated unconditionally during NetworkManager.Initialize and remains valid throughout the active lifecycle of the NetworkManager. When EnableSceneManagement is disabled, ServerSceneHandleToClientSceneHandle remains empty, so SynchronizeNetworkObjectScene safely performs an early no-op without throwing. Leaving this call unguarded is safe.

2. Connection ID Cleanup in UnifiedNetcodeTransport:
   On a client instance, UnifiedNetcodeTransport manages a single server connection where m_ServerClientId is assigned directly from connectionEvent.Id.Value. Because connectionEvent.Id.Value and (int)ServerClientId always match, removing connectionEvent.Id.Value in OnClientDisconnectFromServer clears the only entry in m_Connections. Consequently, wasConnected in DisconnectLocalClient reliably evaluates to false, preventing duplicate disconnect events.

---

PR Review Summary

All changes across the PR are in great shape:

• Ghost Ownership Synchronization: Calling UpdateGhostOwner immediately after SetupOnSpawn in SpawnNetworkObjectLocallyCommon ensures OnNetworkSpawn callbacks see the correct ghost owner. Synchronizing ghost ownership during ChangeOwnership keeps N4E ghost prediction aligned with NGO authority transfers.
• Scene Migration Filtering & Synchronization: Migrations are filtered so only observing clients receive ObjectSceneChanged events, preventing unspawned object warnings. Synchronizing scenes upon NetworkShow and in GhostSpawnManager ensures dynamically revealed and hybrid objects land in the correct server scenes.
• Unified Transport Disconnects & Lifecycle: Immediate disconnect notification in DisconnectLocalClient ensures callbacks fire before shutdown tears down listeners, and per-world cleanup in NetworkManager.ShutdownInternal avoids prematurely disposing other active worlds in multi-manager test setups.
• Test Coverage: The test suite additions (HybridPredictionTests, HybridInteropTests, NetworkObjectSceneMigrationObserverTests, and expanded fixtures in DisconnectTests and PeerDisconnectCallbackTests) thoroughly validate prediction updates, tick-stamped NetworkVariables, observer-scoped scene migrations, and disconnect callbacks.

_{🤖 Helpful? 👍/👎}

[EmandM]

We did a live code review. This looks great!