Skip to content

Route topology-discovered host removals through the cluster lifecycle #1007

Description

@dkropachev

Parent epic: #921

Related to #922, #923, and #813.

Problem

Topology reconciliation removes host IDs that are absent from the latest system-table result with Metadata.remove_host_by_host_id(). This updates metadata directly and bypasses Cluster.on_remove().

The endpoint-collision path can remove a replacement through the full cluster lifecycle, but that detection depends on equality of the constructed EndPoint. Two supported endpoint types incorporate host identity into equality:

  • SniEndPoint includes the server name derived from host_id.
  • ClientRoutesEndPoint includes host_id directly.

When the advertised topology address is retained but the node UUID changes, the new row produces a different endpoint key. The old host therefore reaches the absent-ID cleanup instead of the lifecycle removal path.

Impact

The old host disappears from metadata while runtime state can remain active:

  • its Session pool remains open and indexed;
  • load-balancing policies retain the old host;
  • registered listeners miss the removal;
  • removal of a control host does not initiate reconnection;
  • metrics may continue reporting the stale pool.

Expected behavior

Every host ID removed by topology reconciliation should pass through the complete cluster removal lifecycle exactly once, regardless of the equality semantics of its connection endpoint.

Lifecycle removal performed during reconciliation must not start a recursive topology refresh.

Suggested direction

Introduce a reconciliation-aware removal path that:

  • removes the host from metadata;
  • invokes session, policy, listener, and control-connection cleanup;
  • suppresses redundant topology refresh when the caller is already reconciling;
  • preserves control-connection reconnection when applicable.

Use that path for the final absent-host cleanup instead of calling Metadata.remove_host_by_host_id() directly.

Acceptance criteria

  • Replacing a node behind SniEndPointFactory removes and shuts down the old pool.
  • The equivalent ClientRoutesEndPointFactory replacement behaves identically.
  • Policies and listeners receive exactly one removal.
  • The replacement receives exactly one addition.
  • Metadata contains no stale host or endpoint-index entries.
  • Cleanup does not recursively refresh topology.
  • Removing the active control host still initiates reconnection.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions