Skip to content

LWT prepared statement routing returns only replicas in PRESERVE_REPLICA_ORDER mode (3.x) #833

@dkropachev

Description

@dkropachev

Problem

LWT prepared statements can fail with "No node was available" when using PRESERVE_REPLICA_ORDER routing (the default for LWT).

The PreserveReplicaOrderIterator in TokenAwarePolicy returns only replicas in the query plan. When replica information is incomplete (e.g. prepared statements before parameter binding) or all replicas are down, the query plan can be empty or insufficient.

Root Cause

In TokenAwarePolicy.PreserveReplicaOrderIterator:

  • Pass 1: Returns LOCAL UP replicas
  • Pass 2: Returns REMOTE UP replicas
  • Pass 3: Falls back to child policy only if no replicas were returned at all

If even one replica was returned, non-replica nodes are never included. With RF=3 and 10 nodes, only 3 nodes appear in the query plan. If those replicas go down after plan construction, there's no fallback.

Expected Behavior

The query plan should always include all available nodes, with replicas prioritized:

  1. Local DC replicas (order preserved)
  2. Remote DC replicas (order preserved)
  3. Local DC non-replicas
  4. Remote DC non-replicas

Related

Port of #831 (4.x fix)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions