Skip to content

Fix blocking advisory lock in process_gateways_connections #3578

@r4victor

Description

@r4victor

Related to #3551

Currently dstack replicas take blocking advisory lock for processing gateway connections (check_or_restart() and try_collect_stats()):

async def _process_active_connections():
connections = await gateway_connections_pool.all()
# Two server processes on a single host cannot process
# gateway connections and init gateway connections concurrently:
# Race conditions cause conflicting tunnels being opened.
async with get_session_ctx() as session:
async with advisory_lock_ctx(
bind=session,
dialect_name=get_db().dialect_name,
resource="gateway_tunnels",
):
await asyncio.gather(*(_process_connection(conn) for conn in connections))

This means replicas can hold db connections open for some time potentially and that should be avoided as much as possible. A solution could be non-blocking pg_try_advisory_lock, but we need to ensure all the replicas can processes eventually.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions