fix(backend): add instance-scoped Kafka consumer group and client IDs#273
fix(backend): add instance-scoped Kafka consumer group and client IDs#273
Conversation
Suffix Kafka consumer group IDs and client IDs with the SHIPSEC_INSTANCE number for agent-trace and node-io ingest services. This prevents consumer group conflicts when running multiple backend instances in multi-instance dev mode. Also updates pm2.config.cjs to pass the instance-scoped env vars and updates documentation to reflect the change. Signed-off-by: Aseem Shrey <LuD1161@users.noreply.github.com>
Signed-off-by: Aseem Shrey <LuD1161@users.noreply.github.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: be4ed7523f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| NODE_IO_KAFKA_CLIENT_ID: | ||
| process.env.NODE_IO_KAFKA_CLIENT_ID || `shipsec-backend-node-io-${instanceNum}`, | ||
| NODE_IO_KAFKA_GROUP_ID: | ||
| process.env.NODE_IO_KAFKA_GROUP_ID || `shipsec-node-io-ingestor-${instanceNum}`, |
There was a problem hiding this comment.
In the backend PM2 env block, this default always appends -${instanceNum} even when SHIPSEC_INSTANCE is not provided, because instanceNum falls back to '0' earlier in the file. That means PM2-launched backends without SHIPSEC_INSTANCE now get new group IDs (same pattern also exists for agent trace below), so the service-level “legacy fallback when unset” is bypassed and consumers can replay from the beginning (fromBeginning: true), duplicating ingest data for existing single-instance PM2 deployments.
Useful? React with 👍 / 👎.
1 participant
Summary
SHIPSEC_INSTANCEis unset to avoid offset resets/replaysAGENT_TRACE_KAFKA_*,NODE_IO_KAFKA_*) as highest-priority overridesWhy
This prevents cross-instance consumer collisions in multi-instance dev while avoiding breaking existing deployments that do not set
SHIPSEC_INSTANCE.Validation
bun --cwd=backend run lint