Skip to content

Conversation

@yashmehrotra
Copy link
Member

@yashmehrotra yashmehrotra commented Jan 29, 2026

Fixes: #468

Summary by CodeRabbit

Documentation

  • Added a comprehensive guide on distributed canaries: agentSelector patterns, multi-agent execution flow, synchronization/cleanup behavior, and how results are aggregated and displayed.
  • Added a detailed blog post with step‑by‑step tutorials, pattern matching examples, YAML use cases (regional, exclusion, HTTP checks), real‑world scenarios, and operational tips for multi-cluster monitoring.

✏️ Tip: You can customize this high-level summary in your review settings.

@yashmehrotra yashmehrotra requested a review from moshloop January 29, 2026 12:19
@vercel
Copy link

vercel bot commented Jan 29, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
docs Ready Ready Preview Jan 29, 2026 1:00pm

Request Review

@netlify
Copy link

netlify bot commented Jan 29, 2026

Deploy Preview for canarychecker ready!

Name Link
🔨 Latest commit 165ca77
🔍 Latest deploy log https://app.netlify.com/projects/canarychecker/deploys/697b59c198363f0008224d12
😎 Deploy Preview https://deploy-preview-469--canarychecker.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai
Copy link

coderabbitai bot commented Jan 29, 2026

Walkthrough

Two new documentation files were added: a conceptual guide for distributed canaries in the canary-checker docs and a comprehensive blog post for mission-control, covering agentSelector patterns, YAML examples, execution/sync behavior, and multi-agent monitoring use cases.

Changes

Cohort / File(s) Summary
Canary-checker concept doc
canary-checker/docs/concepts/distributed-canaries.md
New conceptual documentation describing distributed canaries: how they work, agentSelector pattern table, per-agent execution flow, synchronization/cleanup, results aggregation, and multiple YAML examples (HTTP checks, regional monitoring, exclusion patterns).
Mission-control blog post
mission-control/blog/distributed-canaries/index.mdx
New detailed blog post explaining monitoring across multiple clusters with a single canary definition: problem statement, solution approach, derived-canary lifecycle (creation, sync, cleanup), agent registration, verification steps, pattern examples, use cases, tips and gotchas.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title 'chore: add distributed canaries docs + blog' accurately summarizes the main change: adding documentation and a blog post for distributed canaries.
Linked Issues check ✅ Passed The PR fulfills issue #468 by adding both concept documentation (distributed-canaries.md) and a comprehensive blog post (index.mdx) explaining distributed canaries.
Out of Scope Changes check ✅ Passed All changes are directly scoped to the stated objectives: documentation and blog post files for distributed canaries with no unrelated modifications.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch distributed-canaries

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Fix all issues with AI agents
In `@canary-checker/docs/concepts/distributed-canaries.md`:
- Around line 12-19: Rewrite the four numbered "How It Works" bullets to use
active voice: change "The canary does not run locally on the server" to
"Specifying an agentSelector prevents the canary from running locally on the
server" (or similar active phrasing), change "A copy of the canary is created
for each matched agent" to "The system creates a copy of the canary for each
matched agent," change "Each agent runs the check independently and reports
results back" to "Each matched agent runs the check independently and reports
the results," and change "The copies are kept in sync with the parent canary" to
"The system keeps the derived copies in sync with the parent canary"; also
rewrite the next sentence about syncing to active voice (e.g., "A background job
syncs agent selector canaries every 5 minutes" is already active—ensure
consistency).
- Around line 55-60: Change the passive sentence "The check is executed locally
only when `local` agent is provided in selector" to active voice; locate the
step that begins with that phrase and rewrite it to something like "Execute the
check locally only when the `local` agent is specified in the selector" (or "The
agent executes the check locally only when the `local` agent is specified in the
selector") so the sentence is active while preserving meaning and the
`local`/selector tokens.
- Around line 25-31: The Markdown table header and rows (starting with the
header line "| Pattern | Description |" and the pattern rows like "`agent-1`",
"`eu-west-*`", "`*-prod`", "`!staging`", and "`team-*`, `!team-b`") violate
MD060 compact table spacing; fix by ensuring a single space exists on both sides
of every pipe delimiter in the header and each data row (e.g., "| Pattern |
Description |" and "| `agent-1` | Exact match |") so the table uses consistent
spacing and passes markdownlint.

In `@mission-control/blog/distributed-canaries/index.mdx`:
- Around line 64-74: The sentence "The copies are kept in sync automatically."
in the document's "How It Actually Works" section uses passive voice; change it
to active voice by rephrasing to something like "The system keeps the copies in
sync automatically," or "We keep the copies in sync automatically," and apply
the edit where that exact sentence appears to comply with the style guide.
- Around line 289-293: Update the "Agent Installation Guide" link so it removes
the duplicated "docs" segment in its URL; locate the markdown link text "Agent
Installation Guide" in the References section and change the href from
/docs/docs/installation/saas/agent to /docs/installation/saas/agent so it
matches the other links (e.g., the format used at line with the "Canary Spec
Reference").
🧹 Nitpick comments (1)
mission-control/blog/distributed-canaries/index.mdx (1)

1-8: Reminder: run doc formatting and linting.

Please run make fmt and make lint before committing so Markdown formatting and lint rules stay consistent.

As per coding guidelines, run make fmt and make lint before committing changes.

Comment on lines +12 to +19
When you specify an `agentSelector` on a canary:

1. The canary does **not** run locally on the server
2. A copy of the canary is created for each matched agent
3. Each agent runs the check independently and reports results back
4. The copies are kept in sync with the parent canary

A background job syncs agent selector canaries every 5 minutes. When agents are added or removed, the derived canaries are automatically created or cleaned up.
Copy link

@coderabbitai coderabbitai bot Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Rewrite passive voice in the “How It Works” steps.

Lines 14–17 use passive voice. Please switch to active voice to match the documentation style guide.

Suggested rewrite
-1. The canary does **not** run locally on the server
-2. A copy of the canary is created for each matched agent
-3. Each agent runs the check independently and reports results back
-4. The copies are kept in sync with the parent canary
+1. The server does **not** run the canary locally.
+2. The system creates a copy of the canary for each matched agent.
+3. Each agent runs the check independently and reports results back.
+4. The system keeps the copies in sync with the parent canary.

As per coding guidelines, use active voice; never use passive voice.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
When you specify an `agentSelector` on a canary:
1. The canary does **not** run locally on the server
2. A copy of the canary is created for each matched agent
3. Each agent runs the check independently and reports results back
4. The copies are kept in sync with the parent canary
A background job syncs agent selector canaries every 5 minutes. When agents are added or removed, the derived canaries are automatically created or cleaned up.
When you specify an `agentSelector` on a canary:
1. The server does **not** run the canary locally.
2. The system creates a copy of the canary for each matched agent.
3. Each agent runs the check independently and reports results back.
4. The system keeps the copies in sync with the parent canary.
A background job syncs agent selector canaries every 5 minutes. When agents are added or removed, the derived canaries are automatically created or cleaned up.
🤖 Prompt for AI Agents
In `@canary-checker/docs/concepts/distributed-canaries.md` around lines 12 - 19,
Rewrite the four numbered "How It Works" bullets to use active voice: change
"The canary does not run locally on the server" to "Specifying an agentSelector
prevents the canary from running locally on the server" (or similar active
phrasing), change "A copy of the canary is created for each matched agent" to
"The system creates a copy of the canary for each matched agent," change "Each
agent runs the check independently and reports results back" to "Each matched
agent runs the check independently and reports the results," and change "The
copies are kept in sync with the parent canary" to "The system keeps the derived
copies in sync with the parent canary"; also rewrite the next sentence about
syncing to active voice (e.g., "A background job syncs agent selector canaries
every 5 minutes" is already active—ensure consistency).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use Mission Control, not server and system

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moshloop, thank you for the clarification! I'll update the suggestion to use "Mission Control" consistently.

Revised suggestion:

When you specify an `agentSelector` on a canary:

1. Mission Control does **not** run the canary locally.
2. Mission Control creates a copy of the canary for each matched agent.
3. Each agent runs the check independently and reports results back.
4. Mission Control keeps the copies in sync with the parent canary.

A background job syncs agent selector canaries every 5 minutes. When agents are added or removed, the derived canaries are automatically created or cleaned up.

✏️ Learnings added
Learnt from: moshloop
Repo: flanksource/docs PR: 469
File: canary-checker/docs/concepts/distributed-canaries.md:12-19
Timestamp: 2026-01-29T13:01:15.936Z
Learning: In flanksource/docs documentation, use "Mission Control" to refer to the central server/system component, not generic terms like "server" or "system".

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


🧠 Learnings used
Learnt from: CR
Repo: flanksource/docs PR: 0
File: .cursor/rules/global.mdc:0-0
Timestamp: 2025-12-05T13:02:08.910Z
Learning: Applies to **/*.{md,markdown,txt} : Use active voice; never use passive voice

Comment on lines +55 to +60
When this canary is created:

1. The check is executed locally only when `local` agent is provided in selector
2. A derived canary is created for each registered agent
3. Each agent executes the HTTP check against `api-service.default.svc.cluster.local:8080/health` in its own cluster
4. Results from all agents are aggregated and visible in the UI
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Use active voice in the execution steps.

Line 57 uses passive voice. Please switch to active voice.

Suggested rewrite
-1. The check is executed locally only when `local` agent is provided in selector
+1. The system executes the check locally only when you include the `local` agent in the selector.

As per coding guidelines, use active voice; never use passive voice.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
When this canary is created:
1. The check is executed locally only when `local` agent is provided in selector
2. A derived canary is created for each registered agent
3. Each agent executes the HTTP check against `api-service.default.svc.cluster.local:8080/health` in its own cluster
4. Results from all agents are aggregated and visible in the UI
When this canary is created:
1. The system executes the check locally only when you include the `local` agent in the selector.
2. A derived canary is created for each registered agent
3. Each agent executes the HTTP check against `api-service.default.svc.cluster.local:8080/health` in its own cluster
4. Results from all agents are aggregated and visible in the UI
🤖 Prompt for AI Agents
In `@canary-checker/docs/concepts/distributed-canaries.md` around lines 55 - 60,
Change the passive sentence "The check is executed locally only when `local`
agent is provided in selector" to active voice; locate the step that begins with
that phrase and rewrite it to something like "Execute the check locally only
when the `local` agent is specified in the selector" (or "The agent executes the
check locally only when the `local` agent is specified in the selector") so the
sentence is active while preserving meaning and the `local`/selector tokens.

Comment on lines +64 to +74
## How It Actually Works

When you create a canary with an `agentSelector`, something interesting happens: the canary doesn't run on the central server at all. Instead, the system:

1. Looks at all registered agents
2. Matches agent names against your selector patterns
3. Creates a copy of the canary for each matched agent
4. Each agent runs the check independently and reports results back

The copies are kept in sync automatically. If you update the parent canary, all the derived canaries update too. If you add a new agent that matches the pattern, it gets the canary within a few minutes. If you remove an agent, its canary is cleaned up.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Use active voice in the sync description.

Line 73 uses passive voice, which conflicts with the style guide. Please rewrite in active voice.

Suggested rewrite
- The copies are kept in sync automatically. If you update the parent canary, all the derived canaries update too. If you add a new agent that matches the pattern, it gets the canary within a few minutes. If you remove an agent, its canary is cleaned up.
+ The system keeps the copies in sync automatically. If you update the parent canary, all the derived canaries update too. If you add a new agent that matches the pattern, the system assigns the canary within a few minutes. If you remove an agent, the system cleans up its canary.

As per coding guidelines, use active voice; never use passive voice.

🧰 Tools
🪛 LanguageTool

[style] ~73-~73: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...t gets the canary within a few minutes. If you remove an agent, its canary is clea...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

🤖 Prompt for AI Agents
In `@mission-control/blog/distributed-canaries/index.mdx` around lines 64 - 74,
The sentence "The copies are kept in sync automatically." in the document's "How
It Actually Works" section uses passive voice; change it to active voice by
rephrasing to something like "The system keeps the copies in sync
automatically," or "We keep the copies in sync automatically," and apply the
edit where that exact sentence appears to comply with the style guide.

Comment on lines +289 to +293
## References

- [Distributed Canaries Concept](/docs/guide/canary-checker/concepts/distributed-canaries)
- [Canary Spec Reference](/docs/guide/canary-checker/reference/canary-spec)
- [Agent Installation Guide](/docs/docs/installation/saas/agent)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# First, let's locate the file and verify its content
find . -name "index.mdx" -path "*/distributed-canaries/*" | head -20

Repository: flanksource/docs

Length of output: 113


🏁 Script executed:

# If the above doesn't work, let's search more broadly
fd "index.mdx" | grep -i canary

Repository: flanksource/docs

Length of output: 224


🏁 Script executed:

# Also search for agent installation documentation to understand the correct path structure
fd -i "agent" -type f | grep -i install | head -30

Repository: flanksource/docs

Length of output: 230


🏁 Script executed:

# Let's check the overall docs structure
ls -la docs/ 2>/dev/null | head -20

Repository: flanksource/docs

Length of output: 745


🏁 Script executed:

# Search for any paths containing "installation" and "saas"
rg "installation/saas" --type md --type mdx -l

Repository: flanksource/docs

Length of output: 87


🏁 Script executed:

#!/bin/bash
# Search for agent-related docs
find . -type f \( -name "*.md" -o -name "*.mdx" \) | xargs grep -l "agent.*installation\|installation.*agent" | head -20

Repository: flanksource/docs

Length of output: 232


🏁 Script executed:

# Look for installation docs structure
find . -path "*installation*" -type d

Repository: flanksource/docs

Length of output: 265


🏁 Script executed:

# Check if there's a docs folder at the root level and its structure
find . -maxdepth 3 -name "*installation*" -o -name "*agent*" | grep -E "\.(md|mdx)$" | head -30

Repository: flanksource/docs

Length of output: 42


🏁 Script executed:

# Look at the actual file to see the links and understand the pattern
cat -n ./mission-control/blog/distributed-canaries/index.mdx | tail -20

Repository: flanksource/docs

Length of output: 1431


🏁 Script executed:

#!/bin/bash
# List all files in the saas installation directory
ls -la mission-control/docs/installation/saas/

Repository: flanksource/docs

Length of output: 474


🏁 Script executed:

# Also check the installation directory structure more broadly
find ./mission-control/docs/installation -type f | sort

Repository: flanksource/docs

Length of output: 1873


🏁 Script executed:

# Check the getting-started file to understand the structure
head -50 ./mission-control/docs/installation/saas/getting-started.mdx

Repository: flanksource/docs

Length of output: 1843


🏁 Script executed:

# Search for references to agent installation links in other docs
rg "installation.*agent|agent.*installation" --type-add 'mdx:*.mdx' -t mdx | head -20

Repository: flanksource/docs

Length of output: 893


Remove the duplicate "docs" segment in the agent installation link.

Line 293 uses /docs/docs/installation/saas/agent, which contains a duplicated "docs" segment. The correct path is /docs/installation/saas/agent, which matches the format used elsewhere in this file (line 199) and in other documentation files.

🤖 Prompt for AI Agents
In `@mission-control/blog/distributed-canaries/index.mdx` around lines 289 - 293,
Update the "Agent Installation Guide" link so it removes the duplicated "docs"
segment in its URL; locate the markdown link text "Agent Installation Guide" in
the References section and change the href from
/docs/docs/installation/saas/agent to /docs/installation/saas/agent so it
matches the other links (e.g., the format used at line with the "Canary Spec
Reference").

icon: network
sidebar_position: 6
---

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add that this will only work for mission control

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add concept for distributed checks and a blog post as well

3 participants