Add database collection to track semantic domain sense counts per project

[Copilot]

Implementation Complete

• Create new database model ProjectSemanticDomainCount with projectId, domainId, and count fields
• Implement SemanticDomainCountRepository for database operations
• Add unique compound index on (ProjectId, DomainId) for query optimization and uniqueness enforcement
• Implement SemanticDomainCountService to handle count updates
• Update WordService to update counts after words are added/updated/removed
• Register new services in Startup.cs for dependency injection
• Create unit tests for the new repository and service
• Update existing tests to include new dependency
• Update StatisticsService to use the new count collection for better performance
• [Outside this pr] Create one-shot mongosh script to populate this new db collection from the current frontier.

---

Summary by CodeRabbit

Release Notes

• New Features

  • Added domain count statistics query capability via new endpoint for retrieving semantic domain usage metrics per domain.
  • Added unique compound index on (ProjectId, DomainId) for improved query performance and data integrity.

• Refactor

  • Improved statistics architecture by implementing dedicated domain counting infrastructure for more efficient queries.
  • Streamlined domain count retrieval in semantic domain tree views.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

---

This change is 

Summary by CodeRabbit

• New Features

  • Semantic domain counting system to track word senses per domain efficiently.
  • New frontier word clearing and deletion operations for improved word management.

• Refactor

  • Restructured service layer dependencies for better separation of concerns.
  • Optimized semantic domain statistics retrieval using pre-aggregated counts.

• Tests

  • Added comprehensive test coverage for semantic domain counting operations.

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

• ✅ Full review completed - (🔄 Check again to review again)

📝 Walkthrough

Walkthrough

Adds a semantic-domain count system (model, repo, service), rewires services/controllers to use it, removes legacy frontier-count repo methods, updates DI and frontend API, and adjusts tests and mocks to the new counting approach.

Changes

Cohort / File(s)	Summary
Semantic Domain Count Infra Backend/Models/ProjectSemanticDomainCount.cs, Backend/Interfaces/ISemanticDomainCountRepository.cs, Backend/Interfaces/ISemanticDomainCountService.cs, Backend/Repositories/SemanticDomainCountRepository.cs, Backend/Services/SemanticDomainCountService.cs	New model, repository interface and implementation, and service to store and atomically update per-project semantic-domain counts.
WordService & IWordService Backend/Services/WordService.cs, Backend/Interfaces/IWordService.cs	Constructor now requires semantic-domain count service; added bulk/clear/make-deleted frontier APIs; changed DeleteFrontierWord signature/return; integrated domain-count updates across create/update/delete/restore flows.
StatisticsService & API Backend/Services/StatisticsService.cs, Backend/Interfaces/IStatisticsService.cs, src/api/api/statistics-api.ts, src/backend/index.ts	StatisticsService now uses sem-domain count repo and exposes GetDomainCount; frontend/client API gains getDomainCount and frontend facade uses it.
Controllers Backend/Controllers/StatisticsController.cs, Backend/Controllers/WordController.cs, Backend/Controllers/LiftController.cs	StatisticsController constructor changed and new GetDomainCount endpoint added; WordController removed GetDomainWordCount and uses MakeFrontierDeleted; LiftController now injects IWordService and uses ClearFrontier.
Services using WordService Backend/Services/LiftService.cs, Backend/Services/MergeService.cs, Backend/Interfaces/ILiftService.cs	LiftService/ILiftService and MergeService updated to accept/use IWordService (instead of IWordRepository) and to call IWordService deletion APIs (bulk and renamed methods).
Repository/API removals Backend/Repositories/WordRepository.cs, Backend/Interfaces/IWordRepository.cs, src/api/api/word-api.ts	Removed CountFrontierWordsWithDomain and DeleteFrontierWords from repository/interface and removed getDomainWordCount from frontend API surface.
DI registration Backend/Startup.cs	Registered ISemanticDomainCountRepository and ISemanticDomainCountService in DI container.
Tests & Mocks Backend.Tests/..., Backend.Tests/Mocks/SemanticDomainCountRepositoryMock.cs, Backend.Tests/Mocks/StatisticsServiceMock.cs, Backend.Tests/Mocks/WordRepositoryMock.cs	Added SemanticDomainCountRepositoryMock and service tests; updated test setups to instantiate sem-domain mocks/services and removed legacy WordRepositoryMock methods; updated controller and statistics tests.
Frontend component updates src/components/TreeView/TreeDepiction/DomainCountBadge.tsx, tests	Component and tests switched from getDomainWordCount → getDomainCount and updated corresponding mocks.

Sequence Diagram(s)

sequenceDiagram
    actor User
    participant WordController as "WordController"
    participant WordService as "WordService"
    participant SDCountService as "SemanticDomainCountService"
    participant SDCountRepo as "SemanticDomainCountRepository"

    User->>WordController: POST/PUT/DELETE word
    WordController->>WordService: create/update/delete request
    WordService->>WordService: persist or mutate word(s)
    alt domain changes present
        WordService->>SDCountService: UpdateCountsForWord/AfterUpdate/ForDeletion
        SDCountService->>SDCountService: aggregate per-domain deltas
        SDCountService->>SDCountRepo: Increment/Decrement per domain
        SDCountRepo-->>SDCountService: updated counts
    end
    WordService-->>WordController: result / id / status
    WordController-->>User: HTTP response

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~65 minutes

Possibly related PRs

• [TreeView] Add domain count #4046 — overlaps on domain-count functionality and related repository/controller changes.
• [WordRepository, WordService] Refine DeleteFrontier usage #4100 — related frontier-deletion API renames and bulk-delete moves.
• Fix singleton LiftService dependency on transient repositories #4091 — related changes to LiftService / LiftController wiring to use word service.

Suggested Labels

🟨Medium

Suggested Reviewers

• jasonleenaylor

Poem

🐰 I hopped through counts with joyful zeal,
I tallied domains with a tiny squeal,
No more big scans across the frontier,
Small increments now make counts appear —
A carrot for code, a hop and a feel!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 32.32% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly and accurately describes the main change: introducing a new database collection to track semantic domain sense counts, which is the primary objective of the PR.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch copilot/add-sense-count-collection

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[imnasnainaec]

@copilot

• SemanticDomainSenseCount -> SemanticDomainCount
• After adding/updating unit tests, make sure npm run test-backend passes
• For the WordService integrations, the semantic domain count update should be after the successful word update
• The LIFT controller will also need integration to update counts

[Copilot]

@copilot * SemanticDomainSenseCount -> SemanticDomainCount

• After adding/updating unit tests, make sure npm run test-backend passes
• For the WordService integrations, the semantic domain count...

Done in commits 2ea49c0 and 7220b09:

1. ✓ Renamed model to ProjectSemanticDomainCount (to avoid conflict with existing SemanticDomainCount in Statistics.cs)
2. ✓ All tests pass: npm run test-backend shows 1076 passed, 1 pre-existing failure (FFmpeg)
3. ✓ Count updates happen after successful word operations in WordService (Create, Update, Delete, DeleteFrontierWord, RestoreFrontierWords)
4. ✓ LIFT import integration added - LiftMerger now updates counts in SaveImportEntries after words are created

[imnasnainaec]

@copilot

[github-actions]

⚠️ Commit Message Format Issues ⚠️
commit d1c44e1c86:
3: B1 Line exceeds max length (100>80): "- Create unique compound index on (ProjectId, DomainId) in SemanticDomainCountRepository constructor"

commit 8bdcd225ca:
1: T1 Title exceeds max length (75>72): "Refactor SemanticDomainCountService and simplify WordService deletion logic"
6: B1 Line exceeds max length (101>80): "- Update WordService.Delete to use UpdateCountsForWordDeletion instead of UpdateCountsAfterWordUpdate"

commit 2ea49c05dd:
3: B1 Line exceeds max length (83>80): "- Add ProjectSemanticDomainCount model to track sense counts per domain per project"
4: B1 Line exceeds max length (95>80): "- Add ISemanticDomainCountRepository interface and SemanticDomainCountRepository implementation"
5: B1 Line exceeds max length (89>80): "- Add ISemanticDomainCountService interface and SemanticDomainCountService implementation"
6: B1 Line exceeds max length (93>80): "- Integrate count updates into WordService for create, update, delete, and restore operations"
9: B1 Line exceeds max length (83>80): "- Update all existing tests to inject the new SemanticDomainCountService dependency"

[codecov]

Codecov Report

❌ Patch coverage is 91.11111% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.56%. Comparing base (2353111) to head (fae569a).

Files with missing lines	Patch %	Lines
Backend/Services/WordService.cs	81.39%	8 Missing ⚠️
Backend/Services/SemanticDomainCountService.cs	90.66%	6 Missing and 1 partial ⚠️
Backend/Controllers/LiftController.cs	83.33%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #4065       +/-   ##
===========================================
+ Coverage   74.72%   86.56%   +11.83%     
===========================================
  Files         296       56      -240     
  Lines       10945     4883     -6062     
  Branches     1370      597      -773     
===========================================
- Hits         8179     4227     -3952     
+ Misses       2370      513     -1857     
+ Partials      396      143      -253     

Flag	Coverage Δ
backend	86.56% <91.11%> (+0.01%)	⬆️
frontend	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[imnasnainaec]

@copilot In @Backend/Models/ProjectSemanticDomainCount.cs around lines 11 - 43, Add a unique compound index on (ProjectId, DomainId) to optimize queries and enforce uniqueness: in the SemanticDomainCountRepository initialization (where ProjectDomainFilter and Increment upsert are used) create an index model for ProjectSemanticDomainCount using Builders.IndexKeys.Ascending(p => p.ProjectId).Ascending(p => p.DomainId) with new CreateIndexOptions { Unique = true } and call CreateOne/CreateIndex on the collection; ensure this runs once at repo startup or in the repository constructor so subsequent calls to Increment use the indexed/unique pair.

[Copilot]

@copilot In @Backend/Models/ProjectSemanticDomainCount.cs around lines 11 - 43, Add a unique compound index on (ProjectId, DomainId) to optimize queries and enforce uniqueness: in the SemanticDomain...

Done in commit d1c44e1. Added unique compound index on (ProjectId, DomainId) in the SemanticDomainCountRepository constructor. The index is created using Builders<ProjectSemanticDomainCount>.IndexKeys.Ascending(p => p.ProjectId).Ascending(p => p.DomainId) with CreateIndexOptions { Unique = true } to optimize queries and enforce uniqueness. MongoDB's CreateOne will handle creating the index only if it doesn't already exist.

[Copilot]

Pull request overview

This pull request introduces a new database collection to track semantic domain sense counts per project, significantly improving performance for statistics queries. Instead of scanning the entire Frontier collection to count domain usage, counts are now maintained incrementally as words are added, updated, or removed.

Changes:

• Added ProjectSemanticDomainCount model and SemanticDomainCountRepository for efficient count storage with compound index on (ProjectId, DomainId)
• Implemented SemanticDomainCountService to manage count updates across word lifecycle operations
• Refactored StatisticsService to use precomputed counts instead of scanning frontier words
• Migrated domain count endpoint from WordController to StatisticsController for better semantic organization
• Updated frontend to use new statistics API endpoint

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
Backend/Models/ProjectSemanticDomainCount.cs	New model for storing semantic domain sense counts per project
Backend/Repositories/SemanticDomainCountRepository.cs	Repository with atomic operations for count management
Backend/Services/SemanticDomainCountService.cs	Service layer managing count updates during word operations
Backend/Services/WordService.cs	Integrated count updates for word creation, update, deletion, and restoration
Backend/Services/StatisticsService.cs	Refactored to use precomputed counts for better performance
Backend/Services/LiftService.cs	Updated to use WordService instead of WordRepository for LIFT imports
Backend/Services/MergeService.cs	Updated to use new WordService methods that maintain counts
Backend/Controllers/StatisticsController.cs	Added GetDomainCount endpoint
Backend/Controllers/WordController.cs	Removed GetDomainWordCount endpoint (moved to Statistics)
Backend/Interfaces/*	Added new interfaces and updated existing ones
src/api/api/statistics-api.ts	Added getDomainCount API client method
src/api/api/word-api.ts	Removed getDomainWordCount API client method
src/backend/index.ts	Updated to use statistics API for domain counts
src/components/TreeView/TreeDepiction/DomainCountBadge.tsx	Updated to use new getDomainCount function
Backend.Tests/*	Comprehensive test coverage for new functionality

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.