fix(data-branch): improve diff correctness, memory control, and output summary support#23789
Open
gouhongshen wants to merge 20 commits intomatrixorigin:mainfrom
Open
fix(data-branch): improve diff correctness, memory control, and output summary support#23789gouhongshen wants to merge 20 commits intomatrixorigin:mainfrom
gouhongshen wants to merge 20 commits intomatrixorigin:mainfrom
Conversation
# Conflicts: # pkg/frontend/stmt_kind.go
# Conflicts: # pkg/sql/parsers/dialect/mysql/mysql_sql.go
gouhongshen
requested review from
LeftHandCold,
XuPeng-SH,
heni02 and
iamlinjunhong
as code owners
Review Summary by QodoRefactor data branch operations with memory management, hash-based diff, and output formatting improvements
WalkthroughsDescription• Refactored data branch operations with comprehensive memory management improvements, including branchHashmapAllocator and branchHashmapDeallocator for throttled memory allocation • Redesigned in-memory hashmap storage from linked-list buckets to hash-indexed memStore with LRU eviction and tombstone support, plus spillStore for efficient disk-based overflow • Implemented hash-based diff algorithm for comparing data branches with LCA (Lowest Common Ancestor) support and conflict detection/resolution • Added new data branch output operations module supporting multiple output modes: summary statistics, row count, limited rows, and file-based exports (CSV/SQL) • Fixed commit timestamp indexing in in-memory committed insert filtering and added GetObjectCreateTS method for object creation timestamp retrieval • Added support for OUTPUT SUMMARY and OUTPUT COUNT syntax in diff operations with comprehensive parsing and validation • Consolidated data branch type definitions and constants into dedicated file for improved code organization • Added comprehensive test coverage for diff output modes, summary validation, update splitting, and complex type handling • Improved block data read function with proper error handling and DataSource parameter propagation • Added debug logging for commit timestamp placeholder scenarios to diagnose TN nonappendable block issues Diagramflowchart LR
A["Data Branch Operations"] --> B["Memory Management"]
A --> C["Hash-based Diff"]
A --> D["Output Formatting"]
B --> B1["branchHashmapAllocator"]
B --> B2["memStore with LRU"]
B --> B3["spillStore for Overflow"]
C --> C1["LCA Resolution"]
C --> C2["Conflict Detection"]
D --> D1["Summary Statistics"]
D --> D2["CSV/SQL Export"]
D --> D3["Batch Processing"]
File Changes1. pkg/frontend/data_branch.go
|
LeftHandCold
approved these changes
Mar 2, 2026
iamlinjunhong
approved these changes
Mar 5, 2026
heni02
approved these changes
Mar 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Which issue(s) this PR fixes:
Fixes #23751
What this PR does / why we need it:
OUTPUT SUMMARYsyntax fordata branch diff.Behavior changes
data branch diff ... output summaryis now supported.DELETE + INSERTinstead ofREPLACE INTOfor clearer semantics.Tests
OUTPUT SUMMARY.test/distributed/cases/git4data/branch/{diff,merge}.