Skip to content

BE-456: HashQL: Entity type model enrichment and graph store query extensions#8524

Open
indietyp wants to merge 5 commits intobm/be-429-hashql-per-island-entitypath-requirements-and-fetchislandfrom
bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query
Open

BE-456: HashQL: Entity type model enrichment and graph store query extensions#8524
indietyp wants to merge 5 commits intobm/be-429-hashql-per-island-entitypath-requirements-and-fetchislandfrom
bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query

Conversation

@indietyp
Copy link
Member

@indietyp indietyp commented Mar 8, 2026

🌟 What is the purpose of this PR?

Enriches the HashQL entity type model to match the full database schema, extends the graph-store SQL query builder with new expression types needed by the postgres compiler, and adds a database migration for the continuation composite type. This is the foundation layer for the postgres translation stack (BE-457, BE-306).

🔍 What does this change?

hashql-core:

  • Adds IdSnapshotVec, a typed-ID-indexed vector with transactional snapshot/rollback support (used by the postgres compiler for scoped state)
  • Expands the entity type definition with TemporalInterval, EntityTemporalMetadata, Confidence, provenance types, EntityMetadata, and LinkData fields to match the full database schema. Previously only EntityId, EntityRecordId, and partial LinkData were modeled.
  • Adds edition_id and ontology_type_ids symbols

graph-postgres-store query builder:

  • New expression types: FieldAccess (composite field extraction), JsonAgg, Constant::Null, Constant::U128, UnaryOperator::IsNotFalse
  • New cast targets: JsonB, Continuation, Numeric, Int, BigInt
  • Unnest now accepts multiple expressions (UNNEST(a, b, c))
  • SelectStatement gains OFFSET support
  • Convenience methods: .cast(), .not() and .grouped() now take self
  • Alias type changed from &'static str to Identifier for consistency
  • Visibility: mod postgres and mod table made pub for use by the HashQL postgres compiler
  • Unit tests for all new expression types

Database migration (v010__query):

  • Creates the continuation composite type: (filter boolean, block int, locals int[], values jsonb[]). This is the row type returned by compiled filter subqueries; the interpreter reads .filter to decide row inclusion and .block/.locals/.values to resume multi-block CFGs.

hashql-eval:

  • Reworks graph/read/path.rs to handle the expanded entity field structure
  • Scaffolding in lib.rs and error.rs for the postgres module (filled in by BE-306)

Test output updates:

  • ~55 files of updated .jsonc/.stdout/.stderr fixtures across eval (graph/read), HIR (lower/checking, specialization, graph-hoisting), and MIR (reify, inline) test suites, all cascading from the entity type changes

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

The entity type model still does not cover every field in the database schema (e.g., property_metadata inner structure is opaque). Fields are added as needed by the postgres compiler; remaining gaps will be filled incrementally.

🛡 What tests cover this?

  • IdSnapshotVec has doc tests and inline unit tests covering push/set/rollback/nested snapshots
  • New graph-store query builder tests for FieldAccess, JsonAgg, Null, U128, IsNotFalse, UNNEST with multiple args, cast types, and OFFSET
  • Existing compiletest suites (eval graph/read, HIR lowering, MIR reify/inline) with blessed output updates

❓ How to test this?

cargo nextest run -p hashql-core
cargo nextest run -p hashql-eval
cargo nextest run -p hash-graph-postgres-store --lib
cargo run --package hashql-compiletest -- run

indietyp added 3 commits March 8, 2026 16:17
feat: checkpoint (II)

feat: checkpoint (III)

feat: snapshot vec

feat: add dedicated filter

feat: checkpoint

feat: filter implementation

feat: filter implementation (mostly) done

chore: environment capture note

chore: always postgres bigint

feat: target clone

feat: simplify lookup

feat: move storage up

feat: eval entity path

chore: checkpoint

chore: checkpoint

chore: find entrypoint

feat: eval context

feat: eval cleanup

chore: cleanup

feat: track index

feat: wire up filter

feat: add error reporting

chore: checkpoint

feat: add traverse, and first postgres compiler outline

feat: traverse bitmap

feat: move traversal out

feat: projections

feat: projections

fix: clippy

feat: subquery projection for lateral

feat: checkpoint

feat: test plan

feat: checkpoint

feat: checkpoint – failing tests ;-;

feat: checkpoint – failing tests ;-;

feat: checkpoint — passing tests

fix: import

fix: entity type

feat: checkpoint

feat: attribute a cost to terminator placement switches

fix: import

feat: checkpoint

feat: checkpoint

chore: lint
@indietyp indietyp requested a review from a team as a code owner March 8, 2026 15:46
@vercel
Copy link

vercel bot commented Mar 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview, Comment Mar 8, 2026 5:13pm
petrinaut Ready Ready Preview Mar 8, 2026 5:13pm
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign Ignored Ignored Preview Mar 8, 2026 5:13pm
hashdotdesign-tokens Ignored Ignored Preview Mar 8, 2026 5:13pm

@cursor
Copy link

cursor bot commented Mar 8, 2026

PR Summary

Medium Risk
Medium risk because it changes Postgres schema (new composite type) and modifies SQL query compilation primitives (new expression types, casts, aliasing, OFFSET), which could subtly affect generated SQL and downstream compiler behavior.

Overview
Introduces a new Postgres migration creating the continuation composite type (and matching store migration), intended to support resumable/CFG-style query execution.

Extends the graph Postgres query builder with additional SQL constructs: composite FieldAccess, jsonb_agg, multi-arg UNNEST, OFFSET, more cast targets (jsonb, numeric, int, bigint, continuation), and new constants/operators (e.g. SQL NULL, u128, IS NOT FALSE), plus shifts SELECT aliases to a typed Identifier and makes previously-internal postgres/table modules public.

Enriches HashQL’s graph entity type definitions to include metadata/temporal/provenance/confidence/encodings and updates graph read path resolution + compiletest fixtures to reflect the new entity.metadata.record_id... access patterns; adds IdSnapshotVec to hashql-core for snapshot/rollback state management.

Written by Cursor Bugbot for commit 43775ba. This will update automatically on new commits. Configure here.

@indietyp indietyp changed the title HashQL: Entity type model enrichment and graph store query extensions BE-456: HashQL: Entity type model enrichment and graph store query extensions Mar 8, 2026
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

  • ✅ Fixed: Misleading enum variant name for jsonb_agg function
    • Renamed Function::JsonAgg to Function::JsonbAgg and updated all internal usages/tests to match the emitted jsonb_agg(...) SQL.
  • ✅ Fixed: Multiple new symbols defined but never referenced
    • Removed the eight unreferenced symbol definitions from sym.rs to eliminate dead entries and duplicate provenance naming.

Create PR

Or push these changes by commenting:

@cursor push 928f4a2c0b
Preview (928f4a2c0b)
diff --git a/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs b/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
--- a/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
+++ b/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
@@ -17,7 +17,7 @@
 pub enum Function {
     Min(Box<Expression>),
     Max(Box<Expression>),
-    JsonAgg(Box<Expression>),
+    JsonbAgg(Box<Expression>),
     JsonExtractText(Box<Expression>),
     JsonExtractAsText(Box<Expression>, PathToken<'static>),
     JsonExtractPath(Vec<Expression>),
@@ -55,7 +55,7 @@
                 expression.transpile(fmt)?;
                 fmt.write_char(')')
             }
-            Self::JsonAgg(expression) => {
+            Self::JsonbAgg(expression) => {
                 fmt.write_str("jsonb_agg(")?;
                 expression.transpile(fmt)?;
                 fmt.write_char(')')
@@ -821,9 +821,9 @@
     }
 
     #[test]
-    fn transpile_json_agg() {
+    fn transpile_jsonb_agg() {
         assert_eq!(
-            Expression::Function(Function::JsonAgg(Box::new(Expression::Parameter(1))))
+            Expression::Function(Function::JsonbAgg(Box::new(Expression::Parameter(1))))
                 .transpile_to_string(),
             "jsonb_agg($1)"
         );

diff --git a/libs/@local/hashql/core/src/symbol/sym.rs b/libs/@local/hashql/core/src/symbol/sym.rs
--- a/libs/@local/hashql/core/src/symbol/sym.rs
+++ b/libs/@local/hashql/core/src/symbol/sym.rs
@@ -36,7 +36,6 @@
     entity,
     entity_edition_id,
     entity_id,
-    entity_type_ids,
     entity_uuid,
     eq,
     Err,
@@ -55,8 +54,6 @@
     left_entity_confidence,
     left_entity_id,
     left_entity_provenance,
-    left_entity_uuid,
-    left_entity_web_id,
     link_data,
     List,
     lt,
@@ -76,10 +73,7 @@
     or,
     pow,
     properties,
-    property_metadata,
     provenance,
-    provenance_edition,
-    provenance_inferred,
     provided,
     r#as: "as",
     r#as_force: "as!",
@@ -101,8 +95,6 @@
     right_entity_confidence,
     right_entity_id,
     right_entity_provenance,
-    right_entity_uuid,
-    right_entity_web_id,
     Some,
     special_form,
     String,
This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.

@codecov
Copy link

codecov bot commented Mar 8, 2026

Codecov Report

❌ Patch coverage is 66.17954% with 162 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.95%. Comparing base (572197c) to head (43775ba).

Files with missing lines Patch % Lines
libs/@local/hashql/eval/src/graph/read/path.rs 13.54% 134 Missing ⚠️
libs/@local/hashql/core/src/id/snapshot_vec.rs 79.83% 25 Missing ⚠️
...store/src/store/postgres/query/statement/insert.rs 0.00% 2 Missing ⚠️
...src/store/postgres/query/expression/conditional.rs 98.85% 1 Missing ⚠️
Additional details and impacted files
@@                                           Coverage Diff                                           @@
##           bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland    #8524      +/-   ##
=======================================================================================================
+ Coverage                                                                62.94%   62.95%   +0.01%     
=======================================================================================================
  Files                                                                     1320     1321       +1     
  Lines                                                                   135105   135533     +428     
  Branches                                                                  5517     5521       +4     
=======================================================================================================
+ Hits                                                                     85041    85328     +287     
- Misses                                                                   49150    49293     +143     
+ Partials                                                                   914      912       -2     
Flag Coverage Δ
apps.hash-ai-worker-ts 1.40% <ø> (ø)
apps.hash-api 0.00% <ø> (ø)
local.hash-graph-sdk 7.73% <ø> (ø)
local.hash-isomorphic-utils 0.00% <ø> (ø)
rust.hash-graph-api 2.64% <ø> (ø)
rust.hash-graph-postgres-store 27.07% <97.63%> (+0.42%) ⬆️
rust.hashql-ast 87.23% <ø> (ø)
rust.hashql-compiletest 29.69% <ø> (ø)
rust.hashql-core 82.44% <87.30%> (+0.05%) ⬆️
rust.hashql-eval 62.99% <13.54%> (-6.15%) ⬇️
rust.hashql-hir 89.06% <ø> (ø)
rust.hashql-mir 92.45% <ø> (ø)
rust.hashql-syntax-jexpr 94.05% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@augmentcode
Copy link

augmentcode bot commented Mar 8, 2026

🤖 Augment PR Summary

Summary: This PR enriches the HashQL entity type model to better reflect the full graph database schema, and extends the Postgres query-builder/AST to support additional SQL constructs needed by the HashQL→Postgres compiler.

Changes:

  • Adds a new DB migration creating a continuation composite type used as the return row type for compiled filter subqueries.
  • Introduces IdSnapshotVec, a typed-ID-indexed vector with snapshot/rollback support for scoped compiler state.
  • Expands ::graph::types::knowledge::entity with temporal metadata, confidence/provenance, enriched link data, entity metadata, and encodings.
  • Extends the graph Postgres query builder with composite field access, jsonb_agg, additional constants/casts, multi-arg UNNEST, and OFFSET.
  • Updates HashQL eval graph path resolution to traverse the new entity field structure.
  • Blesses updated UI/compiletest fixtures across eval/HIR/MIR to reflect the new entity model.

Technical Notes: Query builder aliasing is standardized on an Identifier type; new expression/unit tests cover the added SQL features.

🤖 Was this summary useful? React with 👍 or 👎

Copy link

@augmentcode augmentcode bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 3 suggestions posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

@codspeed-hq
Copy link

codspeed-hq bot commented Mar 8, 2026

Merging this PR will not alter performance

✅ 80 untouched benchmarks


Comparing bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query (43775ba) with bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland (572197c)

Open in CodSpeed

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from ef9b858 to a91293c Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland branch from 922e42b to 572197c Compare March 8, 2026 17:03
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from a91293c to 43775ba Compare March 8, 2026 17:03
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

2 participants