[Feature] Calcite PPL search result highlighting

[RyanL1997]

Description.

For ease of review, please reference the design doc in issue #5156.

• Add request-level highlight API for PPL queries. Callers (OSD, API, CLI) can include an optional highlight JSON object in the PPL request body, which the backend forwards as-is to OpenSearch.
• Highlight config is carried across the thread boundary (REST handler → sql-worker) via AbstractPlan, then set as a ThreadLocal for Calcite planning and execution.
• Highlight metadata from OpenSearch hits is collected via a side-channel ThreadLocal in OpenSearchIndexEnumerator and merged back into the JDBC/SimpleJson response as a highlights array parallel to datarows.

Related Issues

• Resolve [FEATURE RFC] PPL Search Result Highlighting #5156
• Resolve [FEATURE] Supporting Query Highlight Feature into PPL API #5059

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds per-request PPL highlighting end-to-end: request parsing, plan-level propagation via ThreadLocal, conditional hidden _highlight column in Calcite schema, attaching HighlightBuilder to OpenSearch requests, extracting highlights from responses, and exposing per-row highlights through QueryResult/formatters with unit and integration tests.

Changes

Cohort / File(s)	Summary
Plan context & propagation core/src/main/java/org/opensearch/sql/calcite/CalcitePlanContext.java, core/src/main/java/org/opensearch/sql/executor/execution/AbstractPlan.java	Add ThreadLocal highlightConfig and plan-level highlightConfig field with accessors to carry highlight settings across thread boundary.
Plan execution flow core/src/main/java/org/opensearch/sql/executor/execution/QueryPlan.java, core/src/main/java/org/opensearch/sql/executor/execution/ExplainPlan.java	Set/clear CalcitePlanContext ThreadLocal around execute/explain to ensure worker thread sees plan highlight config.
PPL request surface ppl/src/main/java/org/opensearch/sql/ppl/PPLService.java, ppl/src/main/java/org/opensearch/sql/ppl/domain/PPLQueryRequest.java	Parse highlight from PPL request JSON and attach it to created plan before submission.
Calcite schema / planning opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java, core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java	When highlight config exists, add hidden _highlight column (ANY) to row type and preserve it through projection/visitor logic.
OpenSearch request building opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java, opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteEnumerableIndexScan.java	Add applyHighlightConfig to build/attach OpenSearch HighlightBuilder from CalcitePlanContext and apply it before executing requests; include in explain output.
Response handling & enumerator opensearch/src/main/java/org/opensearch/sql/opensearch/executor/OpenSearchExecutionEngine.java, opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/OpenSearchIndexEnumerator.java, opensearch/src/main/java/org/opensearch/sql/opensearch/response/OpenSearchResponse.java	Carry _highlight as an opaque per-row value (excluded from schema), special-case resolution in enumerator, and centralize highlight field constant usage.
Protocol & formatting protocol/src/main/java/org/opensearch/sql/protocol/response/QueryResult.java, protocol/src/main/java/org/opensearch/sql/protocol/response/format/JdbcResponseFormatter.java, protocol/src/main/java/org/opensearch/sql/protocol/response/format/SimpleJsonResponseFormatter.java	Add QueryResult.highlights() and include highlights in JDBC/Simple JSON outputs when present.
Expressions & constants core/src/main/java/org/opensearch/sql/expression/HighlightExpression.java	Introduce HIGHLIGHT_FIELD constant and use it instead of hard-coded string.
Tests (unit & integration) core/src/test/.../ExplainPlanTest.java, core/src/test/.../QueryPlanTest.java, opensearch/src/test/.../AbstractCalciteIndexScanHighlightTest.java, protocol/src/test/.../*, ppl/src/test/.../PPLQueryRequestTest.java, integ-test/src/test/java/.../CalcitePPLHighlightIT.java, integ-test/src/yamlRestTest/resources/rest-api-spec/test/ppl_highlight.yml	Add unit and integration tests for propagation, request-building, highlighter config application, enumerator extraction, QueryResult highlights, and formatted outputs.
Docs docs/user/ppl/interfaces/endpoint.md	Add Highlight section documenting highlight request object, parameters, examples, response format, and limitations.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant PPLService
    participant Planner as CalcitePlanner
    participant Worker as ExecThread
    participant CalciteCtx as CalcitePlanContext
    participant OpenSearch as OpenSearchNode
    participant Enumerator as OpenSearchIndexEnumerator
    participant Protocol as QueryResult/Formatter

    Client->>PPLService: POST /_plugins/_ppl (with optional "highlight")
    PPLService->>Planner: build execution plan
    PPLService->>Planner: set plan.highlightConfig
    Planner->>Worker: submit plan (worker thread)
    Worker->>CalciteCtx: setHighlightConfig(plan.highlightConfig)
    Worker->>OpenSearch: execute search (HighlightBuilder attached)
    OpenSearch-->>Enumerator: return hits with highlight fragments
    Enumerator->>Worker: produce rows including hidden _highlight value
    Worker->>Protocol: build QueryResult (extract highlights())
    Worker->>CalciteCtx: clearHighlightConfig()
    Protocol-->>Client: JSON response (optional "highlights" array)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• Support spath with dynamic fields #5058 — touches projection/visitor logic in CalciteRelNodeVisitor, overlapping with hidden-field preservation changes.

Suggested labels

calcite

Suggested reviewers

• LantaoJin
• penghuo
• ps48
• kavithacm
• derek-ho
• joshuali925
• anirudha
• Swiddis
• yuancu
• GumpacG

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 17.81% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title '[Feature] Calcite PPL search result highlighting' directly and clearly describes the main feature being added—PPL result highlighting via Calcite.
Description check	✅ Passed	The description is related to the changeset, explaining the highlighting feature, thread-local propagation, response formatting, and referencing design docs and related issues.
Linked Issues check	✅ Passed	The PR successfully implements all key objectives from issues #5156 and #5059: request-level highlight API, thread-local config propagation, conditional _highlight column, HighlightBuilder integration, highlight metadata collection, and response formatting with highlights array.
Out of Scope Changes check	✅ Passed	All changes are scoped to highlighting feature implementation: config propagation (AbstractPlan, ThreadLocal), Calcite integration (_highlight column, projection handling), OpenSearch request building, response extraction/formatting, and comprehensive tests. No unrelated modifications detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

Persistent review updated to latest commit b290e06

[opensearch-trigger-bot]

This PR is stalled because it has been open for 2 weeks with no activity.

[github-actions]

Persistent review updated to latest commit ff9eb4f

[github-actions]

Persistent review updated to latest commit 6abfc1a

[RyanL1997]

Closing this and moving to #5234 for new AST level approach. Thanks. cc @penghuo @dai-chen