Skip to content

feat: add missing extension YAMLs and test for completeness#723

Open
benbellick wants to merge 1 commit intomainfrom
add-missing-extensions-test
Open

feat: add missing extension YAMLs and test for completeness#723
benbellick wants to merge 1 commit intomainfrom
add-missing-extensions-test

Conversation

@benbellick
Copy link
Member

@benbellick benbellick commented Mar 5, 2026

Closes #722

Adds functions_set to DefaultExtensionCatalog and a test that dynamically discovers all extension YAMLs in substrait/extensions/ and verifies they are loaded, so future omissions are caught automatically.

Files not yet loaded are listed in the test's UNSUPPORTED_FILES set with inline comments explaining why.

@benbellick benbellick force-pushed the add-missing-extensions-test branch 19 times, most recently from 8bc6344 to 022b315 Compare March 5, 2026 17:08
@github-actions
Copy link

github-actions bot commented Mar 5, 2026

ACTION NEEDED

Substrait follows the Conventional Commits
specification
for
release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

@benbellick benbellick force-pushed the add-missing-extensions-test branch from 022b315 to 0f694f2 Compare March 5, 2026 19:12
@benbellick benbellick changed the title Add missing extension YAMLs and test for completeness feat: add missing extension YAMLs and test for completeness Mar 5, 2026
@benbellick
Copy link
Member Author

This was rebased off of #726, so I will wait for that one to be merged before merging this one.

@benbellick benbellick force-pushed the add-missing-extensions-test branch from 0f694f2 to f6b2f93 Compare March 5, 2026 20:44
@benbellick benbellick marked this pull request as ready for review March 5, 2026 20:45
@benbellick benbellick force-pushed the add-missing-extensions-test branch 5 times, most recently from 6d9545e to 22e6958 Compare March 5, 2026 21:39
@benbellick benbellick force-pushed the add-missing-extensions-test branch from 22e6958 to c288cc1 Compare March 5, 2026 21:48
@benbellick benbellick requested review from nielspardon and vbarua March 5, 2026 21:49
@benbellick
Copy link
Member Author

As noted when functions_aggregate_decimal_output was added, it is causing problems for substrait-java. So, I have continued to skip it in this PR, but made that skipping explicit.

Comment on lines +60 to +61
File extensionsDir = new File("../substrait/extensions");
assertTrue(extensionsDir.isDirectory(), "substrait/extensions directory not found");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering whether this should check the extension YAML files on the classpath instead of relying on the current working directory (especially given that @vbarua is planning to remove the submodule and move packaging the extension YAMLs into a separate JAR into substrait-packaging). The YAMLs should be already on the classpath. We're already loading classgraph for the isthmus-cli which can be useful to scan the classpath for any *.yaml files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me, I'll make that change shortly. Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw. it might be better to relocate the YAML files on the classpath into a subfolder like substrait/extensions to be easier recognizable as default YAML files. right now they are being added into the root of the classpath

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add test to catch missing extension YAML files in DefaultExtensionCatalog

2 participants