Open
Conversation
Add `s3-elasticsearch` as a new composite storage mode that routes all CRUD, versioning, history, and bulk operations to S3 while offloading all search queries to Elasticsearch. Key differences from SQLite/PostgreSQL+ES composites: - Uses `ElasticsearchBackend::new()` (standalone registry) instead of `with_shared_registry()`, since S3 has no search parameter registry - No `set_search_offloaded()` call needed; S3's stub SearchProvider already returns UnsupportedCapability for all search operations Changes: - rest/config.rs: add `S3Elasticsearch` variant with aliases `s3-elasticsearch` and `s3-es`; update error message and arg doc; add parse/display tests - hfs/main.rs: add match arm and `start_s3_elasticsearch()` with cfg feature guards for `s3` + `elasticsearch`; update module docs - CLAUDE.md: add s3-elasticsearch row to storage backends table - README.md: add S3+ES to configurations table, running example, env var - persistence/README.md: mark S3+Elasticsearch as implemented in both role matrix tables; update search offloading paragraph
S3 alone lacks search capability and cannot pass the full Inferno US Core test suite. Replace it with s3-elasticsearch so CRUD goes to S3 while all FHIR search queries are handled by Elasticsearch. - Matrix: s3 → s3-elasticsearch - Start Elasticsearch: condition now covers both sqlite-elasticsearch and s3-elasticsearch - Start HFS: replace s3 branch with s3-elasticsearch (adds HFS_ELASTICSEARCH_NODES) - Skip condition: matrix.backend != 's3-elasticsearch' when HFS_S3_BUCKET unset
…ite prefix - Add HFS_S3_PREFIX env var support to start_s3 and start_s3_elasticsearch so callers can scope all S3 keys under an optional global prefix - Update inferno.yml: pass HFS_S3_PREFIX=ci/<suite_id>/ per matrix job so parallel s3-elasticsearch jobs are fully isolated within the shared bucket - Empty only the job-scoped S3 prefix (not the whole bucket) before each run - Add AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY to the s3-elasticsearch HFS start command - Guard all post-HFS steps with the s3-elasticsearch/HFS_S3_BUCKET condition so jobs are skipped cleanly when the bucket secret is not configured - Document HFS_S3_PREFIX in README.md
…verwrite prompt) [skip ci]
… [skip ci] Document the S3+ES composite storage configuration including env vars, build/run commands, S3-compatible endpoint setup, key differences from SQLite/PG+ES, and programmatic composite assembly example. Also update the search offloading section to accurately reflect S3 behavior.
Switch S3Backend::new() to S3Backend::from_env_async() for proper async SDK config loading, and include full error details in S3 client error messages instead of dropping them when the message field is empty.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(persistence): add S3 + ElasticSearch composite storage backend
Summary
s3-elasticsearch(s3-es) as a new composite storage backend mode, wiring S3 as the primary store for CRUD/versioning/history and Elasticsearch as the dedicated search engineCompositeStoragepattern already used bysqlite-elasticsearchandpostgres-elasticsearchexecute_parallel_searchwhereUnsupportedCapabilityerrors from the primary backend (S3 has no search) were propagating and crashing full-text search queries — the fix promotes the auxiliary (ES) result to primary when the primary lacks search capabilityChanges
crates/rest/src/config.rs— addsStorageBackendMode::S3ElasticsearchwithDisplay,FromStr, and parse testscrates/hfs/src/main.rs— addsstart_s3_elasticsearch()startup function andbuild_search_registry()helper (S3 has no internal search registry, so ES builds one independently)crates/persistence/src/composite/storage.rs— graceful handling ofUnsupportedCapabilityin parallel searchcrates/persistence/tests/s3_es_tests.rs— integration test file for the S3+ES composite backend using MinIO + Elasticsearch via testcontainersUsage
Test Plan
cargo build -p helios-hfs --features s3,elasticsearchpassescargo test -p helios-persistence --features s3,elasticsearchpassescargo test -p helios-restpasses (config parse tests includeS3Elasticsearch)my-buckets-name,us-east-1) + Elasticsearch, with a 473-entry Synthea FHIR transaction bundle ingested end-to-endLive Server Verification
Tested against
HFS_STORAGE_BACKEND=s3-elasticsearchwith a 473-resource Synthea bundle:GET /healthok, backend:compositeGET /Patient?family=Ritchie586GET /Observation?code=72166-2GET /Condition?_content=sinusitisGET /MedicationRequest?_content=AlbuterolGET /Patient?birthdate=lt1970-01-01GET /Observation?_count=5GET /Patient/355GET /Observation?code=72166-2&_count=5_text(no narrative)GET /Condition?_text=sinusitis