Skip to content

feat: add disk based event buffering#531

Open
mchristopher wants to merge 14 commits intomainfrom
disk-buffer
Open

feat: add disk based event buffering#531
mchristopher wants to merge 14 commits intomainfrom
disk-buffer

Conversation

@mchristopher
Copy link
Member

@mchristopher mchristopher commented Feb 4, 2026

Adding support for buffering chain data to disk before uploading to S3. This is needed due to the growing data volume on chain, and the addition of the full input and output UTXO data now in place.

Cursor Bugbot found 2 potential issues for commit 64e6de2

@cursor
Copy link

cursor bot commented Feb 4, 2026

PR Summary

Medium Risk
Changes the live ingestion/upload path to use new disk-based buffering and streaming S3 uploads, which can affect correctness and recovery behavior under failures. Risk is mitigated by explicit cleanup/reset logic and added safety checks in multipart uploads.

Overview
Switches sv-dune from in-memory event accumulation to a disk-based Avro buffer (DiskBuffer) that flushes per block and finalizes into temporary Avro files for upload, reducing peak memory usage for large chain data.

Updates the service loop to buffer converted blocks/transactions/receipts to disk, then upload sequentially from file paths via Processor::process_data_from_file, resetting the buffer immediately after finalize to avoid infinite retry loops on upload failure.

Enhances Avro tooling with a global schema cache and a new AvroFileWriter, and extends S3 storage with store_from_file streaming uploads (including multipart-from-file) plus stricter multipart handling (abort on missing ETag). Also hardens runtime packaging by running the Docker image as non-root and adding basic debugging utilities.

Written by Cursor Bugbot for commit afe3f7b. This will update automatically on new commits. Configure here.

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments