chore(deps): upgrade to DataFusion 52 by ethan-tyler · Pull Request #1997 · apache/iceberg-rust

ethan-tyler · 2026-01-06T04:18:22Z

Which issue does this PR close?

Validates and adopts DataFusion 52

What changes are included in this PR?

Upgrade DataFusion integration from 51.x to 52.x
Keep the DataFusion Python dependency dynamic within major version 52
Update the Python FFI table provider bridge for DataFusion 52 API/ABI expectations:
- session-aware __datafusion_table_provider__(session) integration
- DF52 compatible FFI table provider construction with task context and logical codec handling
Update sqllogictest physical plan expectations for DataFusion 52 planner output changes
Refresh lockfiles impacted by the upgrade

DataFusion FFI API change

DataFusion 52 expanded table provider FFI construction to include task context and optional logical codec parameters.

Updated the Rust/Python bridge accordingly allowing filter/logical expression serialization remains compatible across the FFI boundary.

Are these changes tested?

Yes

ethan-tyler · 2026-01-06T04:23:10Z

The audit failure (RUSTSEC-2026-0001 for rkyv) is unrelated to this PR - it's being addressed in #1994. Will rebase once that lands.

ethan-tyler · 2026-01-06T04:42:11Z

Fix for Python Bindings CI Failure

The initial PR failed the Bindings Python CI workflow due to a breaking API change in DataFusion 51+'s FFI module.

Root cause: FFI_TableProvider::new signature changed from 3 to 5 arguments.

Fix (commit 33d5608):

Added datafusion and datafusion-execution dependencies to bindings/python/Cargo.toml
Updated datafusion_table_provider.rs to create a TaskContextProvider from SessionContext and pass it to FFI_TableProvider::new

The core iceberg-rust crates were already compatible with DataFusion 52 - only the Python bindings needed this update.

timsaucer · 2026-01-19T21:12:38Z

Please let me know if you run into difficulties with this PR also regarding the FFI change. I think that my approach in apache/datafusion-python#1337 will help resolve the missing elements here.

Smith-Cruise · 2026-02-21T07:49:27Z

Is there any progress now?

ethan-tyler · 2026-02-22T01:59:26Z

Is there any progress now?

I rebased and got CI cleaned up. DF 52 Python wheels should be landing in the next day or so - https://lists.apache.org/thread/76v9pmqh7cflgjwx4wnqsmdzw00v62bl. To limit an additional follow up, I am waiting to include that and will open the PR.

timsaucer · 2026-02-23T18:53:24Z

Let me know if you need any help with this PR.

ethan-tyler · 2026-02-23T19:02:20Z

Let me know if you need any help with this PR.

Thanks Tim - it's been a hot minute since I worked on Iceberg and got myself into a CI pickle. I think we should be good after this most recent push. I'll let you know if I run into any trouble.

.github/workflows/bindings_python_ci.yml

Cargo.toml

blackmwk · 2026-02-24T01:00:52Z

bindings/python/tests/test_datafusion_table_provider.py

-assert (
-    datafusion.__version__ >= "45"
-)  # iceberg table provider only works for datafusion >= 45
+if Version(datafusion.__version__) < Version("52.0.0"):


Not related to this pr, but is it possible to extract the version from workspace Cargo.toml?

yes - I can do this on a follow up PR

+1 should take from datafusion-ffi

bindings/python/DEPENDENCIES.rust.tsv

bindings/python/pyproject.toml

kevinjqliu

mostly lgtm, a couple nits

kevinjqliu · 2026-02-25T17:24:23Z

Cargo.toml

-arrow-schema = "57.0"
-arrow-select = "57.0"
-arrow-string = "57.0"
+arrow-arith = "57.1"


👍
datafusion 51.1.0 uses arrow-* 57.1
https://crates.io/crates/datafusion/52.1.0/dependencies

kevinjqliu · 2026-02-25T17:25:40Z

bindings/python/DEPENDENCIES.rust.tsv

i think we usually bump these during release.

Previous upgrade PR didnt include this #1899

bindings/python/pyproject.toml

kevinjqliu · 2026-02-25T17:33:27Z

bindings/python/tests/test_datafusion_table_provider.py


    # monkey patch the __datafusion_table_provider__ method to the iceberg table
-    def __datafusion_table_provider__(self):
+    def __datafusion_table_provider__(self, session):


👍
session is available here
https://github.com/apache/datafusion-python/blob/22086650c8df8b7bc382130c9560f76762dbe6c0/python/datafusion/context.py#L96-L102

kevinjqliu · 2026-02-25T17:39:30Z

bindings/python/src/datafusion_table_provider.rs


 use crate::runtime::runtime;

+fn validate_pycapsule(capsule: &Bound<PyCapsule>, name: &str) -> PyResult<()> {


👍
these are from the 52.0.0 upgrade guide

https://github.com/apache/datafusion-python/blob/main/docs/source/user-guide/upgrade-guides.rst#datafusion-5200

and matches the example https://github.com/apache/datafusion-python/blob/22086650c8df8b7bc382130c9560f76762dbe6c0/examples/datafusion-ffi-example/src/utils.rs

bindings/python/src/datafusion_table_provider.rs

kevinjqliu · 2026-02-25T18:04:12Z

bindings/python/tests/test_datafusion_table_provider.py

-assert (
-    datafusion.__version__ >= "45"
-)  # iceberg table provider only works for datafusion >= 45
+if Version(datafusion.__version__) < Version("52.0.0"):


+1 should take from datafusion-ffi

Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>

xanderbailey · 2026-02-25T22:11:04Z

Cargo.toml

-datafusion = "51.0"
-datafusion-cli = "51.0"
-datafusion-sqllogictest = "51.0"
+datafusion = "52.1"


Don't know if it needs to block but 52.2 is close to release apache/datafusion#20287. It should be a trivial bump from 51.1 so we can always do it later?

yep! lets do that :)

voting just opened - https://lists.apache.org/thread/8g5ozm5vxfnzf1ows028vfrbxl1sz21z

bindings/python/tests/test_datafusion_table_provider.py

## Which issue does this PR close? - Closes #. ## What changes are included in this PR? While reviewing #1997, i noticed a couple of improvements we can make 1. `make install` should install the local editable `pyiceberg-core` so that i can do `make install && make test` 2. CI should fail on warnings 3. test should not read local env files (we made a similar fix in pyiceberg apache/iceberg-python#3006). Otherwise, tests were reading `~/.pyiceberg.yaml` and polluting the runs ## Are these changes tested? Yes. For (2), see that [it fails in CI](https://github.com/apache/iceberg-rust/actions/runs/22410058261/job/64880767855?pr=2178) before removing the use of deprecated `register_table_provider`

Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>

kevinjqliu

LGTM!

Thanks a bunch

kevinjqliu · 2026-02-26T20:12:06Z

Thanks for the PR @ethan-tyler lets move this forward and follow up with 52.2 if necessary.

Thanks everyone for the review

ethan-tyler force-pushed the chore/datafusion-52-validation branch from 33d5608 to 6e77511 Compare January 6, 2026 04:45

ethan-tyler mentioned this pull request Jan 6, 2026

Release DataFusion 52.0.0 (Dec 2025 / Jan 2026) apache/datafusion#18566

Closed

41 tasks

ethan-tyler force-pushed the chore/datafusion-52-validation branch from 6a154d0 to f4cf8da Compare January 7, 2026 16:42

ethan-tyler force-pushed the chore/datafusion-52-validation branch 2 times, most recently from ae7b70e to 723e3a6 Compare January 23, 2026 23:59

ethan-tyler force-pushed the chore/datafusion-52-validation branch from 723e3a6 to a19062d Compare February 21, 2026 18:03

ethan-tyler changed the title ~~[WIP] chore(deps): validate DataFusion 52 compatibility~~ chore(deps): upgrade to DataFusion 52 Feb 23, 2026

ethan-tyler force-pushed the chore/datafusion-52-validation branch from 19690bf to 3309d6c Compare February 23, 2026 19:59

ethan-tyler marked this pull request as ready for review February 23, 2026 20:15

mbutrovich self-requested a review February 23, 2026 20:20

kevinjqliu reviewed Feb 23, 2026

View reviewed changes

.github/workflows/bindings_python_ci.yml Outdated Show resolved Hide resolved

Cargo.toml Outdated Show resolved Hide resolved

blackmwk reviewed Feb 24, 2026

View reviewed changes

ethan-tyler added 4 commits February 23, 2026 23:17

chore(deps): upgrade DataFusion/Arrow dependencies to 52.x

2ebf9d4

test(sqllogictest): update DataFusion 52 physical plan expectations

3d0870a

fix(python): align DataFusion 52 FFI bridge and dependency locks

19050fb

chore(deps): bump DataFusion dependencies to 52.1

196c528

ethan-tyler force-pushed the chore/datafusion-52-validation branch from 3309d6c to 196c528 Compare February 24, 2026 04:22

blackmwk reviewed Feb 25, 2026

View reviewed changes

bindings/python/DEPENDENCIES.rust.tsv Outdated Show resolved Hide resolved

bindings/python/pyproject.toml Outdated Show resolved Hide resolved

kevinjqliu reviewed Feb 25, 2026

View reviewed changes

kevinjqliu mentioned this pull request Feb 25, 2026

infra: improve python binding setup #2178

Merged

ethan-tyler and others added 2 commits February 25, 2026 16:46

fix(python): address datafusion 52 review feedback

c90dd15

Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>

Merge branch 'main' into chore/datafusion-52-validation

7c1f82c

xanderbailey reviewed Feb 25, 2026

View reviewed changes

kevinjqliu reviewed Feb 25, 2026

View reviewed changes

bindings/python/tests/test_datafusion_table_provider.py Outdated Show resolved Hide resolved

ethan-tyler and others added 2 commits February 25, 2026 21:27

revert: datafusion_test_utils

0089961

Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>

Merge branch 'main' into chore/datafusion-52-validation

a690927

kevinjqliu approved these changes Feb 26, 2026

View reviewed changes

kevinjqliu merged commit b24ab63 into apache:main Feb 26, 2026
20 checks passed

ethan-tyler deleted the chore/datafusion-52-validation branch February 26, 2026 20:26


		use crate::runtime::runtime;

		fn validate_pycapsule(capsule: &Bound<PyCapsule>, name: &str) -> PyResult<()> {

Conversation

ethan-tyler commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

DataFusion FFI API change

Are these changes tested?

Uh oh!

ethan-tyler commented Jan 6, 2026

Uh oh!

ethan-tyler commented Jan 6, 2026

Fix for Python Bindings CI Failure

Uh oh!

timsaucer commented Jan 19, 2026

Uh oh!

Smith-Cruise commented Feb 21, 2026

Uh oh!

ethan-tyler commented Feb 22, 2026

Uh oh!

timsaucer commented Feb 23, 2026

Uh oh!

ethan-tyler commented Feb 23, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kevinjqliu commented Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ethan-tyler commented Jan 6, 2026 •

edited

Loading