feat: Cast numeric (non int) to timestamp by coderfender · Pull Request #3559 · apache/datafusion-comet

coderfender · 2026-02-20T21:06:48Z

Which issue does this PR close?

Closes ##3560 .

Rationale for this change

Adding support for further more native cast support.

Float / Double cast to Timestamp (with ANSI support)
DecimalType -> Timestamp cast (with ANSI support)
Boolean -> Timestamp cast

This should more or less close out the cast matrix in terms of support (barring other not planned casts such as Timestamp -> Int etc)

What changes are included in this PR?

How are these changes tested?

Enabled unit tests in CometCastSuite.scala and added new ones (along with benchmarking scripts) on the rust side to test cast at all Eval Modes

coderfender · 2026-02-21T19:33:18Z

spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala

+   * Uses except (difference) to find differences without using collect()
+   * Checks cometDF and sparkDF including schemas
+   */
+  protected def assertDataFrameEquals(


In my recent PR , I had to cast back the generated I had to resort to this route because we have no native comet support from Timestamp -> Float causing operator / native comet check assertion failures . I tried casting back to String and Long either the cast is incompatible or unsupported on native side

spark/src/test/scala/org/apache/comet/CometCastSuite.scala

coderfender · 2026-02-24T21:44:50Z

@andygrove I moved the 'assertDataFrameEquals' inside cast test to leverage existing Try and ANSI checks . This should help us gain confidence with native code and perhaps merge it

andygrove · 2026-02-26T14:18:53Z

Test failure:

- cast FloatType to TimestampType *** FAILED *** (323 milliseconds)
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 2197.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2197.0 (TID 2396) (localhost executor driver): org.apache.comet.CometNativeException: [CAST_INVALID_INPUT] The value 'NaN' of the type "DOUBLE" cannot be cast to "TIMESTAMP" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

coderfender · 2026-02-26T15:30:17Z

Thank you @andygrove There seems to be failure per spark version after I fixed the error message handling. I am making changes to fix tests with the right error message

coderfender · 2026-02-28T04:45:01Z

it seems like there is some regression between Spark's cast from Float / Double to Timestamp in terms handling extreme values . Investigating into this

coderfender · 2026-02-28T20:18:31Z

Only spark 4 test is failing and it seems like we might not be using right exception parsing

parthchandra · 2026-03-02T16:37:14Z

Only spark 4 test is failing and it seems like we might not be using right exception parsing

Would you like to wait forhttps://github.com//pull/3580 ?

coderfender · 2026-03-03T06:24:44Z

@parthchandra sure !

coderfender · 2026-03-03T06:25:35Z

@andygrove , marking it as a draft to not accidentally merge until @parthchandra 's PR is reviewed

parthchandra · 2026-03-03T22:09:23Z

@coderfender I've marked this as ready for review if you could rebase and resolve the ci failures.

coderfender · 2026-03-03T22:10:38Z

Sure thank you

coderfender · 2026-03-03T22:13:36Z

It seems like the errors are transient @parthchandra . Can you please rerun failed actions whenever you get a chance

native/spark-expr/src/conversion_funcs/numeric.rs

parthchandra · 2026-03-05T23:47:23Z

native/spark-expr/src/conversion_funcs/numeric.rs

+            let value_256 = i256::from_i128(value);
+            let micros_256 = value_256 * i256::from_i128(MICROS_PER_SECOND as i128);
+            let ts = micros_256 / i256::from_i128(scale_factor);
+            builder.append_value(ts.as_i128() as i64);


This doesn't look right. Casting down from i256 to i128 and then to i64 will truncate too many bits silently. Should probably check for overflow here before the cast.

You'll probably need to pass eval_mode here to check whether to return null or throw error on overflow. Or you could just restrict this to legacy mode (probably easier).

@parthchandra , thank you for the comment. Spark doesn't support Eval Mode for Decimal and Int type (while it checks overflow for float and double types) to timestamp casts and rather produces incorrect values
Refer : https://github.com/apache/spark/blob/972897433082b1a7136b877b4fa37970961169d0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L791

parthchandra · 2026-03-06T00:06:47Z

native/spark-expr/src/conversion_funcs/numeric.rs

+                } else {
+                    // Path 2: Multiply then check overflow - error says BIGINT
+                    let micros = val * MICROS_PER_SECOND as f64;
+                    if micros.floor() <= i64::MAX as f64 && micros.ceil() >= i64::MIN as f64 {


There may be a boundary condition issue here but I am not sure if there is a better way.
i64::MAX as f64 is actually greater than i64::MAX so this check here has gap where we might get some incorrect results?

parthchandra · 2026-03-06T00:22:01Z

native/spark-expr/src/conversion_funcs/numeric.rs

+            builder.append_null();
+        } else {
+            let value = arr.value(i);
+            // Note: spark's big decimal 


Incomplete comment?

coderfender force-pushed the cast_float_to_timestamp branch from 41393a5 to 5aa49b0 Compare February 20, 2026 21:09

coderfender changed the title ~~feat: Cast non numeric to timestamp~~ feat: Cast numeric (non ints) to timestamp Feb 20, 2026

coderfender changed the title ~~feat: Cast numeric (non ints) to timestamp~~ feat: Cast numeric (non int) to timestamp Feb 20, 2026

coderfender force-pushed the cast_float_to_timestamp branch from 5aa49b0 to 88e0c83 Compare February 21, 2026 00:42

coderfender commented Feb 21, 2026

View reviewed changes

andygrove reviewed Feb 23, 2026

View reviewed changes

spark/src/test/scala/org/apache/comet/CometCastSuite.scala Outdated Show resolved Hide resolved

coderfender force-pushed the cast_float_to_timestamp branch from bd49514 to 4d3a6a3 Compare February 24, 2026 20:37

coderfender added 4 commits February 26, 2026 15:57

float_to_timestamp

b9b0ad4

non_numeric_to_timestamp

b41a43a

fix_bool_to_timestamp_support

55cae7f

fix_bool_to_timestamp_support

e64e194

coderfender force-pushed the cast_float_to_timestamp branch 2 times, most recently from 67d6e37 to 0546da6 Compare February 27, 2026 20:36

fix_ansi_support_when_non_using_collect

3f3dd19

coderfender force-pushed the cast_float_to_timestamp branch from 0546da6 to 3f3dd19 Compare February 27, 2026 21:35

TEST_SPARK_EXPRESSION_REFACTOR

ac46828

coderfender marked this pull request as draft March 3, 2026 06:24

parthchandra marked this pull request as ready for review March 3, 2026 17:25

Merge branch 'main' into cast_float_to_timestamp

9bb2ea4

parthchandra mentioned this pull request Mar 4, 2026

feat: [ANSI] Ansi sql error messages #3580

Open

coderfender commented Mar 4, 2026

View reviewed changes

native/spark-expr/src/conversion_funcs/numeric.rs Show resolved Hide resolved

parthchandra reviewed Mar 4, 2026

View reviewed changes

native/spark-expr/src/conversion_funcs/numeric.rs Outdated Show resolved Hide resolved

rebase_main

e593089

coderfender force-pushed the cast_float_to_timestamp branch from 7bbe49b to e593089 Compare March 5, 2026 22:50

parthchandra reviewed Mar 6, 2026

View reviewed changes

Conversation

coderfender commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

coderfender Feb 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderfender commented Feb 24, 2026

Uh oh!

andygrove commented Feb 26, 2026

Uh oh!

coderfender commented Feb 26, 2026

Uh oh!

coderfender commented Feb 28, 2026

Uh oh!

coderfender commented Feb 28, 2026

Uh oh!

parthchandra commented Mar 2, 2026

Uh oh!

coderfender commented Mar 3, 2026

Uh oh!

coderfender commented Mar 3, 2026

Uh oh!

parthchandra commented Mar 3, 2026

Uh oh!

coderfender commented Mar 3, 2026

Uh oh!

coderfender commented Mar 3, 2026

Uh oh!

Uh oh!

Uh oh!

parthchandra Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

parthchandra Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

coderfender Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

parthchandra Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

parthchandra Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderfender commented Feb 20, 2026 •

edited

Loading

coderfender Feb 21, 2026 •

edited

Loading

coderfender Mar 6, 2026 •

edited

Loading