Unit test over multiple python versions

[ianwesleyarmstrong]

I started with the currently supported python versions and any minor versions of delta lake > 1.0 - I'm happy to adjust these as needed. I thought it also might be worthwhile to test over different versions of pyspark, so I included the minor versions > 3 as well:

pyspark-version: ["3.1", "3.2", "3.3", "3.4"]

[ianwesleyarmstrong]

I've been running the tests on my fork and it appears that python 3.11 may cause some problems - I'll remove that for the time being.

https://github.com/ianwesleyarmstrong/mack/actions/runs/5652132175

[MrPowers]

How does this deal with the Delta Lake / PySpark versions that are incompatible?

I don't think mack supports Delta 1.0.

Here are the version combinations I think we should run (Delta Lake version | PySpark version):

2.4.x | 3.4.x
2.3.x | 3.3.x
2.2.x | 3.3.x
2.1.x | 3.3.x
2.0.x | 3.2.x

This comes from the docs.

[ianwesleyarmstrong]

How does this deal with the Delta Lake / PySpark versions that are incompatible?

I modified it to only test those pairs listed in the docs using Python 3.10 like you suggested

Successful run

Happy to make any other suggestion improvements you find! @MrPowers

[jaceklaskowski]

LGTM (and just learnt that mack uses poetry 👏👏👏 )

[andreale28]

Hi @MrPowers, Is there any updates on this branch? I'm looking forward to using mack on pyspark 3.4.1 and delta-spark 2.4.0.