Is your feature request related to a problem?
No response
What is the motivation behind your request?
Pyspark 4.x has been around for quite a while now. Is there any plan for Ibis to support this? Currently ibis is restricted to <4.
This would be helpful since we want to use ibis for our databricks jobs, and all the latest databricks runtimes use spark 4.
Describe the solution you'd like
Add support for pyspark 4.x, even if there are some limitations. Or if this is not feasible, just let us know so we can plan an alternative approach.
What version of ibis are you running?
No version of ibis currently supports this.
What backend(s) are you using, if any?
pyspark
Code of Conduct
Is your feature request related to a problem?
No response
What is the motivation behind your request?
Pyspark 4.x has been around for quite a while now. Is there any plan for Ibis to support this? Currently ibis is restricted to <4.
This would be helpful since we want to use ibis for our databricks jobs, and all the latest databricks runtimes use spark 4.
Describe the solution you'd like
Add support for pyspark 4.x, even if there are some limitations. Or if this is not feasible, just let us know so we can plan an alternative approach.
What version of ibis are you running?
No version of ibis currently supports this.
What backend(s) are you using, if any?
pyspark
Code of Conduct