Skip to content

Commit 7022983

Browse files
authored
docs: update roadmap (#3543)
1 parent e29bf65 commit 7022983

1 file changed

Lines changed: 15 additions & 4 deletions

File tree

docs/source/contributor-guide/roadmap.md

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,11 +26,11 @@ helpful to have a roadmap for some of the major items that require coordination
2626

2727
### Iceberg Integration
2828

29-
Iceberg integration is still a work-in-progress ([#2060]), with major improvements expected in the next few
30-
releases. The default `auto` scan mode now uses `native_iceberg_compat` instead of `native_comet`, enabling
31-
support for complex types.
29+
Iceberg tables reads are now fully native, powered by a scan operator backed by Iceberg-rust ([#2528]). We anticipate
30+
major improvements expected in the next few releases, including bringing Iceberg table format V3 features (_e.g._,
31+
encryption) to the reader.
3232

33-
[#2060]: https://github.com/apache/datafusion-comet/issues/2060
33+
[#2528]: https://github.com/apache/datafusion-comet/pull/2528
3434

3535
### Spark 4.0 Support
3636

@@ -40,6 +40,17 @@ more Spark SQL tests and fully implementing ANSI support ([#313]) for all suppor
4040
[#313]: https://github.com/apache/datafusion-comet/issues/313
4141
[#1637]: https://github.com/apache/datafusion-comet/issues/1637
4242

43+
### Dynamic Partition Pruning
44+
45+
Iceberg table scans support Dynamic Partition Pruning (DPP) filters generated by Spark's `PlanDynamicPruningFilters`
46+
optimizer rule ([#3349]). However, we still need to bring this functionality to our Parquet reader. Furthermore,
47+
Spark's `PlanAdaptiveDynamicPruningFilters` optimizer rule runs after Comet's rules, so DPP with Adaptive Query
48+
Execution requires a redesign of Comet's plan translation. We are focused on implementing DPP to keep Comet competitive
49+
with benchmarks that benefit from this feature like TPC-DS. This effort can be tracked at [#3510].
50+
51+
[#3349]: https://github.com/apache/datafusion-comet/pull/3349
52+
[#3510]: https://github.com/apache/datafusion-comet/issues/3510
53+
4354
### Removing the native_comet scan implementation
4455

4556
The `native_comet` scan implementation is now deprecated and will be removed in a future release ([#2186], [#2177]).

0 commit comments

Comments
 (0)