[FLINK-40420][python] Add set operations to DataFrame API

[beetle0915]

What is the purpose of the change

Implement FLINK-40420 by exposing set operations on the PyFlink DataFrame API.

Brief change log

• Add union, union_all, intersect, intersect_all, minus, and minus_all as PublicEvolving methods that delegate to the corresponding Table API operations and return new DataFrames.
• Reject non-DataFrame operands while leaving schema, environment, and execution-mode validation to the underlying Table API.
• Add API documentation and tests for row multiplicities, NULLs, empty inputs, positional column matching, validation, chaining, type conversion, and streaming union_all.

Verifying this change

Local verification uses Python 3.12.11, Java 17, and an existing Flink 2.4-SNAPSHOT Java distribution. Python sources are from this branch.

• python -m pytest -q pyflink/dataframe/tests: 351 passed, including 12 new test methods and batch/streaming MiniCluster execution tests. Eight existing deprecation warnings remain.
• python -m flake8 --config=tox.ini pyflink/dataframe: passed.
• python -m mypy --config-file tox.ini: passed (83 source files).
• Sphinx HTML build with -W --keep-going: passed.
• git diff --check: passed.

This PR is a draft: full-repository ./mvnw clean verify, end-to-end tests, and the multi-version CI matrix have not been run locally. Java artifacts were reused rather than rebuilt for this Python-only patch.

Existing Table API mixed-type limitation

During execution testing, intersect, intersect_all, minus, and minus_all failed for mixed INT/BIGINT inputs even though the resolved result schema was BIGINT. This also reproduces by calling the original Table API directly from a clean Python checkout of the base commit, e01bbcacd962189c726d8324a35d66cc162712e7, without importing DataFrame. union and union_all succeed; explicitly casting the inputs to matching BIGINT types makes all six operations succeed.

The clean-checkout comparison had 14 passing cases and four failing mixed-type cases across the six operations and three input scenarios. This PR does not change the planner or add implicit casts in the DataFrame wrappers. The four affected method docstrings advise explicit casts. Should the underlying Table API problem be tracked in a separate issue?

Minimal direct Table API reproduction (no DataFrame usage)

from pyflink.table import EnvironmentSettings, TableEnvironment

t_env = TableEnvironment.create(EnvironmentSettings.in_batch_mode())
left = t_env.sql_query(
    "SELECT * FROM (VALUES (1), (2), (2), (3)) AS T(id)"
)
right = t_env.sql_query(
    "SELECT * FROM (VALUES (CAST(2 AS BIGINT)), "
    "(CAST(2147483648 AS BIGINT))) AS T(other_id)"
)
with left.intersect(right).execute().collect() as rows:
    print(list(rows))

Expected: one row containing 2. Observed: a type mismatch while applying ReplaceIntersectWithSemiJoinRule. minus fails in ReplaceMinusWithAntiJoinRule; the two ALL variants fail with a UnionTransformation input type mismatch.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): yes, adds six PublicEvolving DataFrame methods
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? Python API docstrings and the DataFrame reference autosummary index, with versionadded:: 2.4.0.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: OpenAI Codex (codex-cli 0.154.0-alpha.6.2)

[flinkbot]

CI report:

• a939a1b Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build