[WIP] Add Python 3.14 support

[jrmccluskey]

Adds support for Python 3.14 to Beam. Part of #37247

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[gemini-code-assist]

Summary of Changes

Hello @jrmccluskey, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces initial support for Python 3.14 across the Apache Beam project. It involves significant updates to the build system, container configurations, and test infrastructure to integrate the new Python version. The changes ensure that Beam can be built and tested with Python 3.14, while also addressing specific dependency challenges, such as the current incompatibility of TensorFlow with Python 3.14, through conditional logic.

Highlights

• Python 3.14 Integration: Added Python 3.14 to the list of officially supported Python versions, enabling its use across various build configurations and test environments within Apache Beam.
• Build System Updates: Modified Gradle build files (build.gradle.kts, buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy, gradle.properties, settings.gradle.kts) to incorporate Python 3.14 into pre-commit, post-commit, and Docker build tasks, and updated the maximum supported Python version to 14.
• Container Image Support: Introduced new Gradle build files and base_image_requirements.txt for Python 3.14 base, distroless, and ML container images, facilitating the creation and deployment of Beam SDKs with Python 3.14.
• Dependency Management Adjustments: Updated sdks/python/setup.py to specify different pyarrow dependency versions based on the Python version (specifically for Python 3.14) and adjusted the warning for untested Python versions to apply to >= 3.15.
• Test Suite Expansion: Extended Dataflow, Direct, Portable, and Tox test suites to include Python 3.14, updating sdks/python/test-suites/gradle.properties and sdks/python/tox.ini to ensure comprehensive testing for the new Python version.
• Conditional ML Dependency Handling: Implemented a condition in sdks/python/container/common.gradle to temporarily skip ML dependency generation for Python 3.14 due to current incompatibilities with TensorFlow.
• Minor Dependency Bumps: Updated several Python package versions in base_image_requirements.txt for Python 3.12 and 3.13 to ensure up-to-date dependencies.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• build.gradle.kts
  • Added py314 to pythonPreCommit and pythonDockerBuildPreCommit tasks.
  • Registered a new python314PostCommit task with dependencies for Dataflow, Direct, HDFS, and Portable test suites.
  • Updated portablePythonPreCommit and pythonSparkPostCommit to reference py314 instead of py313.
• buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
  • Increased maxPythonVersion from 13 to 14.
• gradle.properties
  • Appended 3.14 to the python_versions list.
• sdks/python/conftest.py
  • Updated MAX_SUPPORTED_PYTHON_VERSION tuple from (3, 13) to (3, 14).
• sdks/python/container/common.gradle
  • Added a conditional block to skip ML dependency generation for Python 3.14, noting TensorFlow's current lack of support.
  • Modified the condition for skipping GPU requirements to include Python 3.14 alongside 3.13.
• sdks/python/container/distroless/py314/build.gradle
  • New file: Configures the build for the Python 3.14 distroless container (note: pythonVersion is set to '3.13' in the provided diff, likely a typo and intended to be '3.14').
• sdks/python/container/ml/py314/build.gradle
  • New file: Configures the build for the Python 3.14 ML container.
• sdks/python/container/py312/base_image_requirements.txt
  • Updated cachetools from 6.2.4 to 6.2.6.
  • Updated cryptography from 46.0.3 to 46.0.4.
  • Updated google-api-python-client from 2.188.0 to 2.189.0.
  • Updated google-auth from 2.47.0 to 2.49.0.dev0.
  • Updated google-cloud-aiplatform from 1.133.0 to 1.135.0.
  • Updated google-cloud-kms from 3.9.0 to 3.10.0.
  • Updated google-genai from 1.59.0 to 1.61.0.
  • Removed greenlet==3.3.0.
  • Updated httplib2 from 0.31.1 to 0.31.2.
  • Removed jeepney==0.9.0.
  • Updated multidict from 6.7.0 to 6.7.1.
  • Updated numpy from 2.4.1 to 2.4.2.
  • Updated oracledb from 3.4.1 to 3.4.2.
  • Updated orjson from 3.11.5 to 3.11.7.
  • Removed overrides==7.7.0.
  • Updated packaging from 25.0 to 26.0.
  • Updated pip from 25.3 to 26.0.
  • Updated proto-plus from 1.27.0 to 1.27.1.
  • Updated pycparser from 2.23 to 3.0.
  • Updated pyparsing from 3.3.1 to 3.3.2.
  • Removed SecretStorage==3.5.0.
  • Updated setuptools from 80.9.0 to 80.10.2.
  • Updated SQLAlchemy from 2.0.45 to 2.0.46.
  • Updated testcontainers from 4.14.0 to 4.14.1.
  • Updated tqdm from 4.67.1 to 4.67.3.
  • Updated wheel from 0.45.1 to 0.46.3.
  • Updated wrapt from 2.0.1 to 2.1.1.
• sdks/python/container/py313/base_image_requirements.txt
  • Updated cachetools from 6.2.4 to 6.2.6.
  • Updated cryptography from 46.0.3 to 46.0.4.
  • Updated google-auth from 2.47.0 to 2.49.0.dev0.
  • Updated google-cloud-aiplatform from 1.133.0 to 1.135.0.
  • Updated google-cloud-kms from 3.9.0 to 3.10.0.
  • Updated google-genai from 1.59.0 to 1.61.0.
  • Removed greenlet==3.3.0.
  • Updated httplib2 from 0.31.1 to 0.31.2.
  • Removed jeepney==0.9.0.
  • Updated multidict from 6.7.0 to 6.7.1.
  • Updated numpy from 2.4.1 to 2.4.2.
  • Updated oracledb from 3.4.1 to 3.4.2.
  • Updated orjson from 3.11.5 to 3.11.7.
  • Removed overrides==7.7.0.
  • Updated packaging from 25.0 to 26.0.
  • Updated pip from 25.3 to 26.0.
  • Updated proto-plus from 1.27.0 to 1.27.1.
  • Updated protobuf from 6.33.4 to 6.33.5.
  • Updated pycparser from 2.23 to 3.0.
  • Updated pymilvus from 2.6.6 to 2.6.8.
  • Updated pyparsing from 3.3.1 to 3.3.2.
  • Removed SecretStorage==3.5.0.
  • Updated setuptools from 80.9.0 to 80.10.2.
  • Updated SQLAlchemy from 2.0.45 to 2.0.46.
  • Updated testcontainers from 4.14.0 to 4.14.1.
  • Updated tqdm from 4.67.1 to 4.67.3.
  • Updated wheel from 0.45.1 to 0.46.3.
  • Updated wrapt from 2.0.1 to 2.1.1.
• sdks/python/container/py314/base_image_requirements.txt
  • New file: Contains a comprehensive list of pinned Python package dependencies for the Python 3.14 base container image.
• sdks/python/container/py314/build.gradle
  • New file: Configures the build for the Python 3.14 container.
• sdks/python/container/run_generate_requirements.sh
  • Updated the example usage in the help message from Python 3.8 to 3.10.
• sdks/python/setup.py
  • Modified pyarrow_dependency to conditionally specify pyarrow>=3.0.0,<19.0.0 for python_version <= "3.13" and pyarrow>=22.0.0,<24.0.0 for python_version >= "3.14".
  • Adjusted the Python version warning for untested versions from >= 3.14 to >= 3.15.
  • Added Programming Language :: Python :: 3.14 to the classifiers.
• sdks/python/test-suites/dataflow/py314/build.gradle
  • New file: Configures the build for the Python 3.14 Dataflow test suite.
• sdks/python/test-suites/direct/py314/build.gradle
  • New file: Configures the build for the Python 3.14 Direct test suite.
• sdks/python/test-suites/gradle.properties
  • Updated various test suite configurations (e.g., dataflow_precommit_it_task_py_versions, flink_validates_runner_precommit_py_versions) to include 3.14 instead of 3.13.
• sdks/python/test-suites/portable/py314/build.gradle
  • New file: Configures the build for the Python 3.14 Portable test suite.
• sdks/python/test-suites/tox/py314/build.gradle
  • New file: Configures the build for the Python 3.14 Tox test suite.
• sdks/python/tox.ini
  • Added py314 to the envlist.
  • Included py314 in the testenv configurations for general tests, macOS, Windows, cloud, and dask environments.
  • Added a new testenv:py314-ml section for ML-specific tests with accelerate dependency.
• settings.gradle.kts
  • Added include statements for all new Python 3.14 related modules: container:py314, container:distroless:py314, container:ml:py314, test-suites:dataflow:py314, test-suites:direct:py314, test-suites:portable:py314, and test-suites:tox:py314.

Activity

• The pull request is currently marked as Work In Progress (WIP) by the author, jrmccluskey.
• The primary goal is to add Python 3.14 support to the Apache Beam project.
• No specific review comments or interactions have been recorded yet beyond the initial submission.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[codecov]

Codecov Report

❌ Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 35.88%. Comparing base (22d1c57) to head (771e9ec).
⚠️ Report is 8 commits behind head on master.

Files with missing lines	Patch %	Lines
sdks/python/setup.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #37500      +/-   ##
============================================
- Coverage     35.88%   35.88%   -0.01%     
  Complexity     1691     1691              
============================================
  Files          1063     1063              
  Lines        166752   166741      -11     
  Branches       1227     1227              
============================================
- Hits          59839    59829      -10     
+ Misses       104716   104713       -3     
- Partials       2197     2199       +2     

Flag	Coverage Δ
python	39.68% <75.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.