Add gunicorn support for API server with rolling worker restarts

.pre-commit-config.yaml

@@ -533,7 +533,11 @@ repos:
           ^airflow-core/newsfragments/41368\.significant\.rst$|
           ^airflow-core/newsfragments/41761.significant\.rst$|
           ^airflow-core/newsfragments/43349\.significant\.rst$|
+          ^airflow-core/newsfragments/60921\.significant\.rst$|
           ^airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/pnpm-lock\.yaml$|
+          ^airflow-core/src/airflow/api_fastapi/gunicorn_config\.py$|
+          ^airflow-core/src/airflow/cli/commands/api_server_command\.py$|
+          ^airflow-core/src/airflow/api_fastapi/gunicorn_monitor\.py$|
           ^airflow-core/src/airflow/cli/commands/local_commands/fastapi_api_command\.py$|
           ^airflow-core/src/airflow/config_templates/|
           ^airflow-core/src/airflow/models/baseoperator\.py$|

airflow-core/docs/administration-and-deployment/web-stack.rst

@@ -57,3 +57,84 @@ separately. This might be useful for scaling them independently or for deploying
    airflow api-server --apps core
    # serve only the Execution API Server
    airflow api-server --apps execution
+
+Server Types
+------------
+
+The API server supports two server types: ``uvicorn`` (default) and ``gunicorn``.
+
+Uvicorn (Default)
+~~~~~~~~~~~~~~~~~
+
+Uvicorn is the default server type. It's simple to set up and works on all platforms including Windows.
+
+.. code-block:: bash
+
+   airflow api-server
+
+Gunicorn
+~~~~~~~~
+
+Gunicorn provides additional features for production deployments:
+
+- **Memory sharing**: Workers share memory via copy-on-write after fork, reducing total memory usage
+- **Rolling worker restarts**: Zero-downtime worker recycling to prevent memory accumulation
+- **Proper signal handling**: SIGTTOU kills the oldest worker (FIFO), enabling true rolling restarts
+
+.. note::
+
+   Gunicorn requires the ``gunicorn`` extra: ``pip install 'apache-airflow-core[gunicorn]'``
+
+   Gunicorn is Unix-only and does not work on Windows.
+
+To enable gunicorn mode:
+
+.. code-block:: bash
+
+   export AIRFLOW__API__SERVER_TYPE=gunicorn
+   airflow api-server
+
+Rolling Worker Restarts
+^^^^^^^^^^^^^^^^^^^^^^^
+
+To enable periodic worker recycling (useful for long-running processes to prevent memory accumulation):
+
+.. code-block:: bash
+
+   export AIRFLOW__API__SERVER_TYPE=gunicorn
+   export AIRFLOW__API__WORKER_REFRESH_INTERVAL=43200  # Restart workers every 12 hours
+   export AIRFLOW__API__WORKER_REFRESH_BATCH_SIZE=1   # Restart one worker at a time
+   airflow api-server
+
+The rolling restart process:
+
+1. Spawns new workers before killing old ones (zero downtime)
+2. Waits for new workers to be ready (process title check)
+3. Performs HTTP health check to verify workers can serve requests
+4. Kills old workers (oldest first)
+5. Repeats until all original workers are replaced
+
+Configuration Options
+^^^^^^^^^^^^^^^^^^^^^
+
+The following configuration options are available in the ``[api]`` section:
+
+- ``server_type``: ``uvicorn`` (default) or ``gunicorn``
+- ``worker_refresh_interval``: Seconds between worker refresh cycles (0 = disabled, default)
+- ``worker_refresh_batch_size``: Number of workers to refresh per cycle (default: 1)
+- ``reload_on_plugin_change``: Reload when plugin files change (default: False)
+
+When to Use Gunicorn
+^^^^^^^^^^^^^^^^^^^^
+
+Use gunicorn when you need:
+
+- Long-running API server processes where memory accumulation is a concern
+- Multi-worker deployments where memory sharing matters
+- Production environments requiring zero-downtime worker recycling
+
+Use the default uvicorn when:
+
+- Running on Windows
+- Running in development or testing environments
+- Running short-lived containers (e.g., Kubernetes pods that get recycled)

airflow-core/docs/extra-packages-ref.rst

@@ -126,6 +126,8 @@ other packages that can be used by airflow or some of its providers.
 +---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
 | graphviz            | ``pip install 'apache-airflow[graphviz]'``          | Graphviz renderer for converting Dag to graphical output                   |
 +---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
+| gunicorn            | ``pip install 'apache-airflow[gunicorn]'``          | Gunicorn server with rolling worker restarts for the API server            |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
 | ldap                | ``pip install 'apache-airflow[ldap]'``              | LDAP authentication for users                                              |
 +---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
 | leveldb             | ``pip install 'apache-airflow[leveldb]'``           | Required for use leveldb extra in google provider                          |

airflow-core/newsfragments/60921.significant.rst

@@ -0,0 +1,51 @@
+Add gunicorn support for API server with zero-downtime worker recycling
+
+The API server now supports gunicorn as an alternative server with rolling worker restarts
+to prevent memory accumulation in long-running processes.
+
+**Key Benefits:**
+
+* **Rolling worker restarts**: New workers spawn and pass health checks before old workers
+  are killed, ensuring zero downtime during worker recycling.
+
+* **Memory sharing**: Gunicorn uses preload + fork, so workers share memory via
+  copy-on-write. This significantly reduces total memory usage compared to uvicorn's
+  multiprocess mode where each worker loads everything independently.
+
+* **Correct FIFO signal handling**: Gunicorn's SIGTTOU kills the oldest worker (FIFO),
+  not the newest (LIFO), which is correct for rolling restarts.
+
+**Configuration:**
+
+.. code-block:: ini
+
+    [api]
+    # Use gunicorn instead of uvicorn
+    server_type = gunicorn
+
+    # Enable rolling worker restarts every 12 hours
+    worker_refresh_interval = 43200
+
+    # Restart workers one at a time
+    worker_refresh_batch_size = 1
+
+Or via environment variables:
+
+.. code-block:: bash
+
+    export AIRFLOW__API__SERVER_TYPE=gunicorn
+    export AIRFLOW__API__WORKER_REFRESH_INTERVAL=43200
+
+**Requirements:**
+
+Install the gunicorn extra: ``pip install 'apache-airflow-core[gunicorn]'``
+
+**Note on uvicorn (default):**
+
+The default uvicorn mode does not support rolling worker restarts because:
+
+1. With workers=1, there is no master process to send signals to
+2. uvicorn's SIGTTOU kills the newest worker (LIFO), defeating rolling restart purposes
+3. Each uvicorn worker loads everything independently with no memory sharing
+
+If you need worker recycling or memory-efficient multi-worker deployment, use gunicorn.

airflow-core/pyproject.toml

@@ -188,14 +188,17 @@ dependencies = [
 "memray" = [
     "memray>=1.19.0",
 ]
+"gunicorn" = [
+    "gunicorn>=23.0.0",
+]
 "otel" = [
     "opentelemetry-exporter-prometheus>=0.47b0",
 ]
 "statsd" = [
     "statsd>=3.3.0",
 ]
 "all" = [
-    "apache-airflow-core[graphviz,kerberos,otel,statsd]"
+    "apache-airflow-core[graphviz,gunicorn,kerberos,otel,statsd]"
 ]
 
 [project.scripts]