[ML] Automate patch version bump in CI pipeline by edsavage · Pull Request #3030 · elastic/ml-cpp

edsavage · 2026-04-30T23:35:10Z

Summary

Automates patch elasticsearchVersion bumps for ml-cpp via the ml-cpp-version-bump Buildkite pipeline: validate parameters, optionally run the bump (topic branch + GitHub PR + merge), notify Slack, then wait for DRA artifacts. Minor version bumps remain out of scope for a follow-up change.

Dev-tools Python tests for this work run under the existing dev-tools pytest PR check (see #3052); this PR does not add duplicate pytest wiring in format_and_validation.yml.sh.

What changed (current tree)

Two-phase pipeline (Buildkite meta-data limitation)

Phase 1 — .buildkite/job-version-bump.json.py
- Validate (dev-tools/validate_version_bump_params.sh on Wolfi): patch-only rules, WORKFLOW=patch, NEW_VERSION / BRANCH checks, optional git fetch + Python re-validation after pull when enabled.
- Sets Buildkite meta-data when the bump would be a no-op (origin already at NEW_VERSION).
- Schedule follow-up (dev-tools/version_bump_upload_phase2.sh): uploads phase 2 only when Slack / bump / DRA work should run (step if cannot depend on meta-data per Buildkite docs).
Phase 2 — .buildkite/job-version-bump-phase2.json.py (uploaded above)
- Bump — dev-tools/bump_version.sh: topic branch ci/ml-cpp-version-bump-*, commit gradle.properties, push, gh pr create, default gh pr merge --auto --squash, optional immediate merge, optional VERSION_BUMP_MERGE_ADMIN. Sets meta-data ml_cpp_version_bump_changed and ml_cpp_version_bump_pr_url (non-empty URL only; Buildkite rejects empty values). DRY_RUN=true skips push/PR/merge. Does not edit .backportrc.json.
- Slack — .buildkite/pipelines/send_slack_version_bump_notification.sh: Wolfi image (so buildkite-agent meta-data works), step-level notify so the message is not blocked on build end; link to the opened PR when present.
- DRA wait — dev-tools/wait_version_bump_dra.py: long poll; skipped when DRY_RUN or when validation/bump indicated no-op / no PR opened.

Supporting scripts

dev-tools/create_github_pull_request.sh + dev-tools/ensure_github_cli.sh — install/use gh on Wolfi for PR create/merge.
dev-tools/version_bump_validation.py — shared rules used by validate shell + pytest.
.buildkite/pipelines/send_version_bump_notification.sh — drop obsolete Workflow line from legacy blocked/finished Slack templates.

Tests

dev-tools/unittest/test_version_bump_validation.py — validation behaviour (opt-in git integration via VERSION_BUMP_GIT_INTEGRATION).
dev-tools/unittest/test_job_version_bump_pipeline.py — JSON shape / env wiring for both pipeline generators.

History / review note (2026-05-07)

Prior 32 exploratory commits were squashed into one for review (25e681cd8 on feature/version-bump-patch-only). Net diff vs main is unchanged aside from dropping a stray blank line in format_and_validation.yml.sh that had no functional effect.

Labels

Please keep :ml and appropriate version labels for backport automation when merging.

prodsecmachine · 2026-04-30T23:35:21Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Widen ml-cpp-version-bump branch_configuration for PR elastic#3030 smoke tests; revert to main-only after validation. Made-with: Cursor

The temporary ml-cpp-version-bump branch_configuration change is proposed separately so PR elastic#3030 stays focused on the bump automation. Made-with: Cursor

Copilot

Pull request overview

This PR updates the ML CI “version bump” pipeline from a stubbed/manual flow to an automated patch-version bump workflow that (1) notifies Slack, (2) bumps elasticsearchVersion on a target branch, and (3) optionally polls published artifact JSON until the expected versions appear.

Changes:

Add dev-tools/bump_version.sh to bump elasticsearchVersion (and update .backportrc.json when targeting main) with optional DRY_RUN=true.
Add a dedicated Buildkite pipeline YAML generator for Slack notifications in the version-bump pipeline.
Update .buildkite/job-version-bump.json.py to run the bump step and then poll staging/snapshot artifact JSON via elastic/json-watcher.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
`dev-tools/bump_version.sh`	New automation script to bump `elasticsearchVersion`, commit, and push (with `DRY_RUN` support).
`.buildkite/pipelines/send_slack_version_bump_notification.sh`	New pipeline snippet generator to configure Slack notifications specifically for version bump runs.
`.buildkite/job-version-bump.json.py`	Replaces the old stub/block flow with bump + (optional) artifact polling steps.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

elasticsearchmachine · 2026-05-04T22:58:37Z

Pinging @elastic/ml-core (Team:ML)

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Widen ml-cpp-version-bump branch_configuration for PR elastic#3030 smoke tests; revert to main-only after validation. Made-with: Cursor

The temporary ml-cpp-version-bump branch_configuration change is proposed separately so PR elastic#3030 stays focused on the bump automation. Made-with: Cursor

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.

Replace the ml-cpp-version-bump stub with a two-phase Buildkite flow. Phase 1 (job-version-bump.json.py): validate_version_bump_params.sh on Wolfi (patch-only rules; optional git fetch re-check when enabled), then version_bump_upload_phase2.sh uploads phase 2 only when work remains (step conditionals cannot rely on build meta-data). Phase 2 (job-version-bump-phase2.json.py): bump_version.sh bumps elasticsearchVersion on a ci/ml-cpp-version-bump-* topic branch, pushes, opens a PR via GitHub CLI (ensure_github_cli.sh, create_github_pull_request.sh), defaults to gh pr merge --auto --squash, optional immediate merge and VERSION_BUMP_MERGE_ADMIN; records meta-data for no-op and PR URL. Slack step (send_slack_version_bump_notification.sh) uses Wolfi and step notify. DRA wait (wait_version_bump_dra.py) skips on DRY_RUN or when validation/bump report no change. Adds dev-tools/unittest for validation and pipeline JSON. Does not modify .backportrc.json (reserved for future release automation). Minor version bumps stay a follow-up change. Co-authored-by: Cursor <cursoragent@cursor.com>

edsavage · 2026-05-07T23:31:35Z

Cleanup (2026-05-07): Commit history was squashed to a single commit (25e681cd8) for review. The PR description was rewritten to match the current diff vs main (two-phase version-bump pipeline, gh PR flow, Slack + DRA gating, tests). If anything in the thread above contradicts this, treat the description and latest commit as source of truth.

- Reject leading/trailing whitespace in version_bump_validation; trim NEW_VERSION/BRANCH in shell after env read - Use temp-file sed rewrite in bump_version.sh for BusyBox/GNU/BSD portability - wait_version_bump_dra: fail when buildkite-agent missing or meta-data get errors; treat exit 1 + not-found stderr as missing key - Verify gh tarball against official gh_*_checksums.txt before install - Require --body in create_github_pull_request.sh; extend pipeline unit tests Co-authored-by: Cursor <cursoragent@cursor.com>

edsavage · 2026-05-07T23:42:57Z

Copilot follow-up (pushed 239958d1a):

Topic	Change
Whitespace on semver / branch	`version_bump_validation.py` rejects outer whitespace on `NEW_VERSION`, `BRANCH`, and `current_version`; `parse_semver` / `parse_release_branch` no longer strip. Shell `validate_version_bump_params.sh` + `bump_version.sh` trim CR/spaces after reading env so CI payloads stay consistent.
`sed -i` on Wolfi/BusyBox	`bump_version.sh` `sed_inplace` writes via `mktemp` + `mv` instead of `sed -i`.
DRA wait meta-data	`wait_version_bump_dra.py` exits 2 if `buildkite-agent` is missing or `meta-data get` fails unexpectedly; treats exit 1 with “not found”-style stderr as missing key.
`gh` tarball integrity	`ensure_github_cli.sh` downloads official `gh_*_checksums.txt` and checks `sha256sum` before extracting.
`--body` docs vs behaviour	`create_github_pull_request.sh` now requires `--body`; tests updated + new negative test.

Pytest: dev-tools/unittest/test_version_bump_validation.py + test_job_version_bump_pipeline.py (23 passed, 2 skipped).

elasticsearchmachine added the v9.5.0 label Apr 30, 2026

edsavage added a commit to edsavage/ml-cpp that referenced this pull request May 1, 2026

[ML] TEMP: allow version-bump pipeline on PR test branch

e18afd3

Widen ml-cpp-version-bump branch_configuration for PR elastic#3030 smoke tests; revert to main-only after validation. Made-with: Cursor

edsavage requested a review from Copilot May 1, 2026 02:01

Copilot started reviewing on behalf of edsavage May 1, 2026 02:02 View session

Copilot AI reviewed May 1, 2026

View reviewed changes

Comment thread dev-tools/bump_version.sh Outdated

edsavage requested a review from Copilot May 4, 2026 22:56

Copilot started reviewing on behalf of edsavage May 4, 2026 22:57 View session

edsavage marked this pull request as ready for review May 4, 2026 22:57

edsavage added >build >non-issue :ml labels May 4, 2026

Copilot AI reviewed May 4, 2026

View reviewed changes

edsavage requested a review from Copilot May 4, 2026 23:55

Copilot started reviewing on behalf of edsavage May 4, 2026 23:56 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

Comment thread dev-tools/git_push_auth_probe.sh Outdated

Comment thread dev-tools/validate_version_bump_params.sh Outdated

Comment thread .buildkite/job-version-bump.json.py Outdated

Comment thread dev-tools/run_dev_tools_tests.sh Outdated

edsavage force-pushed the feature/version-bump-patch-only branch from a558843 to 46c5b73 Compare May 7, 2026 04:12

edsavage added a commit to edsavage/ml-cpp that referenced this pull request May 7, 2026

[ML] TEMP: allow version-bump pipeline on PR test branch

0fad7e0

Widen ml-cpp-version-bump branch_configuration for PR elastic#3030 smoke tests; revert to main-only after validation. Made-with: Cursor

edsavage force-pushed the feature/version-bump-patch-only branch from 46c5b73 to 9293f24 Compare May 7, 2026 04:46

edsavage added auto-backport Automatically merge backport PRs when CI passes v9.4.1 v9.3.5 labels May 7, 2026

edsavage requested a review from Copilot May 7, 2026 05:05

Copilot started reviewing on behalf of edsavage May 7, 2026 05:07 View session

Copilot AI reviewed May 7, 2026

View reviewed changes

edsavage force-pushed the feature/version-bump-patch-only branch from 9293f24 to 25e681c Compare May 7, 2026 23:30

edsavage requested review from jan-elastic and valeriy42 May 8, 2026 02:44

Conversation

edsavage commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed (current tree)

Two-phase pipeline (Buildkite meta-data limitation)

Supporting scripts

Tests

History / review note (2026-05-07)

Labels

Uh oh!

prodsecmachine commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

elasticsearchmachine commented May 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

edsavage commented May 7, 2026

Uh oh!

edsavage commented May 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

edsavage commented Apr 30, 2026 •

edited

Loading

prodsecmachine commented Apr 30, 2026 •

edited

Loading