Skip to content

feat: support epoch-based checkpoint intervals - #9782

Open
tongchen126 wants to merge 1 commit into
modelscope:mainfrom
tongchen126:main
Open

feat: support epoch-based checkpoint intervals#9782
tongchen126 wants to merge 1 commit into
modelscope:mainfrom
tongchen126:main

Conversation

@tongchen126

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Add the --save_epochs training argument for epoch-based checkpoint intervals.

When set to a positive integer N, save_epochs automatically selects epoch-based checkpointing and saves at the
end of epochs N, 2N, and so on. The option is supported by standard Transformers-based Swift training, native
Megatron training, and Ray Megatron training. Epoch-based evaluation follows the same interval so that
load_best_model_at_end and best-checkpoint tracking remain consistent.

The implementation validates that save_epochs >= 1, converts epoch intervals to step intervals for Megatron and Ray
Megatron, and rejects streaming datasets in native Megatron and Ray Megatron when save_epochs is requested because
their epoch length is unknown.
Existing save_steps and save_strategy behavior remains unchanged when save_epochs is not set. The training UI and
English/Chinese command-line documentation were updated accordingly.

Experiment results

  • Python compilation and git diff --check passed for all modified files.
  • Isolated behavior checks passed for HF save/evaluation cadence, Ray step conversion, streaming rejection, and
    compatibility with existing Ray save_strategy='epoch' configurations.
  • A focused regression test was added. It could not complete in the available environment because the installed
    dependency stack hung during modelscope import; no test failure was reported.

Co-authored-by: andrewwchen <andrewwchen@tencent.com>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant