What would you like?
When a checkpoint response arrives without a CheckpointToken, the SDK should treat it as a signal that the service will accept no further checkpoints from this invocation. The SDK should stop issuing checkpoints and end the invocation cleanly with Status: PENDING.
Why PENDING and not FAILED or a thrown error:
- A missing token does not mean the execution is finished. It means this invocation cannot make further progress. The invocation result must not claim the execution finished.
- A thrown error is an invocation failure. Lambda retries it, but the retry has no valid token and can do nothing useful. It also puts an error in customer logs for a condition the SDK understood.
- PENDING is already what the SDK returns for every suspend (wait, scheduled retry, pending callback). This is a suspend: the invocation is done for now and the execution continues later.
Current behavior
The checkpoint loop already detects a missing token on a non-empty batch (state.py, _process_checkpoint_batch), but it reads the missing token as "the execution reached a terminal state":
if output.checkpoint_token:
current_checkpoint_token = output.checkpoint_token
elif updates:
execution_completed = TrueIt then calls _settle_after_execution_completed(), which stops checkpointing and settles every queued operation with OrphanedChildException. The handler thread keeps running. On its next durable call, _reject_if_execution_completed raises OrphanedChildException. That class extends BaseException, and execution.py has no handler for it. So the exception propagates out of the Lambda handler as an unhandled error. The invocation fails; the SDK never returns PENDING. If the handler needs no further durable call it returns SUCCEEDED normally.
So Python already has the detection point. What is missing is the classification: a missing token on a non-empty batch is not always "execution completed"; it can also mean "no further checkpoints from this invocation, execution continues later."
Possible Implementation
- At the
elif updates: branch, treat the missing token as a suspend rather than as "execution completed". A completed execution never invokes the handler again, so answering PENDING for it is harmless; a suspended one needs PENDING.
- Raise
SuspendExecution (or a new sibling) in the handler thread instead of OrphanedChildException when the missing token is observed. execution.py already maps SuspendExecution to InvocationStatus.PENDING.
- Keep
_settle_after_execution_completed semantics for queued operations: they cannot be checkpointed, so they are abandoned and replay on the next invocation. AT_MOST_ONCE steps whose START landed in the last accepted checkpoint will not run again, which is the defined semantics of AT_MOST_ONCE.
- Unit test: a checkpoint response without
checkpoint_token on a non-empty batch produces status: PENDING, no further checkpoint calls, and no raised exception.
- Testing SDK: the local runner needs a way to omit the token on a chosen checkpoint so customers can test their handlers against this path.
Is this a breaking change?
No. The SDK's response set is unchanged; a new internal termination reason is added.
Additional Context
What would you like?
When a checkpoint response arrives without a
CheckpointToken, the SDK should treat it as a signal that the service will accept no further checkpoints from this invocation. The SDK should stop issuing checkpoints and end the invocation cleanly withStatus: PENDING.Why PENDING and not FAILED or a thrown error:
Current behavior
The checkpoint loop already detects a missing token on a non-empty batch (
state.py,_process_checkpoint_batch), but it reads the missing token as "the execution reached a terminal state":It then calls
_settle_after_execution_completed(), which stops checkpointing and settles every queued operation withOrphanedChildException. The handler thread keeps running. On its next durable call,_reject_if_execution_completedraisesOrphanedChildException. That class extendsBaseException, andexecution.pyhas no handler for it. So the exception propagates out of the Lambda handler as an unhandled error. The invocation fails; the SDK never returns PENDING. If the handler needs no further durable call it returns SUCCEEDED normally.So Python already has the detection point. What is missing is the classification: a missing token on a non-empty batch is not always "execution completed"; it can also mean "no further checkpoints from this invocation, execution continues later."
Possible Implementation
elif updates:branch, treat the missing token as a suspend rather than as "execution completed". A completed execution never invokes the handler again, so answering PENDING for it is harmless; a suspended one needs PENDING.SuspendExecution(or a new sibling) in the handler thread instead ofOrphanedChildExceptionwhen the missing token is observed.execution.pyalready mapsSuspendExecutiontoInvocationStatus.PENDING._settle_after_execution_completedsemantics for queued operations: they cannot be checkpointed, so they are abandoned and replay on the next invocation. AT_MOST_ONCE steps whose START landed in the last accepted checkpoint will not run again, which is the defined semantics of AT_MOST_ONCE.checkpoint_tokenon a non-empty batch producesstatus: PENDING, no further checkpoint calls, and no raised exception.Is this a breaking change?
No. The SDK's response set is unchanged; a new internal termination reason is added.
Additional Context