{182998236}: dispatch retry sql to pool by emelialei88 · Pull Request #5873 · bloomberg/comdb2

emelialei88 · 2026-04-16T16:03:47Z

When a verified retry happens, what happens currently is that the all the statements within this transaction will rerun in the COMMIT SQL thread. This will cause the thread to block and unfair for the other normal transactions. In this PR, we schedule each statement into its own SQL thread.

When a retry happens, we call srs_tran_replay_prepare to do some preparations like saving the original done_cb and assign the new done_cb to be srs_tran_replay_async. Then for each retry, srs_tran_replay_begin is called to set some states and dispatch the first statement. If srs_tran_replay_prepare returns without failure, we will return error code RC_INTERNAL_RETRY so that the callers will return the thread without any cleanup.

At the end of execution, srs_tran_replay_async will be called. It will either dispatch the next statement, start another retry if this one fails, handle error or successfully return. For the last two cases we restore the done_cb before returning.

roborivers

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
selectv_rcode_serialretry_generated [failed with core dumped]
logfill [db unavailable at finish] **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
api_tst
mem_tracker

roborivers

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
comdb2sys_queueodh_generated [db unavailable at finish]
truncatesc_offline_generated **quarantined**
sc_resume_logicalsc_generated **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
replay_trans
mem_tracker

roborivers

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
truncatesc_offline_generated **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
replay_trans
mem_tracker
sc_truncate_lockorder_generated [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**

roborivers

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
sc_truncate_multiddl_generated [db unavailable at finish] **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
remsql_locks_rte_connect_generated **quarantined**
remsql_locks **quarantined**
mem_tracker
reco-ddlk-sql [timeout] **quarantined**

roborivers

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
logfill [db unavailable at finish] **quarantined**
scindex
sc_resume_logicalsc_generated **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
remsql_locks_rte_connect_generated **quarantined**
remsql_locks **quarantined**
mem_tracker
reco-ddlk-sql [timeout] **quarantined**

mohitkhullar · 2026-04-28T15:20:14Z

failing mem_tracker test

mohitkhullar · 2026-04-28T16:59:12Z

-    if (done_cb_evbuffer(clnt) != 0) {
-        return -1;
+    int rc = done_cb_evbuffer(clnt);
+    if (rc != 0) {


handle rc RC_INTERNAL_RETRY here?

This is handled in newsql_done_cb by returning without doing anything. I still think the failure could be a timing issue but I'm looking into it more.

roborivers

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
sc_resume_logicalsc_generated **quarantined**
sc_resume
cldeadlock
consumer_non_atomic_default_consumer_generated **quarantined**
remsql_locks_rte_connect_generated **quarantined**
remsql_locks **quarantined**
mem_tracker

roborivers

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
scindex_logicalsc_generated
sc_resume_logicalsc_generated **quarantined**
noresetgen
consumer_non_atomic_default_consumer_generated **quarantined**
remsql_locks_rte_connect_generated **quarantined**
remsql_locks **quarantined**
manual_partition
skipscan **quarantined**
reco-ddlk-sql [timeout] **quarantined**

akshatsikarwar · 2026-04-30T12:58:00Z

Please check robomark failures
cdb2test Apr 29 14:59:11 2026 failed feat/retry-pool.R20260429.6

roborivers

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
sc_redo [failed with core dumped]
logfill [db unavailable at finish] **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
remsql_locks_rte_connect_generated **quarantined**
remsql_locks **quarantined**
longreq_stats
truncatesc_offline_generated [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**

roborivers

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
sc_truncate [db unavailable at finish]
sc_resume_logicalsc_generated **quarantined**
reco-ddlk-sql **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
remsql_locks_rte_connect_generated **quarantined**
remsql_locks **quarantined**

emelialei88 · 2026-05-01T19:44:44Z

Please check robomark failures cdb2test Apr 29 14:59:11 2026 failed feat/retry-pool.R20260429.6

I think it was a github clone issue and the most recently robomark run is fine.

akshatsikarwar · 2026-05-04T12:37:08Z

I think it was a github clone issue and the most recently robomark run is fine.

It's still failing a few tests..
cdb2test Apr 30 18:05:43 2026 failed feat/retry-pool.R20260430.12

roborivers

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
sc_truncate [db unavailable at finish]
consumer_non_atomic_default_consumer_generated **quarantined**
reco-ddlk-sql [timeout] **quarantined**

roborivers

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
sc_resume [db unavailable at finish]
consumer_non_atomic_default_consumer_generated **quarantined**
remtran_origin
reco-ddlk-sql [timeout] **quarantined**

Signed-off-by: Emelia Lei <wlei29@bloomberg.net>

roborivers

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
logfill [db unavailable at finish] **quarantined**
scindex_logicalsc_generated
sc_resume
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**
truncatesc_offline_generated [timeout] **quarantined**

emelialei88 force-pushed the feat/retry-pool branch 2 times, most recently from 0a230d5 to 221983a Compare April 16, 2026 17:17

roborivers approved these changes Apr 16, 2026

View reviewed changes

emelialei88 force-pushed the feat/retry-pool branch 3 times, most recently from ad3a581 to f6fb6d5 Compare April 17, 2026 19:04

roborivers approved these changes Apr 17, 2026

View reviewed changes

emelialei88 force-pushed the feat/retry-pool branch 2 times, most recently from 3e7966d to 2412e0c Compare April 27, 2026 15:42

roborivers suggested changes Apr 27, 2026

View reviewed changes

roborivers approved these changes Apr 27, 2026

View reviewed changes

emelialei88 marked this pull request as ready for review April 27, 2026 20:03

mohitkhullar reviewed Apr 28, 2026

View reviewed changes

emelialei88 force-pushed the feat/retry-pool branch from 2412e0c to e265b77 Compare April 29, 2026 18:23

roborivers approved these changes Apr 29, 2026

View reviewed changes

roborivers suggested changes Apr 30, 2026

View reviewed changes

emelialei88 force-pushed the feat/retry-pool branch 2 times, most recently from e31f69e to fb958db Compare April 30, 2026 18:29

roborivers approved these changes Apr 30, 2026

View reviewed changes

emelialei88 force-pushed the feat/retry-pool branch from fb958db to 0583eff Compare May 4, 2026 20:53

roborivers suggested changes May 4, 2026

View reviewed changes

roborivers approved these changes May 5, 2026

View reviewed changes

dispatch retry sql to pool

4d66537

Signed-off-by: Emelia Lei <wlei29@bloomberg.net>

emelialei88 force-pushed the feat/retry-pool branch from 0583eff to 4d66537 Compare May 6, 2026 20:01

roborivers suggested changes May 6, 2026

View reviewed changes

Conversation

emelialei88 commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

mohitkhullar commented Apr 28, 2026

Uh oh!

mohitkhullar Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

emelialei88 Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

akshatsikarwar commented Apr 30, 2026

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

emelialei88 commented May 1, 2026

Uh oh!

akshatsikarwar commented May 4, 2026

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

roborivers left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

emelialei88 commented Apr 16, 2026 •

edited

Loading