Skip to content

GPU Workflow: Fix possible deadlock with DoublePipeline when not receiving EndOfStream#15634

Open
davidrohr wants to merge 2 commits into
AliceO2Group:devfrom
davidrohr:PR
Open

GPU Workflow: Fix possible deadlock with DoublePipeline when not receiving EndOfStream#15634
davidrohr wants to merge 2 commits into
AliceO2Group:devfrom
davidrohr:PR

Conversation

@davidrohr

Copy link
Copy Markdown
Collaborator

No description provided.

@alibuild

Copy link
Copy Markdown
Collaborator

Error while checking build/O2/fullCI_slc9 for 9992aee at 2026-07-25 13:52:

## sw/BUILD/O2-full-system-test-latest/log
command o2-sim-digitizer-workflow -n 5 --simPrefixQED qed/o2sim --qed-x-section-ratio 4477.35 --disable-mc --sims sig,o2sim --tpc-lanes 24 --shm-segment-size 8000000000 -b --skipDet TRD --configKeyValues "HBFUtils.nHBFPerTF=128;HBFUtils.orbitFirst=256;HBFUtils.orbitFirstSampled=256;HBFUtils.obligatorySOR=false;HBFUtils.runNumber=310000;;HBFUtils.maxNOrbits=384;" --interactionRate 50000 --early-forward-policy always --incontext collisioncontext.root had nonzero exit code 1
[3428:internal-dpl-ccdb-backend]: [13:52:28][ERROR] CCDBDownloader CURL transfer error - Timeout was reached
[3428:internal-dpl-ccdb-backend]: [13:52:28][ERROR] CcdbDownloader finished transfer http://alice-ccdb.cern.ch/CTP/Calib/OrbitReset for 1550600800000 (agent_id: o2-sim-digitize/o2-sim-digitizer-workflow@3428 target/Linux_x86_64 init/1784980337786 id/mesosci@alimetal07.cern.ch session/8FQvbqNs) with http code: 0
[3428:internal-dpl-ccdb-backend]: [13:52:28][ERROR] File CTP/Calib/OrbitReset could not be retrieved. No more hosts to try.
[3428:internal-dpl-ccdb-backend]: [13:52:28][FATAL] Unable to find CCDB object CTP/Calib/OrbitReset/1550600800000
[3428:internal-dpl-ccdb-backend]: [13:52:31][ERROR] Exception while running: Fatal error. Rethrowing.
[3428:internal-dpl-ccdb-backend]: [13:52:31][FATAL] Unhandled o2::framework::runtime_error reached the top of main of o2-sim-digitizer-workflow, device shutting down. Reason: Fatal error
[ERROR] Workflow crashed - PID 3428 (internal-dpl-ccdb-backend) did not exit correctly however it's not clear why. Exit code forced to 128.
[ERROR]  - Device internal-dpl-ccdb-backend: pid 3428 (exit 128)
[INFO]    - First error: [13:52:28][FATAL] Unable to find CCDB object CTP/Calib/OrbitReset/1550600800000
[ERROR] SEVERE: Device internal-dpl-ccdb-backend (3428) had at least one message above severity 7: Unable to find CCDB object CTP/Calib/OrbitReset/1550600800000


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
grep: error-log.txt: binary file matches
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/6fd142f4bdaeff4be273b40f055e9850b5207227/slc9_x86-64/o2checkcode/1.0-local201/etc/modulefiles
++ alibuild-generate-module
+ HAS_BIN=
+ HAS_LIB=
+ HAS_CMAKE=
+ HAS_ROOT=
+ '[' 0 '!=' 0 ']'
+ cat
+ printf 'if ![ is-loaded '\''BASE/1.0'\'' ] {\n module load BASE/1.0\n}'
+ echo 'BUILD_REQUIRES=CMake alibuild-recipe-tools defaults-release'
BUILD_REQUIRES=CMake alibuild-recipe-tools defaults-release
+ FULL_BUILD_REQUIRES='Alice-GRID-Utils googlebenchmark FairCMakeModules Python-modules-list MPFR safe_int CMake ninja UUID alibuild-recipe-tools O2-customization defaults-release json-c ninja-fortran date pytorch_cpuinfo bz2 double-conversion'
+ echo 'FULL_BUILD_REQUIRES=Alice-GRID-Utils googlebenchmark FairCMakeModules Python-modules-list MPFR safe_int CMake ninja UUID alibuild-recipe-tools O2-customization defaults-release json-c ninja-fortran date pytorch_cpuinfo bz2 double-conversion'
FULL_BUILD_REQUIRES=Alice-GRID-Utils googlebenchmark FairCMakeModules Python-modules-list MPFR safe_int CMake ninja UUID alibuild-recipe-tools O2-customization defaults-release json-c ninja-fortran date pytorch_cpuinfo bz2 double-conversion
++ env
++ cut -f1 -d=
++ grep -v '^DEFAULT_'
++ grep -v PKGREVISION
++ grep REVISION
++ grep -v ALIBUILD_RECIPE_TOOLS
++ sed -e s/_REVISION//
+ for x in $(env | cut -f1 -d= | grep -v "^DEFAULT_" | grep -v PKGREVISION | grep -v ALIBUILD_RECIPE_TOOLS | grep REVISION | sed -e 's/_REVISION//')
++ eval 'echo $LIBFFI_REVISION'
+++ echo 10
+ REVISION_VALUE=10
++ eval 'echo $LIBFFI_VERSION'
[0 more errors; see full log]

Full log here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants