Conversation
Before this change, daos_test invocations from suite.py/yaml relied on FI_LOG_LEVEL and D_LOG_STDERR_IN_LOG environment variables in an attempt to capture libfabric-level warnings. However, that configuration seems to be ineffective - i.e., it is seen in test results that no such warnings are saved in engine logs. With this change, HG_LOG_SUBSYS (with value including 'libfabric') and HG_LOG_LEVEL environnment variables are set to correctly capture libfabric warnings - e.g., to log entries like the following: mercury->libfabric [warning] ... rxm_handle_error() fi_eq_readerr ... Test-tag: daos_core_test Skip-unit-tests: true Skip-fault-injection-test: true Skip-test-rpms: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
|
Ticket title is 'daos_test/suite.py:DaosCoreTest.test_daos_rebuild_ec - pool reintegrate timeout' |
and is never invalidated on a transient SWIM death+restart. RPCs then take the cache-hit path and reuse a possibly-stale endpoint from the peer's previous incarnation, which can silently fail (e.g. a CORPC timing out to a just-restarted rank) with no CaRT-level indication of the cause. Add a diagnostic-only D_WARN, emitted when a peer's incarnation increases (i.e. it restarted) while a cached address for it is still present for ctx0/tag0. The check is read-only and does not change behavior. The incarnation increase can be observed on either of two paths, and which one records it first is a race: - crt_swim_set_member_state() (SWIM gossip), and - crt_swim_rank_check() (primary-group membership / rejoin update). The warning is emitted from both sites so it fires regardless of which path observes the restart first. The gossip-path check is gated on a real incarnation increase and a transition to ALIVE to keep it low-volume (once per restart, not per ping). This is intended to be combined with HG_LOG_SUBSYS/HG_LOG_LEVEL and the existing RPC timeout logs to correlate a stale-endpoint reuse at runtime. Test-tag: daos_core_test Skip-unit-tests: true Skip-fault-injection-test: true Skip-test-rpms: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this change, daos_test invocations from suite.py/yaml relied on FI_LOG_LEVEL and D_LOG_STDERR_IN_LOG environment variables in an attempt to capture libfabric-level warnings. However, that configuration seems to be ineffective - i.e., it is seen in test results that no such warnings are saved in engine logs.
With this change, HG_LOG_SUBSYS (with value including 'libfabric') and HG_LOG_LEVEL environnment variables are set to correctly capture libfabric warnings - e.g., to log entries like the following:
mercury->libfabric [warning] ... rxm_handle_error() fi_eq_readerr ...
Test-tag: daos_core_test
Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-test-rpms: true
Steps for the author:
After all prior steps are complete: