gh-155725: Remove PyGILState_Ensure usage from tracemalloc - #156409
Open
kumaraditya303 wants to merge 3 commits into
Open
gh-155725: Remove PyGILState_Ensure usage from tracemalloc#156409kumaraditya303 wants to merge 3 commits into
kumaraditya303 wants to merge 3 commits into
Conversation
tracemalloc no longer acquires the GIL nor creates a temporary thread state when tracing memory allocations. A thread with no attached thread state now records the trace with the "<unknown>" traceback instead of attaching a thread state to capture the Python traceback. Threads without a thread state used to pay for a GIL acquisition plus a full thread state creation and destruction on every traced raw allocation, only to record an empty traceback anyway.
Documentation build overview
|
…e attached Store traceback frame filenames as interned NUL terminated UTF-8 strings instead of Python str objects, so that capturing a traceback no longer uses or modifies Python objects. Threads without an attached thread state now capture their Python traceback by walking the frames of the thread state most recently bound to the thread; only threads which never had a thread state record the traceback as "<unknown>".
ZeroIntensity
left a comment
ZeroIntensity
left a comment
Member
There was a problem hiding this comment.
Thanks, this is a much better approach.
Comment on lines
+952
to
+954
| filename_obj = PyUnicode_DecodeUTF8(filename, | ||
| (Py_ssize_t)strlen(filename), | ||
| "surrogatepass"); |
Member
There was a problem hiding this comment.
This can call arbitrary Python code through the codec error handler, meaning this can encounter a re-entrancy deadlock with the tables lock. I'm okay with not fixing that since there's no reason an error handler should be invoking tracemalloc, but it's probably worth adding a comment.
Comment on lines
+420
to
+423
| if (_PyRuntimeState_GetFinalizing(&_PyRuntime) != NULL) { | ||
| // non-attached thread states can be cleared during finalization | ||
| return tracemalloc_empty_traceback; | ||
| } |
Member
There was a problem hiding this comment.
I think there's a subtle race condition here. The main thread can clear thread states after this check has passed.
You could try one of the new PyInterpreterView/PyInterpreterGuard APIs for this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tracemalloc no longer acquires the GIL nor creates a temporary thread state when tracing memory allocations.
tracemallocraw-domain allocator hook violatesPYMEM_DOMAIN_RAWcontract #155725