Fix duplicate Celery logs via after_setup_logger - #347
Merged
cahillsf merged 4 commits intoJul 24, 2026
Conversation
Celery's worker log setup hijacks the root logger and attaches its own plain-text handler. Since Flask's app.logger propagates to root, every current_app.logger call (e.g. report_revoked_task) was emitted twice — once through Lemur's structured JSON handler, once through Celery's plain-text root handler that Datadog can't parse a level out of. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Celery's worker log setup hijacks the root logger. Because app.logger propagates by default, every current_app.logger call inside a task was emitted twice - once through Lemur's own handler and once through Celery's root handler. Set app.logger.propagate = False in configure_logging instead of disabling Celery's root-logger hijack (worker_hijack_root_logger). This is more targeted: it stops Lemur's records from being re-emitted through root while leaving Celery's own task logging untouched. Also add a Celery worker to the local supervisor config so the behaviour can be reproduced in local dev. CLOUDR-2097 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cahillsf
changed the title
Author
|
deployed in staging at DEBUG log level, can see here with latest master image we get ~2x the logs from the celery-worker before:
after:
|
app.logger.propagate = False kept Lemur's own unformatted StreamHandler output (which Datadog's agent was mangling into mislabeled, duplicated multiline entries) and dropped the clean copy that flows through Celery's root-hijacked handler. Instead, drop app.logger's own handlers once Celery actually sets up its root logger (via the after_setup_logger signal), so only the clean, propagated copy remains. This only fires during real worker/beat startup - not when the web process lazily imports this module to submit a task. CLOUDR-2097 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cahillsf
changed the title
cahillsf
commented
Jul 24, 2026
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cahillsf
marked this pull request as ready for review
July 24, 2026 13:08
maperu
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stops Celery task logs from being written twice.
make_celeryclearsapp.logger's own handlers once Celery sets up its root logger, leaving only the clean copy Celery's hijacked root handler emits.Fixes https://datadoghq.atlassian.net/browse/CLOUDR-2132.
Why
Celery's worker log setup hijacks the root logger. Since
app.loggerpropagates by default, everycurrent_app.loggercall inside a task was emitted twice: once through Lemur's bare, unformattedstream_handler(which Datadog's agent mangled into mislabeled, multiline-duplicated entries), and once cleanly through Celery's hijacked root handler.Pulled actual pre-fix log lines from staging Datadog to confirm which copy to keep — Lemur's own handler was the messy one; Celery's was clean and correctly leveled.
Approach
make_celeryconnects anafter_setup_loggerreceiver that clearsapp.logger's handlers, leaving the propagated/Celery-formatted copy as the sole output. This signal only fires during real worker/beat startup, not when the web process lazily importslemur.common.celeryto submit a task — so web logging is untouched.Changes
lemur/common/celery.py— clearapp.loggerhandlers viaafter_setup_logger.lemur/tests/test_factory.py,lemur/tests/test_dcv_celery.py— updated/new coverage.local/supervisor.conf— local Celery worker to reproduce this in dev.Testing
kube_namespace:cert-orchestration source:celery-worker).make test-pythonpasses.