logger: apply a log config file to the live logger without a restart

[cosmin-staicu]

Set LK_LOG_CONFIG_PATH to a YAML file and the logger applies its level and component_levels to the running process, so a log level can change without a restart.

Why

The parts for this are already in logger. The root level and every component level are zap.AtomicLevel, the root ComponentLeveler resolves them through Config, and FromZapLogger registers a Config observer that refreshes the leveler. Nothing ever changed a level on a running process, though, so a level change still meant restarting it and losing the state you wanted to look at.

What

WatchConfigFile polls the file every 30s (LK_LOG_CONFIG_INTERVAL overrides that) and applies it to the Config the service holds. The hook is in FromZapLogger, which every consumer reaches (livekit-server through InitFromConfig, livekit-sip through NewZapLogger), so no binary needs its own flag or call site. It starts once, for the first logger the process builds. With the env var unset, nothing changes.

It polls instead of using fsnotify because the usual target is a mounted ConfigMap: kubelet swaps the ..data symlink rather than rewriting the file, so a watch on the file never fires.

Only level and component_levels are taken. The rest of the logging block (json, sample and the sampler settings) is read once when a logger is built, so changing it on a running process would do nothing. The file is decoded strictly, and a file that names any other key is rejected with a warning instead of being half applied.

The file is an overlay on the startup config, not on whatever was applied last. WatchConfigFile snapshots the config once at startup and applies every file over that baseline. A level the file omits falls back to its startup value, and emptying the file to {} restores the levels the process booted with. component_levels entries merge over the startup ones, so an entry the file doesn't name, such as the pion_level livekit-server sets, keeps its startup value, and an entry that disappears from the file stops applying.

Worth a look

Levels are written through a new Config.updateLevels rather than Config.Update. Update assigns every field, so applying a partial file through it would reset the rest, and it would rewrite the item sampler settings that WithItemSampler reads without Config's lock. updateLevels sets the two level fields under the lock and notifies the same observers. The read side needs no change: the leveler resolves through Config.ResolveComponentLevel, which takes the same lock.

An unreadable file (an optional ConfigMap that isn't mounted yet), unchanged bytes and a rejected file all leave the config in force alone. A rejected file is reported once when it appears, not on every poll. A file that goes away isn't treated as a reset, because a transient read error would then flap levels on a live process. To reset, empty the file to {}.

Tests are in logger/configwatch_test.go.

[changeset-bot]

🦋 Changeset detected

Latest commit: 907d9f7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
github.com/livekit/protocol	Patch
@livekit/protocol	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR