Skip to content

collector: add nvmesubsystem collector for NVMe-oF path health#3579

Open
sradco wants to merge 2 commits intoprometheus:masterfrom
sradco:add_collector_multipath
Open

collector: add nvmesubsystem collector for NVMe-oF path health#3579
sradco wants to merge 2 commits intoprometheus:masterfrom
sradco:add_collector_multipath

Conversation

@sradco
Copy link
Copy Markdown

@sradco sradco commented Mar 11, 2026

Summary

Add a new disabled-by-default nvmesubsystem collector that reads /sys/class/nvme-subsystem/ to expose NVMe over Fabrics subsystem path health metrics.

Exposed metrics

Metric Type Labels Description
node_nvmesubsystem_info Gauge (info) subsystem, nqn, model, serial, iopolicy Subsystem identity
node_nvmesubsystem_namespace_info Gauge (info) subsystem, device Maps namespace block device to subsystem
node_nvmesubsystem_paths Gauge subsystem Total controller paths
node_nvmesubsystem_paths_live Gauge subsystem Controller paths in live state
node_nvmesubsystem_path_state Gauge subsystem, controller, transport, state Per-controller state

The namespace_info metric enables precise correlation between NVMe block devices (e.g. nvme0n1) and their parent subsystems, which is needed for workload-aware storage path health alerting in Kubernetes environments.

Enable

--collector.nvmesubsystem

No special permissions required — reads only world-readable sysfs attributes.

Dependencies

Test plan

  • Unit tests for metrics emission
  • Tests for no-devices case
  • Shared test helpers extracted to avoid collision with dmmultipath tests

@sradco sradco force-pushed the add_collector_multipath branch from 742c1b1 to a0a146e Compare March 11, 2026 19:18
@sradco
Copy link
Copy Markdown
Author

sradco commented Mar 11, 2026

Hi @SuperQ , I created this PR for a new multipath collector.
I would appreciate your review.

@sradco sradco force-pushed the add_collector_multipath branch from a0a146e to 1fa2099 Compare March 12, 2026 08:55
@sradco sradco changed the title Add multipath collector Add multipath collector for NVMe-oF subsystem path health Mar 12, 2026
@sradco sradco force-pushed the add_collector_multipath branch from 1fa2099 to 635b613 Compare March 12, 2026 09:13
@sradco sradco changed the title Add multipath collector for NVMe-oF subsystem path health collector: add nvmesubsystem collector for NVMe-oF path health Mar 12, 2026
Copy link
Copy Markdown

@jsafrane jsafrane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good to me. I tested it with 2 NVMe over TCP devices, I got:

node_nvmesubsystem_info{iopolicy="numa",model="Linux",nqn="tempdisk",serial="bab529a0f32e397e1319",subsystem="nvme-subsys0"} 1
node_nvmesubsystem_path_state{controller="nvme0",state="connecting",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme0",state="dead",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme0",state="live",subsystem="nvme-subsys0",transport="tcp"} 1
node_nvmesubsystem_path_state{controller="nvme0",state="resetting",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme0",state="unknown",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme1",state="connecting",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme1",state="dead",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme1",state="live",subsystem="nvme-subsys0",transport="tcp"} 1
node_nvmesubsystem_path_state{controller="nvme1",state="resetting",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme1",state="unknown",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_paths_live{subsystem="nvme-subsys0"} 2
node_nvmesubsystem_paths_total{subsystem="nvme-subsys0"} 2

Which looks reasonable.

Comment on lines +64 to +70
switch raw {
case "live", "connecting", "resetting", "dead":
return raw
case "deleting", "deleting (no IO)", "new":
return raw
default:
return "unknown"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the record, I checked that this is a complete list of all states reported by the kernel today.

@sradco sradco force-pushed the add_collector_multipath branch 3 times, most recently from 51fb644 to 8acec17 Compare March 16, 2026 10:15
@sradco sradco force-pushed the add_collector_multipath branch 2 times, most recently from 17875ea to d7d2b27 Compare March 16, 2026 17:33
@sradco sradco closed this Mar 17, 2026
@sradco sradco force-pushed the add_collector_multipath branch from d7d2b27 to 1a4cac6 Compare March 17, 2026 08:42
@sradco sradco reopened this Mar 17, 2026
@sradco
Copy link
Copy Markdown
Author

sradco commented Mar 17, 2026

PR is failing since its based on prometheus/procfs#797, which is still not merged

sysfs metrics

Add a new disabled-by-default collector that reads
/sys/block/dm-* to discover Device Mapper multipath
devices and expose path health metrics. Multipath
devices are identified by checking that dm/uuid
starts with "mpath-", which distinguishes them from
LVM or other DM device types.

The path state is reported as-is from
/sys/block/<dev>/device/state, supporting both
SCSI devices (running, offline, blocked, etc.) and
NVMe devices (live, connecting, dead, etc.) without
hardcoding a fixed set of states.

All device-level metrics include both the DM
friendly name (device) and the kernel block device
name (sysfs_name, e.g. dm-0) to enable direct
correlation with node_disk_* I/O metrics without
recording rules.

No special permissions are required — the collector
reads only world-readable sysfs attributes.

Exposed metrics:
- node_dmmultipath_device_info
- node_dmmultipath_device_active
- node_dmmultipath_device_size_bytes
- node_dmmultipath_device_paths
- node_dmmultipath_device_paths_active
- node_dmmultipath_device_paths_failed
- node_dmmultipath_path_state

Signed-off-by: Shirly Radco <sradco@redhat.com>
Co-authored-by: AI Assistant <noreply@cursor.com>
Add a new disabled-by-default collector that reads
/sys/class/nvme-subsystem/ to expose NVMe over Fabrics subsystem
path health metrics.

Exposed metrics:
- node_nvmesubsystem_info: subsystem identity (NQN, model, serial, iopolicy)
- node_nvmesubsystem_namespace_info: maps namespace device to subsystem
- node_nvmesubsystem_paths: total controller paths per subsystem
- node_nvmesubsystem_paths_live: live controller paths per subsystem
- node_nvmesubsystem_path_state: per-controller state (live, connecting, dead, etc.)

The namespace_info metric enables precise correlation between NVMe
block devices (e.g. nvme0n1) and their parent subsystems for
workload-aware storage path health alerting.

Enable with --collector.nvmesubsystem

Depends on: prometheus/procfs#797

Signed-off-by: Shirly Radco <sradco@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@sradco sradco force-pushed the add_collector_multipath branch from 831958f to b614bb3 Compare May 4, 2026 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants