Skip to content

fibre_channel: fix crash when attempting to dereference invalid counters#3573

Merged
SuperQ merged 1 commit into
prometheus:masterfrom
eaibmz:collector-fibrechannel-crash-fix
Jul 15, 2026
Merged

fibre_channel: fix crash when attempting to dereference invalid counters#3573
SuperQ merged 1 commit into
prometheus:masterfrom
eaibmz:collector-fibrechannel-crash-fix

Conversation

@eaibmz

@eaibmz eaibmz commented Mar 5, 2026

Copy link
Copy Markdown
Contributor

The statistics counters of a disabled FC host cannot be read from sysfs and any read attempt fails with errno ENOENT. Therefore, all statistics counters returned by procfs are nil in such a case. This results in a crash on s390x Linux when a zFCP host is disabled with chzdev -d <host id>.

Crash stacktrace in a GDB

$ gdb /root/node_exporter/node_exporter
...
Thread 4 "node_exporter" received signal SIGSEGV, Segmentation fault. [Switching to LWP 35579]
0x00000000007be86c in github.com/prometheus/node_exporter/collector.(*fibrechannelCollector).Update (c=0xc00012fef0, ch=0xc0002fc8c0, ~r0=...)
    at /root/node_exporter/collector/fibrechannel_linux.go:133
133                     c.pushCounter(ch, "dumped_frames_total", *host.Counters.DumpedFrames, *host.Name)
(gdb) bt
    at /root/node_exporter/collector/fibrechannel_linux.go:133
(gdb) p host
$1 = {Name = 0xc00038faf0, Speed = 0xc00038fb00, PortState = 0xc00038fb10, PortType = 0xc00038fb30, SymbolicName = 0xc00038fba0, NodeName = 0xc00038fb40, PortID = 0xc00038fb50,
  PortName = 0xc00038fb60, FabricName = 0xc00038fb80, DevLossTMO = 0xc00038fb90, SupportedClasses = 0xc00038fbb0, SupportedSpeeds = 0xc00038fbd0, Counters = 0xc0001cb2d0}
(gdb) p host.Name
$2 = (string *) 0xc00038faf0
(gdb) p *host.Name
$3 = "host0"
(gdb) p *host.Counters.DumpedFrames
❌️ Cannot access memory at address 0x0
(gdb) p *host.Counters
$4 = {DumpedFrames = 0x0, ErrorFrames = 0x0, InvalidCRCCount = 0x0, RXFrames = 0x0, RXWords = 0x0, TXFrames = 0x0, TXWords = 0x0, SecondsSinceLastReset = 0x0, InvalidTXWordCount = 0x0,
  LinkFailureCount = 0x0, LossOfSyncCount = 0x0, LossOfSignalCount = 0x0, NosCount = 0x0, FCPPacketAborts = 0x0}
(gdb) bt
    at /root/node_exporter/collector/fibrechannel_linux.go:133
    name="fibrechannel", c=..., ch=0xc0002fc8c0, logger=0xc0002104a0)
    at /root/node_exporter/collector/collector.go:160
(gdb) p *host.Counters
$5 = {DumpedFrames = 0x0, ErrorFrames = 0x0, InvalidCRCCount = 0x0, RXFrames = 0x0,
  RXWords = 0x0, TXFrames = 0x0, TXWords = 0x0, SecondsSinceLastReset = 0x0,
  InvalidTXWordCount = 0x0, LinkFailureCount = 0x0, LossOfSyncCount = 0x0,
  LossOfSignalCount = 0x0, NosCount = 0x0, FCPPacketAborts = 0x0}
(gdb)

@eaibmz
eaibmz force-pushed the collector-fibrechannel-crash-fix branch 3 times, most recently from 5d36126 to 5bb3fb4 Compare March 5, 2026 07:53

@discordianfish discordianfish left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be fixed in procfs to return empty (non-nil) Counters

@eaibmz

eaibmz commented Mar 23, 2026

Copy link
Copy Markdown
Contributor Author

I think this should be fixed in procfs to return empty (non-nil) Counters

The PR prometheus/procfs#623 in particular introduced references to
to distinguish between non-existing values and 0 values.
I don't think we want it undone ?

@SuperQ

SuperQ commented Apr 3, 2026

Copy link
Copy Markdown
Member

Yes, this is the correct fix. But the tests are not passing and this needs a rebase.

@eaibmz
eaibmz force-pushed the collector-fibrechannel-crash-fix branch 3 times, most recently from 5db7558 to e8ef6a0 Compare April 15, 2026 10:04
@eaibmz
eaibmz requested a review from discordianfish April 16, 2026 08:05
@eaibmz
eaibmz force-pushed the collector-fibrechannel-crash-fix branch from e8ef6a0 to c2dc381 Compare April 22, 2026 06:36
@eaibmz
eaibmz force-pushed the collector-fibrechannel-crash-fix branch from c2dc381 to 781a86b Compare July 15, 2026 05:05
@eaibmz

eaibmz commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

polite bump

@SuperQ

SuperQ commented Jul 15, 2026

Copy link
Copy Markdown
Member

Need to update collector/fixtures/e2e-64k-page-output.txt to match the new output.

The statistics counters of a disabled FC host cannot be read from sysfs
and any read attempt fails with errno ENOENT. Therefore, all statistics
counters returned by procfs are nil in such a case. This results in
a crash on s390x Linux when a zFCP host is disabled with
`chzdev -d <host id>`.

Crash stacktrace in a GDB
-------------------------

$ gdb /root/node_exporter/node_exporter
...
Thread 4 "node_exporter" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 35579]
0x00000000007be86c in github.com/prometheus/node_exporter/collector.(*fibrechannelCollector).Update (c=0xc00012fef0, ch=0xc0002fc8c0, ~r0=...)
    at /root/node_exporter/collector/fibrechannel_linux.go:133
133                     c.pushCounter(ch, "dumped_frames_total", *host.Counters.DumpedFrames, *host.Name)
(gdb) bt
    at /root/node_exporter/collector/fibrechannel_linux.go:133
(gdb) p host
$1 = {Name = 0xc00038faf0, Speed = 0xc00038fb00, PortState = 0xc00038fb10, PortType = 0xc00038fb30, SymbolicName = 0xc00038fba0, NodeName = 0xc00038fb40, PortID = 0xc00038fb50,
  PortName = 0xc00038fb60, FabricName = 0xc00038fb80, DevLossTMO = 0xc00038fb90, SupportedClasses = 0xc00038fbb0, SupportedSpeeds = 0xc00038fbd0, Counters = 0xc0001cb2d0}
(gdb) p host.Name
$2 = (string *) 0xc00038faf0
(gdb) p *host.Name
$3 = "host0"
(gdb) p *host.Counters.DumpedFrames
❌️ Cannot access memory at address 0x0
(gdb) p *host.Counters
$4 = {DumpedFrames = 0x0, ErrorFrames = 0x0, InvalidCRCCount = 0x0, RXFrames = 0x0, RXWords = 0x0, TXFrames = 0x0, TXWords = 0x0, SecondsSinceLastReset = 0x0, InvalidTXWordCount = 0x0,
  LinkFailureCount = 0x0, LossOfSyncCount = 0x0, LossOfSignalCount = 0x0, NosCount = 0x0, FCPPacketAborts = 0x0}
(gdb) bt
    at /root/node_exporter/collector/fibrechannel_linux.go:133
    name="fibrechannel", c=..., ch=0xc0002fc8c0, logger=0xc0002104a0)
    at /root/node_exporter/collector/collector.go:160
(gdb) p *host.Counters
$5 = {DumpedFrames = 0x0, ErrorFrames = 0x0, InvalidCRCCount = 0x0, RXFrames = 0x0,
  RXWords = 0x0, TXFrames = 0x0, TXWords = 0x0, SecondsSinceLastReset = 0x0,
  InvalidTXWordCount = 0x0, LinkFailureCount = 0x0, LossOfSyncCount = 0x0,
  LossOfSignalCount = 0x0, NosCount = 0x0, FCPPacketAborts = 0x0}
(gdb)

Signed-off-by: Alexander Egorenkov <eaibmz@gmail.com>
@eaibmz
eaibmz force-pushed the collector-fibrechannel-crash-fix branch from 781a86b to 571a1da Compare July 15, 2026 10:27
@SuperQ
SuperQ requested review from discordianfish and removed request for discordianfish July 15, 2026 12:57
@SuperQ
SuperQ merged commit 366f908 into prometheus:master Jul 15, 2026
17 checks passed
@eaibmz

eaibmz commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants