VR healthcheck wrong state #12630
Description
problem
The listRouters API-call always returns healthcheckfailed false, even if the healthcheck fails.
Alert in CloudStack UI:
Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16
Error in EventQueue
{"eventDateTime":"2026-02-09 14:01:22 +0100","subject":"Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16","dataCenterId":"a8f95837-c05c-4a97-b100-53f87de8df24","event":"ALERT.SERVICE.DOMAINROUTER","body":"Health checks failed: 1 failing checks on router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16","podId":"090ccb90-9411-4bab-bb0c-f5ab61e34ebb"}
{"eventDateTime":"2026-02-09 14:01:22 +0100","entityuuid":"c0c6a15f-3ad8-4936-ba6e-abe802ddba16","description":"Router r-8-VM / c0c6a15f-3ad8-4936-ba6e-abe802ddba16 has failing checks: router_version_check.py","event":"ROUTER.HEALTH.CHECKS","user":"c302beae-cb9d-11f0-a957-fa163ed04c4b","account":"c302a9b2-cb9d-11f0-a957-fa163ed04c4b","entity":"DomainRouter","status":"Completed"}
API call via cmk
(cloudstackmsu) 🐄 > list routers name=r-8-VM filter=name,hostname,healthchecksfailed,healthcheckresults,state, fetchhealthcheckresults=true listall=true
router:
count = 1
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
| NAME | HOSTNAME | HEALTHCHECKSFAILED | HEALTHCHECKRESULTS | STATE |
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
| r-8-VM | srv1-214 | false | [{"checkname":"connectivity.test","checktype":"basic","details":"Failed to fetch results with details: java.io.IOException: Stream closed\n\tat | Running |
| | | | java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)\n\tat java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)\n\tat java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)\n\tat java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)\n\tat | |
| | | | java.base/java.io.BufferedReader.fill(BufferedReader.java:162)\n\tat java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)\n\tat | |
| | | | java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)\n\tat com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)\n\tat | |
| | | | com.cloud.utils.script.Script.execute(Script.java:336)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:632)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:620)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:431)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:403)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:239)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:130)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)\n\tat | |
| | | | com.cloud.agent.Agent.processRequest(Agent.java:813)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)\n\tat | |
| | | | com.cloud.utils.nio.Task.call(Task.java:83)\n\tat com.cloud.utils.nio.Task.call(Task.java:29)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat | |
| | | | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat | |
| | | | java.base/java.lang.Thread.run(Thread.java:840)\n","lastupdated":"2026-02-11T13:01:46+0100","status":"FAILED","success":false},{"checkname":"filesystem.writable.test","checktype":"basic","details":"Failed | |
| | | | to fetch results with details: java.io.IOException: Stream closed\n\tat java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)\n\tat | |
| | | | java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)\n\tat java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)\n\tat | |
| | | | java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)\n\tat java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)\n\tat | |
| | | | java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)\n\tat java.base/java.io.BufferedReader.fill(BufferedReader.java:162)\n\tat | |
| | | | java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)\n\tat java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)\n\tat | |
| | | | com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)\n\tat com.cloud.utils.script.Script.execute(Script.java:336)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:632)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:620)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:431)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:403)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:239)\n\tat | |
| | | | com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:130)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat | |
| | | | com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)\n\tat | |
| | | | com.cloud.agent.Agent.processRequest(Agent.java:813)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)\n\tat | |
| | | | com.cloud.utils.nio.Task.call(Task.java:83)\n\tat com.cloud.utils.nio.Task.call(Task.java:29)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat | |
| | | | java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat | |
| | | | java.base/java.lang.Thread.run(Thread.java:840)\n","lastupdated":"2026-02-11T13:01:46+0100","status":"UNKNOWN","success":false}] | |
+--------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+
Expected result: healthcheckfailed=true instead of healthcheckfailed=false
versions
ACS 4.22.0
The steps to reproduce the bug
- ssh into one VR VM
- shutdown the interfaces e.g.
ifdown eth0 ; ifdown eth1 - Wait about 10 minutes
- Check Alerts in the UI
- Check
cmk list routers filter=name,hostname,healthchecksfailed,healthcheckresults,state, fetchhealthcheckresults=true listall=true
What to do about it?
No response