KVM hosts are being reset by a script

[khmarochos]

ISSUE TYPE

Bug Report

COMPONENT NAME

KVM

CLOUDSTACK VERSION

All 4.x and 2.x versions seem to be affected.

CONFIGURATION

N/A

OS / ENVIRONMENT

This issue is related to KVM hosts. As far as I can see, XEN hosts are affected too.

SUMMARY

KVM hosts are being rebooted by kvmheartbeat.sh when the NFS-server that is being used as a primary storage becomes unreachable.

I'd like to propose change behavior of kvmheartbeat.sh, as it's not necessary to reboot a whole host when only one of VMs is having difficulties while other VMs are working fine with other storages.

STEPS TO REPRODUCE

It's easy to reproduce: just deploy a VM with a volume on an NFS-based storage, break NFS and wait.

EXPECTED RESULTS

The KVM host will be reset.

ACTUAL RESULTS

The kvmscript.sh will reboot the server.

[serverchief]

As of 4.11 CloudStack has an enhancement for KVM HA. Instead of relying on shell script and specific primary store - we will check the write time stamp on the disks for all VMs that are known to reside on hypervisor.

All VM disks/files must not update the write timestamp for an extended amount of time(configurable) before a specifc action is taken (configurable). Even If a single vm disk will have a write timestamp and all other don’t - we will not power off the host.

You can read more about KVM HA on cloudstack homepage, there is also a video talk by Rohit Yadav that explains it in great details.

[khmarochos]

Thank you! I didn't notice that 4.11.x have it fixed, in 4.9.2 it still can reset a host.

In any case, the script and echo b > /proc/sysrq-trigger is still present. Won't it be executed at some moment? Looks a bit dangerous, as for me.

[wido]

The main problem is that it is hard to determine if the host isn't responding or that something else is going on.

You do not want the VM to run twice as that might (will!) cause data corruption. That's why a hard reboot is being issued if the storage is having trouble.

[khmarochos]

Thank you for notation.

That's exactly why I proposed (d821625) to find the VMs using the mountpoint (by lsof and awk) and then kill their qemu-kvm processes.

[Slair1]

Please see PR #2722 and #2472
I agree and created these PRs to address the built-in reboot

[DaanHoogland]

long time no activity, closing. please reopen @Melnik13 if you think i'm at err