Fencing in Libvirt/KVM virtualized cluster nodes

Often, people deploy the Pacemaker stack in virtual environments for purposes of testing and evaluation. In such environments, it's easy to test Pacemaker's fencing capabilities by tying in with the hypervisor.

This quick howto illustrates how to configure fencing for two virtual cluster nodes hosted on a libvirt/KVM hypervisor host.

libvirt configuration (hypervisor)

In order to do libvirt fencing, your hypervisor should have its libvirtd daemon listen on a network socket. libvirtd is capable of doing this, both on an encrypted TLS socket, and on a regular, unencrypted TCP port. Needless to say, for production use you should only use TLS, but for testing and evaluation – and for that purpose only – TCP is fine.

In order for your hypervisor to listen on an unauthenticated, insecure, unencrypted network socket (did we mention that's unsuitable for production?), add the following lines to your libvirtd configuration file:

listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
auth_tcp = "none"

You can also set the listen_addr parameter, for example to have libvirtd listen only on the network that your virtual machines run in. If you don't set listen_addr, libvirtd will simply listen on the wildcard address.

You'll also have to add the -l or --listen flag to your libvirtd invocation. On Debian/Ubuntu platforms, you can do so by editing the /etc/default/libvirt-bin configuration file.

Once you've done that, you can use netstat -ltp to check whether libvirtd is in fact listening on its configured port, 16509/tcp. Also, make sure that you don't have a firewall blocking that port.

libvirt configuration (virtual machines)

Inside your virtual machines, you'll also have to install the libvirt client binaries – the fencing mechanism uses the virsh utility under the covers. Some platforms provide a libvirt-client package for that purpose; for other's, you'll simply have to install the full libvirt package.

Once that is set up, you should be able to run this command from inside your virtual machines:

virsh --connect=qemu+tcp://<IP of your hypervisor>/system \
  list --all

... and that command should list all the domains running on that host, including the one you're connecting from.

Pacemaker configuration

In one of your virtual machines, you can now set up your fencing configuration.

This example assumes that you have two nodes named alice and bob, that their corresponding virtual machine domain names are also alice and bob, and that they can reach their hypervisor by TCP at 192.168.0.1:

primitive p_fence_alice stonith:external/libvirt \
  params hostlist="alice" \
   hypervisor_uri="qemu+tcp://192.168.0.1/system" \
  op monitor interval="60"
primitive p_fence_bob stonith:external/libvirt \
  params hostlist="bob" \
    hypervisor_uri="qemu+tcp://192.168.0.1/system" \
  op monitor interval="60"
location l_fence_alice p_fence_alice -inf: alice
location l_fence_bob p_fence_bob -inf: bob
property stonith-enabled=true

Now you can test fencing to the best of your abilities.

Comments

Where can I find stonith:external/libvirt?

Hi.  I'd love to explore this alternative to fence_virtd - I have VM guests on multiple physical hosts and the documentation for that scenario - using a backend of libvirt-qpid? - seems to be scanty, but I'm puzzled as to where the resource you reference here, 'stonith:external/libvirt', can be found.

I'm running Redhat Enterprise Linux 6.2 and I've installed all of the various libvirt and fence_virt* RPM packages.  When I do a 'stonith_admin --list-installed' command I don't see any 'external/libvirt' - or just 'libvirt' on its own - listed.

Similarly when I use crmsh to run the command 'crm ra list stonith' I don't see your 'external/libvirt' mentioned anywhere.

I'm probably betraying ignorance of special semantics attached to the 'external/' qualifier for a resource name or some other magic I haven't come across yet in my research into Pacemaker.

In any case I'd appreciate very much advice on how your resource agent works or where it is found, or pointers to "what external/ means in Pacemaker Resource Agents" :) if you could be so kind.

Thanks,  Brad.

STONITH agents in CentOS and elsewhere

Pacemaker supports both its own fencing API (dating back to the Heartbeat days) and that which originated with Red Hat Cluster. Depending on which platform you're on, Pacemaker ships with different fencing agents:

  • SUSE: agents from the cluster-glue package only (including the external/* agents mentioned here);
  • RHEL/CentOS/Fedora: agents from Red Hat Cluster only (those are the fence_* agents from the fence-agents package you're seeing);
  • Recent Debian/Ubuntu versions: both (in cluster-glue and fence-agents, respectively).

Thus in your setup, fence_virsh is really the correct agent to use; external/libvirt simply isn't available.