RHEL 7 – How to configure the Fencing on Pacemaker ?

Fencing (STONITH) is an important mechanism in cluster to avoid the data corruption on shared storage. It also helps to bring the cluster into the known state when there is a split brain occurs between the nodes. Cluster nodes talks to each other over communication channels, which are typically standard network connections, such as Ethernet. Each resources and nodes have “state” (Ex: started , stopped) in the cluster and nodes report every changes that happens on resources. This reporting works well until communication breaks between the nodes. Fencing  will come to play when nodes can’t communicate with each other. Majority of nodes will form the cluster based on quorum votes and rest of the nodes will be rebooted or halted based on fencing actions what we have denied.

 

There are two type of fencing available in pacemaker.

• Resource Level Fencing
• Node Level Fencing

Using the resource level fencing, the cluster can make sure that a node cannot access same resources on both the nodes. The node level fencing makes sure that a node does not run any resources at all. This is usually done in a very simple, yet brutal way: the node is simply reset using a power switch. This may ultimately be necessary because the node may not be responsive at all.  In Pacamaker/corosync cluster, we will call the fencing method as “STONITH”  (Shoot The Other Node In The Head).

 

For more information , please visit clusterlabs.org . Here we will see the node level fencing.

 

Have a look at the cluster setup.

[root@Node1-LAB ~]# pcs status
Cluster name: GFSCLUS
Last updated: Wed Jan 20 12:43:36 2016
Last change: Wed Jan 20 09:57:06 2016 via cibadmin on Node1
Stack: corosync
Current DC: Node1 (1) - partition with quorum
Version: 1.1.10-29.el7-368c726
2 Nodes configured
2 Resources configured


Online: [ Node1 Node2 ]

PCSD Status:
  Node1: Online
  Node2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@Node1-LAB ~]#

 

In this article ,we will see that how to configure stonith/fencing  “fence_xvm”  for KVM cluster nodes. The purpose of this setup is to demonstrate the STONITH/FENCING.

 

Environment:  (Demo Purpose only)

• Node 1 & Node 2  – Pacemaker/corosync cluster
• UNIXKB-CP  – KVM host which hosts Node1 & Node2

 

Configure KVM host to use fence_xvm:

1.Login to the KVM host.

2.List the running virtual Machines .

[root@UNIXKB-CP ~]# virsh list
 Id    Name                           State
----------------------------------------------------
 6     Node1                          running
 7     Node2                          running

[root@UNIXKB-CP ~]#

 

3.Install the required fencing packages on KVM host (Non-Cluster node)

[root@UNIXKB-CP ~]# yum install fence-virt fence-virtd fence-virtd-libvirt fence-virtd-multicast fence-virtd-serial
Loaded plugins: langpacks, product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Package fence-virt-0.3.0-16.el7.x86_64 already installed and latest version
Package fence-virtd-0.3.0-16.el7.x86_64 already installed and latest version
Package fence-virtd-libvirt-0.3.0-16.el7.x86_64 already installed and latest version
Package fence-virtd-multicast-0.3.0-16.el7.x86_64 already installed and latest version
Package fence-virtd-serial-0.3.0-16.el7.x86_64 already installed and latest version
Nothing to do
[root@UNIXKB-CP ~]#

 

4. Create the new directory to store the fence key. Create the random key to use for fencing.

[root@UNIXKB-CP ~]# mkdir -p /etc/cluster
[root@UNIXKB-CP ~]# cd /etc/cluster/
[root@UNIXKB-CP cluster]# dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=4k count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000506736 s, 8.1 MB/s
[root@UNIXKB-CP cluster]#

 

5. Copy the fence keys to cluster nodes. (Node1 & Node2)

[root@UNIXKB-CP cluster]# scp -r /etc/cluster/fence_xvm.key root@Node1:/etc/cluster/fence_xvm.key
root@node1's password:
fence_xvm.key                                                                                                                      100% 4096     4.0KB/s   00:00
[root@UNIXKB-CP cluster]# scp -r /etc/cluster/fence_xvm.key root@Node2:/etc/cluster/fence_xvm.key
root@node2's password:
fence_xvm.key                                                                                                                      100% 4096     4.0KB/s   00:00
[root@UNIXKB-CP cluster]#

Note: You must create a “/etc/cluster” directory on the cluster nodes in an order to copy the xvm keys.

 

6.Use “fence_virtd -c” command to create “/etc/fence_virt.conf” file.

[root@UNIXKB-CP ~]# fence_virtd -c
Module search path [/usr/lib64/fence-virt]:

Available backends:
    libvirt 0.1
Available listeners:
    multicast 1.2

Listener modules are responsible for accepting requests
from fencing clients.

Listener module [multicast]:

The multicast listener module is designed for use environments
where the guests and hosts may communicate over a network using
multicast.

The multicast address is the address that a client will use to
send fencing requests to fence_virtd.

Multicast IP Address [225.0.0.12]:

Using ipv4 as family.

Multicast IP Port [1229]:

Setting a preferred interface causes fence_virtd to listen only
on that interface.  Normally, it listens on all interfaces.
In environments where the virtual machines are using the host
machine as a gateway, this *must* be set (typically to virbr0).
Set to 'none' for no interface.

Interface [virbr0]: br0:1

The key file is the shared key information which is used to
authenticate fencing requests.  The contents of this file must
be distributed to each physical host and virtual machine within
a cluster.

Key File [/etc/cluster/fence_xvm.key]:

Backend modules are responsible for routing requests to
the appropriate hypervisor or management layer.

Backend module [libvirt]:

Configuration complete.

=== Begin Configuration ===
backends {
        libvirt {
                uri = "qemu:///system";
        }

}

listeners {
        multicast {
                port = "1229";
                family = "ipv4";
                interface = "br0:1";
                address = "225.0.0.12";
                key_file = "/etc/cluster/fence_xvm.key";
        }

}

fence_virtd {
        module_path = "/usr/lib64/fence-virt";
        backend = "libvirt";
        listener = "multicast";
}

=== End Configuration ===
Replace /etc/fence_virt.conf with the above [y/N]? y
[root@UNIXKB-CP ~]#

Make sure that you are proving the correct interface as the bridge. In My setup , I am using br0:1 virtual interface to communicate with KVM guests.

 

7. Start the fence_virtd service.

[root@UNIXKB-CP ~]# systemctl enable fence_virtd.service
[root@UNIXKB-CP ~]# systemctl start fence_virtd.service
[root@UNIXKB-CP ~]# systemctl status fence_virtd.service
fence_virtd.service - Fence-Virt system host daemon
   Loaded: loaded (/usr/lib/systemd/system/fence_virtd.service; enabled)
   Active: active (running) since Wed 2016-01-20 23:36:14 IST; 1s ago
  Process: 3530 ExecStart=/usr/sbin/fence_virtd $FENCE_VIRTD_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 3531 (fence_virtd)
   CGroup: /system.slice/fence_virtd.service
           └─3531 /usr/sbin/fence_virtd -w

Jan 20 23:36:14 UNIXKB-CP systemd[1]: Starting Fence-Virt system host daemon...
Jan 20 23:36:14 UNIXKB-CP systemd[1]: Started Fence-Virt system host daemon.
Jan 20 23:36:14 UNIXKB-CP fence_virtd[3531]: fence_virtd starting.  Listener: libvirt  Backend: multicast
[root@UNIXKB-CP ~]#

 

Configure the Fencing on Cluster Nodes:

1.Login to one of the cluster node.

2.Make sure that both the nodes have “fence_virt” package.

[root@Node1-LAB ~]# rpm -qa fence-virt
fence-virt-0.3.0-16.el7.x86_64
[root@Node1-LAB ~]#

 

3. The following commands much be scuesscced in an order to configure the fencing in cluster.

[root@Node1-LAB ~]# fence_xvm -o list
Node1                6daac670-c494-4e02-8d90-96cf900f2be9 on
Node2                17707dcb-7bcc-4b36-9498-a5963d86dc2f on
[root@Node1-LAB ~]#

 

4.Cluster nodes entry must be present in the /etc/hosts.

[root@Node1-LAB ~]# cat /etc/hosts |grep Node
192.168.2.10    Node1-LAB  Node1
192.168.2.11    Node2-LAB  Node2
[root@Node1-LAB ~]#

 

5.Configure fence_xvm fence agent on pacemaker cluster.

[root@Node1-LAB ~]# pcs stonith create xvmfence  fence_xvm key_file=/etc/cluster/fence_xvm.key
[root@Node1-LAB ~]# 
[root@Node1-LAB ~]# pcs stonith
 xvmfence       (stonith:fence_xvm):    Started
[root@Node1-LAB ~]#
[root@Node1-LAB ~]# pcs stonith --full
 Resource: xvmfence (class=stonith type=fence_xvm)
  Attributes: key_file=/etc/cluster/fence_xvm.key
  Operations: monitor interval=60s (xvmfence-monitor-interval-60s)
[root@Node1-LAB ~]#

We have successfully configure the fencing on RHEL 7 – Pacemaker/Corosync cluster. (Cluster has been configured between two KVM guests).

 

Validate the STONITH:

 

How should I test my “stonith” configuration ? Here is the small demonstration.

1. Login to the one of the cluster node.

2. Try to fence on of the node.

[root@Node1-LAB ~]# pcs stonith fence Node2
Node: Node2 fenced
[root@Node1-LAB ~]#

 

This will eventually reboot Node2 . Reboot happens based on cluster property.

[root@Node1-LAB ~]# pcs property --all |grep stonith-action
 stonith-action: reboot
[root@Node1-LAB ~]#

 

Stonith also can be ON/OFF using pcs property command.

[root@Node1-LAB ~]# pcs property --all |grep stonith-enabled
 stonith-enabled: true
[root@Node1-LAB ~]#

 

Hope this article is informative to you.