In Pacemaker/Corosync cluster (RHEL 7 HA), resources management and resource group management are important tasks . Depends on the cluster HA services, you might need to configure N-number of resources. In most of the cases , you might need to start set of resources sequentially, and stop in the reverse order. To simplify this configuration, Pacemaker supports the concept of groups (Resource groups). For an example, to provide the web-services in HA model, you need resources like , File system(To store website data) , IP (Clustered IP to access website) and Apache (To provide the web-services) . To start the Apache service , you need a filesystem which stores the website data. So the resources must start in the following order ,
- IP
- File-system
- Apache service
Let’s see that how to configure the Highly available Apache service (website) in Redhat cluster (Pacemaker/Corosync). In the previous article, we have already created the IP resource.
[root@UA-HA ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 18:24:10 2015 Last change: Mon Dec 28 18:09:30 2015 by root via crm_resource on UA-HA
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 1 resource configured
Online: [ UA-HA UA-HA2 ]
Full list of resources:
ClusterIP (ocf::heartbeat:IPaddr2): Started UA-HA
PCSD Status:
UA-HA: Online
UA-HA2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@UA-HA ~]# pcs resource show ClusterIP
Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.203.190 cidr_netmask=24
Operations: start interval=0s timeout=20s (ClusterIP-start-interval-0s)
stop interval=0s timeout=20s (ClusterIP-stop-interval-0s)
monitor interval=30s (ClusterIP-monitor-interval-30s)
[root@UA-HA ~]#
Create the File-system and Apache resources quickly:
Filesystem :
- Shared LUN – /dev/sdc
- Volume Group – webvg
- Volume – webvol1
- Filesystem Type – ext4
Quick Setup for Filesystem resource:
[root@UA-HA2 ~]# vgcreate webvg /dev/sdc [root@UA-HA2 ~]# lvcreate -L 90M -n /dev/webvg/webvol1 [root@UA-HA2 ~]# mkfs.ext4 /dev/webvg/webvol1
Apache:
- httpd
Quick Setup:
[root@UA-HA www]# yum install -y httpd
Pre-prerequisites for LVM :
(Perform the following changes on both the cluster nodes)
1.Make sure that “use_lvmetad” parameter is set to “0”. This is mandatory when you use “Pacemaker”.
[root@UA-HA ~]# grep use_lvmetad /etc/lvm/lvm.conf |grep -v "#"
use_lvmetad = 0
[root@UA-HA ~]#
2.To prevent the automatic volume group activation, update the volume_list parameter with local VG’s which needs to be activated automatically.
[root@UA-HA ~]# grep volume_list /etc/lvm/lvm.conf |grep -v "#"
volume_list = [ "nfsvg", "rhel" ]
[root@UA-HA ~]# vgs
VG #PV #LV #SN Attr VSize VFree
nfsvg 2 1 0 wz--n- 1.94g 184.00m
rhel 1 2 0 wz--n- 19.51g 0
webvg 1 1 0 wz--n- 92.00m 0
[root@UA-HA ~]#
In My case, “webvg” will be managed through cluster.
3. Mount the volume in “/var/www” and create the following directories and files.
[root@UA-HA2 ~]# mount /dev/webvg/webvol1 /var/www [root@UA-HA2 ~]# cd /var/www [root@UA-HA2 www]# mkdir errror html cgi-bin total 3 drwxr-xr-x 2 root root 1024 Dec 28 20:26 cgi-bin drwxr-xr-x 2 root root 1024 Dec 28 20:26 errror drwxr-xr-x 2 root root 1024 Dec 28 20:27 html [root@UA-HA2 www]# cd html/ [root@UA-HA2 html]# vi index.html Hello, Welcome to UnixArena [root@UA-HA2 html]#
3.Rebuild the “initramfs” boot image to guarantee that the boot image will not try to activate a volume group controlled by the cluster. Update the initramfs device using the following command.
[root@UA-HA ~]# dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r) [root@UA-HA ~]#
4. Reboot the nodes.
Create the LVM Cluster resources (vg & lv ), File-system cluster Resources:
1.Create the cluster volume group resource.
[root@UA-HA ~]# pcs resource create vgres LVM volgrpname=webvg exclusive=true
[root@UA-HA ~]# pcs resource show vgres
Resource: vgres (class=ocf provider=heartbeat type=LVM)
Attributes: volgrpname=webvg exclusive=true
Operations: start interval=0s timeout=30 (vgres-start-interval-0s)
stop interval=0s timeout=30 (vgres-stop-interval-0s)
monitor interval=10 timeout=30 (vgres-monitor-interval-10)
[root@UA-HA ~]#
vgres – Resource Name (Any Unique Name)
webvg – Volume Group
2. Create the cluster mount resource.
[root@UA-HA ~]# pcs resource create webvolfs Filesystem device="/dev/webvg/webvol1" directory="/var/www" fstype="ext4"
[root@UA-HA ~]# pcs resource show webvolfs
Resource: webvolfs (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/webvg/webvol1 directory=/var/www fstype=ext4
Meta Attrs:
Operations: start interval=0s timeout=60 (webvolfs-start-interval-0s)
stop interval=0s timeout=60 (webvolfs-stop-interval-0s)
monitor interval=20 timeout=40 (webvolfs-monitor-interval-20)
[root@UA-HA ~]#
3. Before adding the resource, you must update the local /etc/httpd/conf/httpd.conf with following contents. This entries required for pacemaker to get the web-server status .
4. Check the apache server status . (httpd.service). Make sure that httpd.service is stopped & disabled on both the cluster nodes. This service will be managed by cluster.
[root@UA-HA ~]# systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:httpd(8)
man:apachectl(8)
Dec 27 13:55:52 UA-HA systemd[1]: Starting The Apache HTTP Server...
Dec 27 13:55:55 UA-HA httpd[2002]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.203.134. Set the...is message
Dec 27 13:55:55 UA-HA systemd[1]: Started The Apache HTTP Server.
Dec 27 15:16:02 UA-HA httpd[11786]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.203.134. Set th...is message
Dec 27 15:16:02 UA-HA systemd[1]: Reloaded The Apache HTTP Server.
Dec 28 18:06:57 UA-HA systemd[1]: Started The Apache HTTP Server.
Dec 28 20:30:56 UA-HA systemd[1]: Stopping The Apache HTTP Server...
Dec 28 20:30:57 UA-HA systemd[1]: Stopped The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@UA-HA ~]#
3. Create the Apache cluster resource.
[root@UA-HA ~]# pcs resource create webres apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status" [root@UA-HA ~]# pcs status Cluster name: UABLR Last updated: Mon Dec 28 20:11:51 2015 Last change: Mon Dec 28 20:11:44 2015 by root via cibadmin on UA-HA Stack: corosync Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum 2 nodes and 4 resources configured Online: [ UA-HA UA-HA2 ] Full list of resources: vgres (ocf::heartbeat:LVM): (target-role:Stopped) Stopped webvolfs (ocf::heartbeat:Filesystem): (target-role:Stopped) Stopped ClusterIP (ocf::heartbeat:IPaddr2): Started UA-HA2 webres (ocf::heartbeat:apache): Stopped PCSD Status: UA-HA: Online UA-HA2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@UA-HA ~]#
In normal cases, resource group will be created (by specifying –group in the end of command line) when you add the first cluster resource to make the dependency tree. To understand the cluster resources and resource group management concept , I am creating the resource group at the end.
If you see any resource was started , just stop it to avoid the errors.
[root@UA-HA ~]# pcs resource disable vgres webvolfs webres stop ClusterIP [root@UA-HA ~]# pcs resource vgres (ocf::heartbeat:LVM): Stopped webvolfs (ocf::heartbeat:Filesystem): Stopped ClusterIP (ocf::heartbeat:IPaddr2): Stopped webres (ocf::heartbeat:apache): Stopped [root@UA-HA ~]#
4. Create the resource group to form the resource dependencies to stop & start in resources in sequence.
[root@UA-HA ~]# pcs resource group add WEBRG1 ClusterIP vgres webvolfs webres
As per the above command , here is the resource start up sequence
- ClusterIP – Website URL
- vgres – Volume Group
- webvolfs – Mount Resource
- webres – httpd Resource
Stop sequence is just reverse to the start.
- webres – httpd Resource
- webvolfs – Mount Resource
- vgres – Volume Group
- ClusterIP – Website URL
5. Check the resources status. You should be able to see that all the resources are bundled as one resource group with “WEBRG1” .
[root@UA-HA ~]# pcs resource
Resource Group: WEBRG1
ClusterIP (ocf::heartbeat:IPaddr2): Started UA-HA2
vgres (ocf::heartbeat:LVM): Stopped
webvolfs (ocf::heartbeat:Filesystem): Stopped
webres (ocf::heartbeat:apache): Stopped
[root@UA-HA ~]#
6. Enable the disabled resources in following sequence.
[root@UA-HA ~]# pcs resource enable ClusterIP [root@UA-HA ~]# pcs resource enable vgres [root@UA-HA ~]# pcs resource enable webvolfs [root@UA-HA ~]# pcs resource enable webres
7. Verify the cluster status.
[root@UA-HA ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 20:54:43 2015 Last change: Mon Dec 28 20:51:30 2015 by root via crm_resource on UA-HA2
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 4 resources configured
Online: [ UA-HA UA-HA2 ]
Full list of resources:
Resource Group: WEBRG1
ClusterIP (ocf::heartbeat:IPaddr2): Started UA-HA2
vgres (ocf::heartbeat:LVM): Started UA-HA2
webvolfs (ocf::heartbeat:Filesystem): Started UA-HA2
webres (ocf::heartbeat:apache): Started UA-HA2
PCSD Status:
UA-HA: Online
UA-HA2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@UA-HA ~]#
8. Let’s move the resources from UA-HA2 to UA-HA. In this case, we no need to move each resources manually.We just need to move the Resource group since we have bundled the required resource in to that.
[root@UA-HA ~]# pcs resource move WEBRG1 UA-HA
[root@UA-HA ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 20:58:55 2015 Last change: Mon Dec 28 20:58:41 2015 by root via crm_resource on UA-HA
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 4 resources configured
Online: [ UA-HA UA-HA2 ]
Full list of resources:
Resource Group: WEBRG1
ClusterIP (ocf::heartbeat:IPaddr2): Started UA-HA
vgres (ocf::heartbeat:LVM): Started UA-HA
webvolfs (ocf::heartbeat:Filesystem): Started UA-HA
webres (ocf::heartbeat:apache): Started UA-HA
PCSD Status:
UA-HA: Online
UA-HA2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@UA-HA ~]#
You should be able to see the webpage like following .
9. How to stop the pacemaker resource group ? Just disable the resource group.
[root@UA-HA2 ~]# pcs resource disable WEBRG1
[root@UA-HA2 ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 21:12:18 2015 Last change: Mon Dec 28 21:12:14 2015 by root via crm_resource on UA-HA2
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 4 resources configured
Online: [ UA-HA UA-HA2 ]
Full list of resources:
Resource Group: WEBRG1
ClusterIP (ocf::heartbeat:IPaddr2): (target-role:Stopped) Stopped
vgres (ocf::heartbeat:LVM): (target-role:Stopped) Stopped
webvolfs (ocf::heartbeat:Filesystem): (target-role:Stopped) Stopped
webres (ocf::heartbeat:apache): (target-role:Stopped) Stopped
PCSD Status:
UA-HA: Online
UA-HA2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@UA-HA2 ~]#
10. How to start the resource group ? Use enable option for the RG.
[root@UA-HA2 ~]# pcs resource enable WEBRG1
[root@UA-HA2 ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 21:14:04 2015 Last change: Mon Dec 28 21:14:01 2015 by root via crm_resource on UA-HA2
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 4 resources configured
Online: [ UA-HA UA-HA2 ]
Full list of resources:
Resource Group: WEBRG1
ClusterIP (ocf::heartbeat:IPaddr2): Started UA-HA2
vgres (ocf::heartbeat:LVM): Started UA-HA2
webvolfs (ocf::heartbeat:Filesystem): Started UA-HA2
webres (ocf::heartbeat:apache): Started UA-HA2
PCSD Status:
UA-HA: Online
UA-HA2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@UA-HA2 ~]#
Note:
Redhat cluster (Pacemaker/corosync) have many parameters like resource stickiness and failure counts.These attributes will play a role that where to start the resources.
To clear the errors , use the following command
# pcs resource cleanup
To clear the resource fail counts , use the following command.
[root@UA-HA2 ~]# pcs resource clear ClusterIP [root@UA-HA2 ~]# pcs resource clear vgres [root@UA-HA2 ~]# pcs resource clear webvolfs [root@UA-HA2 ~]# pcs resource clear webres [root@UA-HA2 ~]#
Hope this article is informative to you.
Share it ! Comment it !! Be Sociable !!!
Marcio says
Hi Guys,
thanks for this nice site and share it. It helps me a lot.
Looks it is missing to create ClusterIP
# pcs resource create ClusterIP ocf:heartbeat:IPaddr2 ip=192.168.203.190 cidr_netmask=24 op monitor interval=20
thanks
Marcio
Lingeswaran R says
I think that the previous article is already covered that.
Lingesh
tom says
in this post you have not mentioned anything about the DLM and CLVM like it is necessary for GFS2 filesystem, in this post you have taken EXT4 as a filesystem so does this mean that we don’t need to use DLM and CLVM in the pacemaker cluster resource for EXT4 filesystem.