UnixArena

RHEL 7 – Pacemaker – Define the Resource Behaviour – Part 9

By 2 Comments

In Pacemaker/Corosync cluster, there are may aspects/key elements that we need to understand before playing with cluster operations. Otherwise , it might cause unnecessary outage/downtime for the services. The most important elements are setting the preferred resource location , ordering (defining dependencies) , resource fail counts, resource-stickiness, colocation, clone, master/slave, promote/demote etc. Let’s go through the article and understand how these elements contributes in cluster operation. 

 

Cluster Status:

[root@UA-HA ~]# pcs status
Cluster name: UABLR
Last updated: Sat Oct 17 19:44:40 2015          Last change: Sun Jan 10 14:18:25 2016 by root via crm_resource on UA-HA2
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 5 resources configured

Online: [ UA-HA UA-HA2 ]

Full list of resources:

 Resource Group: WEBRG1
     vgres      (ocf::heartbeat:LVM):   Started UA-HA
     webvolfs   (ocf::heartbeat:Filesystem):    Started UA-HA
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA
     webres     (ocf::heartbeat:apache):        Started UA-HA
 Resource Group: UAKVM2
     UAKVM2_res (ocf::heartbeat:VirtualDomain): Started UA-HA

PCSD Status:
  UA-HA: Online
  UA-HA2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@UA-HA ~]#

 

Preferred Resource Location:

Pacaemaker/corosync allows resource to choose the preferred location. You can defined the preferred location using  “pcs constraint” command. Here  we are just intimating “UAKVM2” resource’s preferred node as UA-HA with a score of 50. The score here indicates how badly we would like the resource to run somewhere.

 

[root@UA-HA ~]# pcs constraint location UAKVM2 prefers UA-HA=50
[root@UA-HA ~]# pcs constraint
Location Constraints:
  Resource: UAKVM2
    Enabled on: UA-HA (score:50)
Ordering Constraints:
Colocation Constraints:
[root@UA-HA ~]# pcs resource
 Resource Group: WEBRG1
     vgres      (ocf::heartbeat:LVM):   Started UA-HA
     webvolfs   (ocf::heartbeat:Filesystem):    Started UA-HA
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA
     webres     (ocf::heartbeat:apache):        Started UA-HA
 Resource Group: UAKVM2
     UAKVM2_res (ocf::heartbeat:VirtualDomain): Started UA-HA
[root@UA-HA ~]#

 

If you didn’t specify any score, then resource prefer to run all the time on UA-HA.  The default score is INFINITY.

 

[box type=”info” align=”” class=”” width=””]
From PCS Man Page,

 location <resource id> prefers <node[=score]>...
        Create a location constraint on a resource to prefer the specified
        node and score (default score: INFINITY)

    location <resource id> avoids <node[=score]>...
        Create a location constraint on a resource to avoid the specified
        node and score (default score: INFINITY)

[/box]

 

Using location constraint, you can also avoid specific node to run  a particular resource.

[root@UA-HA ~]# pcs constraint location UAKVM2 avoids UA-HA2=50
[root@UA-HA ~]# pcs constraint
Location Constraints:
  Resource: UAKVM2
    Enabled on: UA-HA (score:50)
    Disabled on: UA-HA2 (score:-50)
Ordering Constraints:
Colocation Constraints:
[root@UA-HA ~]#

 

At any time , you can remove the location constraint using the constraint ID.

To get the constraint id, use “–full” option.

[root@UA-HA ~]# pcs constraint --full
Location Constraints:
  Resource: UAKVM2
    Enabled on: UA-HA (score:50) (id:location-UAKVM2-UA-HA-50)
    Disabled on: UA-HA2 (score:-50) (id:location-UAKVM2-UA-HA2--50)
Ordering Constraints:
Colocation Constraints:
[root@UA-HA ~]#

 

Remove the constraint which we have created.

[root@UA-HA ~]# pcs constraint location remove location-UAKVM2-UA-HA-50
[root@UA-HA ~]# pcs constraint location remove location-UAKVM2-UA-HA2--50
[root@UA-HA ~]# pcs constraint
Location Constraints:
Ordering Constraints:
Colocation Constraints:
[root@UA-HA ~]#

 

[box type=”info” align=”” class=”” width=””]When defining constraints, you also need to deal with scores. Scores of all kinds are integral to how the cluster works. Practically everything from migrating a resource to deciding which resource to stop in a degraded cluster is achieved by manipulating scores in some way. Scores are calculated on a per-resource basis and any node with a negative score for a resource cannot run that resource. After calculating the scores for a resource, the cluster then chooses the node with the highest score. INFINITY is currently defined as 1,000,000. Additions or subtractions with it stick to the following three basic rules:

Any value + INFINITY = INFINITY

Any value – INFINITY = -INFINITY

INFINITY – INFINITY = -INFINITY

When defining resource constraints, you specify a score for each constraint. The score indicates the value you are assigning to this resource constraint. Constraints with higher scores are applied before those with lower scores. By creating additional location constraints with different scores for a given resource, you can specify an order for the nodes that a resource will fail over to.[/box]

 

Resource Ordering: (Defining Resource dependencies):

You need to define the resource ordering if you are not using the resource groups. Most of the cases , resources are need to start in sequential. For an example, File-system resource can’t be started prior to volume group resource. Similar to that IP resource should be on-line before starting the Apache resource.

Let’s assume that we do not have the resource group and following resources are configured in cluster

[root@UA-HA ~]# pcs resource
     vgres      (ocf::heartbeat:LVM):   Started UA-HA2
     webvolfs   (ocf::heartbeat:Filesystem):    Started UA-HA2
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA2
     webres     (ocf::heartbeat:apache):        Started UA-HA2

 

At this point , no constraint has been configured.

[root@UA-HA ~]# pcs constraint
Location Constraints:
Ordering Constraints:
Colocation Constraints:
[root@UA-HA ~]#

 

Plan for the Resource order:

  1.   Volume Group (LVM)  – vgres
  2.   Filesystem – webvolfs   (To store the website data)
  3.   IP address – Cluster IP  (To access the website )
  4.   Apache –  webres    (To provide the web services)

 

To achieve the above resource order, use the following set of commands.

[root@UA-HA ~]# pcs constraint order vgres then webvolfs
Adding vgres webvolfs (kind: Mandatory) (Options: first-action=start then-action=start)
[root@UA-HA ~]# pcs constraint order webvolfs then ClusterIP
Adding webvolfs ClusterIP (kind: Mandatory) (Options: first-action=start then-action=start)
[root@UA-HA ~]# pcs constraint order ClusterIP then webres
Adding ClusterIP webres (kind: Mandatory) (Options: first-action=start then-action=start)
[root@UA-HA ~]# pcs constraint
Location Constraints:
Ordering Constraints:
  start vgres then start webvolfs (kind:Mandatory)
  start webvolfs then start ClusterIP (kind:Mandatory)
  start ClusterIP then start webres (kind:Mandatory)
Colocation Constraints:
[root@UA-HA ~]#

We have successfully configured the resource dependencies.

 

To remove the resource dependencies, use the following set of commands.
1. List the constraints with id.

[root@UA-HA ~]# pcs constraint --full
Location Constraints:
Ordering Constraints:
  start vgres then start webvolfs (kind:Mandatory) (id:order-vgres-webvolfs-mandatory)
  start webvolfs then start ClusterIP (kind:Mandatory) (id:order-webvolfs-ClusterIP-mandatory)
  start ClusterIP then start webres (kind:Mandatory) (id:order-ClusterIP-webres-mandatory)
Colocation Constraints:
[root@UA-HA ~]#

2. Remove the order constraint using following command.

[root@UA-HA ~]# pcs constraint order remove vgres order-vgres-webvolfs-mandatory
[root@UA-HA ~]# pcs constraint order remove webvolfs order-webvolfs-ClusterIP-mandatory
[root@UA-HA ~]# pcs constraint order remove ClusterIP order-ClusterIP-webres-mandatory
[root@UA-HA ~]#  pcs constraint --full
Location Constraints:
Ordering Constraints:
Colocation Constraints:
[root@UA-HA ~]#

[box type=”note” align=”” class=”” width=””]You need to configure the resource order constraint when you do not have the resource group. If you have resource group, it does the resource ordering and reduces the manual work. [/box]

 

Resource fail counts & Migration Threshold:

Migration thresholds defines that how many times failed resource should try to start on the running node. For an example, if you define migration-threshold=2  for a resource , it will automatically migrate to a new node after 2 failures.

To set the migration threshold, use the following command.

 

[root@UA-HA ~]# pcs resource update  UAKVM2_res meta migration-threshold="4"
[root@UA-HA ~]# pcs resource show UAKVM2_res
 Resource: UAKVM2_res (class=ocf provider=heartbeat type=VirtualDomain)
  Attributes: hypervisor=qemu:///system config=/kvmpool/qemu_config/UAKVM2.xml migration_transport=ssh
  Meta Attrs: allow-migrate=true priority=100 migration-threshold=4
  Operations: start interval=0s timeout=120s (UAKVM2_res-start-interval-0s)
              stop interval=0s timeout=120s (UAKVM2_res-stop-interval-0s)
              monitor interval=10 timeout=30 (UAKVM2_res-monitor-interval-10)
              migrate_from interval=0 timeout=120s (UAKVM2_res-migrate_from-interval-0)
              migrate_to interval=0 timeout=120 (UAKVM2_res-migrate_to-interval-0)
[root@UA-HA ~]#

Resource fail count will come in to play, when it reaches the configured  migration-threshold value. If this resource failed on running node , it will tried to start the resource on same node for 4 times. If it still fails, then it will move the resource to the next available node based configured constraint.

 

To see the fail counts , use the one of the following command.

[root@UA-HA ~]# pcs resource failcount show UAKVM2_res
Failcounts for UAKVM2_res
 UA-HA: 1
[root@UA-HA ~]# crm_failcount -r UAKVM2_res
scope=status  name=fail-count-UAKVM2_res value=1
[root@UA-HA ~]#

 

Do you would like to reset the fail-counts manually ?

[root@UA-HA ~]# pcs resource cleanup UAKVM2_res
Waiting for 2 replies from the CRMd.. OK
Cleaning up UAKVM2_res on UA-HA, removing fail-count-UAKVM2_res
Cleaning up UAKVM2_res on UA-HA2, removing fail-count-UAKVM2_res

[root@UA-HA ~]#

OR

[root@UA-HA ~]# pcs resource failcount reset UAKVM2_res UA

 

Check the resource fail-count again.

[root@UA-HA ~]# crm_failcount  --r UAKVM2_res
scope=status  name=fail-count-UAKVM2_res value=0
[root@UA-HA ~]#
[root@UA-HA ~]# pcs resource failcount show UAKVM2_res
No failcounts for UAKVM2_res
[root@UA-HA ~]#

 

Resource-Stickiness:

In some circumstances, it is highly desirable to prevent healthy resources from being moved around the cluster. Moving resources almost always requires a period of downtime. For complex services like Oracle databases, this period can be quite long. To address this, Pacemaker has the concept of resource stickiness which controls how much a service prefers to stay running where it is.

1. Check the Resource status:

Full list of resources:

 Resource Group: WEBRG1
     vgres      (ocf::heartbeat:LVM):   Started UA-HA2
     webvolfs   (ocf::heartbeat:Filesystem):    Started UA-HA2
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA2
     webres     (ocf::heartbeat:apache):        Started UA-HA2
 Resource Group: UAKVM2
     UAKVM2_res (ocf::heartbeat:VirtualDomain): Started UA-HA

 

2. Let’s stop the cluster services on UA-HA.

[root@UA-HA ~]# pcs cluster stop
Stopping Cluster (pacemaker)... Stopping Cluster (corosync)...
[root@UA-HA ~]#

 

3. UAKVM2 resource group should be moved to UA-HA2 automatically.

[root@UA-HA ~]# ssh UA-HA2 pcs status
Cluster name: UABLR
Last updated: Mon Jan 11 05:30:25 2016          Last change: Mon Jan 11 05:29:44 2016 by root via crm_attribute on UA-HA
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 5 resources configured

Online: [ UA-HA2 ]
OFFLINE: [ UA-HA ]

Full list of resources:

 Resource Group: WEBRG1
     vgres      (ocf::heartbeat:LVM):   Started UA-HA2
     webvolfs   (ocf::heartbeat:Filesystem):    Started UA-HA2
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA2
     webres     (ocf::heartbeat:apache):        Started UA-HA2
 Resource Group: UAKVM2
     UAKVM2_res (ocf::heartbeat:VirtualDomain): Started UA-HA2

 

4. Start the cluster service on UA-HA and see what happens to UAKVM2.

[root@UA-HA ~]# pcs cluster start
Starting Cluster...
[root@UA-HA ~]# ssh UA-HA2 pcs status
Cluster name: UABLR
Last updated: Mon Jan 11 05:30:39 2016          Last change: Mon Jan 11 05:29:44 2016 by root via crm_attribute on UA-HA
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 5 resources configured

Online: [ UA-HA UA-HA2 ]

Full list of resources:

 Resource Group: WEBRG1
     vgres      (ocf::heartbeat:LVM):   Started UA-HA2
     webvolfs   (ocf::heartbeat:Filesystem):    Started UA-HA2
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA2
     webres     (ocf::heartbeat:apache):        Started UA-HA2
 Resource Group: UAKVM2
     UAKVM2_res (ocf::heartbeat:VirtualDomain): Started UA-HA

UAKVM2 is moved back to UA-HA node but this will create some sort downtime. If you have configured the resource-stickness, you could prevent the resource moving from one node to another.

[root@UA-HA ~]# pcs resource defaults resource-stickiness=100
[root@UA-HA ~]# pcs resource defaults
resource-stickiness: 100
[root@UA-HA ~]#

Perform Step 2 to Step 4 & see the difference . UAKVM2 should be running on UA-HA2.

 

colocation:

When the location of one resource depends on the location of another one, we call this colocation. I would say colocaiton and resource order required when you are not using resource group.

Assuming that you have configured the volume group resource and File-system resource. You have also configured the resource order that which one to start first. But cluster might try to start volume group resource on one node and filesystem resource on other node. In such a cases, we need to tell to the cluster that run the filesystem resource where you are running the volume group resource.

Let’s see that how we can configure the colocation between vgres (LVM VG) & webvolfs (filesystem).

[root@UA-HA ~]# pcs constraint colocation add vgres with webvolfs INFINITY
[root@UA-HA ~]# pcs constraint
Location Constraints:
Ordering Constraints:
Colocation Constraints:
  vgres with webvolfs (score:INFINITY)
[root@UA-HA ~]#

We have successfully configured the colocation parameter between vgres & webvolfs. In this case, webvolfs resource follows the vgres.

We will see clone,Master/slave and Promote/Demote stuffs  on up coming articles.

Hope this article is informative to you. Share it ! Comment it !! Be Sociable !!!

Reader Interactions

Comments

  1. Tapas Mallick says

    Great Article – Thanks a lot.

    Can we have article on “clone,Master/slave and Promote/Demote” as mentioned in this article?

    Waiting for the same.

    Thanks again.

    Reply

  2. Steven Silk says

    This is really a great article – I wish we could clean up all the old Pacemaker outdated information that is out there.

    I have a question. I have two nodes operating in a master/master configuration. When one fails the services migrate from one to the other. The problem is — the virtual IP that migrates. One of the services that is running on the original node needs to re-read its configuration file to know that IP is now on that host. I actually need to stop the service and restart it after the migration. Does that make sense? Or is there something in the secret sauce that I have overlooked?

    thank you,

    Steve

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *