Professional Documents
Culture Documents
EVICTION DUE TO
MEMBER KILL
ESCALATION
November 9, 201211g R2 RAC, Uncategorized
If the Oracle Clusterware itself is working perfectly but one of the RAC instances is
hanging , the database LMON process will request a member kill escalation and ask the
CSS process to remove the hanging database instance from the cluster.
The following example will demonstrate it in a cluster consisting of two nodes:
SQL> select instance_name, host_name from gv$instance;
SQL> col host_name for a20
select instance_name, host_name from gv$instance;
INSTANCE_NAME HOST_NAME
-
orcl1 host01.example.com
orcl2 host02.example.com
- On host02 server stop the execution of all rdbms processes (by sending the
STOP signal)
Find out current database processes
[root@host02 ~]# ps -ef | grep ora_ | grep orcl2
oracle 6215 1 0 11:20 ? 00:00:00 ora_pmon_orcl2
oracle 6217 1 0 11:20 ? 00:00:00 ora_vktm_orcl2
oracle 6221 1 0 11:20 ? 00:00:00 ora_gen0_orcl2
oracle 6223 1 0 11:20 ? 00:00:00 ora_diag_orcl2
oracle 6225 1 0 11:20 ? 00:00:00 ora_dbrm_orcl2
oracle 6227 1 0 11:20 ? 00:00:00 ora_ping_orcl2
oracle 6229 1 0 11:20 ? 00:00:00 ora_psp0_orcl2
oracle 6231 1 0 11:20 ? 00:00:00 ora_acms_orcl2
oracle 6233 1 0 11:20 ? 00:00:00 ora_dia0_orcl2
oracle 6235 1 0 11:20 ? 00:00:00 ora_lmon_orcl2
oracle 6237 1 0 11:20 ? 00:00:02 ora_lmd0_orcl2
stop the execution of all rdbms processes (by sending the STOP signal)
[root@host02 ~]# ps -ef | grep ora_ | grep orcl2 | awk {print $2} | while read PID
do
kill -STOP $PID
done
. From the client point of view the Real Application Cluster database is hanging
on both nodes. No queries or DMLs are possible. Try to execute a query. The
query will hang.
SQL> select instance_name, host_name from gv$instance;
no output, query hangs
. Due to missing heartbeats the healthy RAC instance on node host01 will remove the
hanging RAC instance by requesting a member kill escalation.
Check the database alert log file on host01 : LMS process issues a request to
CSSD to reboot the node.
The node is evicted and instance is restarted after node joins the cluster.
[root@host01 trace]# tailf /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/alert_orcl1.log
LMS0 (ospid: 31771) has detected no messaging activity from instance 2
LMS0 (ospid: 31771) issues an IMR to resolve the situation
Please check LMS0 trace file for more detail.
Fri Nov 09 11:15:04 2012
Remote instance kill is issued with system inc 30
Remote instance kill map (size 1) : 2
LMON received an instance eviction notification from instance 1
The instance eviction reason is 0x20000000
The instance eviction map is 2
Fri Nov 09 11:15:13 2012
IPC Send timeout detected. Sender: ospid 6308 [oracle@host01.example.com (PZ97)]
Receiver: inst 2 binc 429420846 ospid 6251
Waiting for instances to leave:
2
Reconfiguration started (old inc 4, new inc 8)
List of instances:
1 (myinst: 1)
.. Recovery of instance 2 starts
Global Resource Directory frozen
.
All grantable enqueues granted
Post SMON to start 1st pass IR
-
Instance recovery: looking for dead threads
Beginning instance recovery of 1 threads
Started redo scan
IPC Send timeout to 2.0 inc 4 for msg type 12 from opid 42
Completed redo scan
read 93 KB redo, 55 data blocks need recovery
Started redo application at
Thread 2: logseq 9, block 42
Recovery of Online Redo Log: Thread 2 Group 3 Seq 9 Reading mem 0
Mem# 0: +DATA/orcl/onlinelog/group_3.266.798828557
Mem# 1: +FRA/orcl/onlinelog/group_3.259.798828561
Completed redo application of 0.05MB
Completed instance recovery at
Thread 2: logseq 9, block 228, scn 1069404
52 data blocks read, 90 data blocks written, 93 redo k-bytes read
Thread 2 advanced to log sequence 10 (thread recovery)
Fri Nov 09 12:18:55 2012
.
Check the cluster clusterware alert log of host01
The node is evicted and rebooted to join the cluster
[grid@host01 host01]$ tailf /u01/app/11.2.0/grid/log/host01/alerthost01.log
[cssd(14493)]CRS-1607:Node host02 is being evicted in cluster incarnation
247848838; details at (:CSSNM00007:) in
/u01/app/11.2.0/grid/log/host01/cssd/ocssd.log.
2012-11-09 11:15:56.140
[ohasd(12412)]CRS-8011:reboot advisory message from host: host02, component:
mo103324, with time stamp: L-2012-11-09-
11:15:56.580
[ohasd(12412)]CRS-8013:reboot advisory message text: clsnomon_status: need to
reboot, unexpected failure 8 received from
CSS
2012-11-09 11:16:17.365
[cssd(14493)]CRS-1601:CSSD Reconfiguration complete. Active nodes are host01 .
2012-11-09 11:16:17.400
[crsd(14820)]CRS-5504:Node down event reported for node host02.
2
Node 2 joins the cluster
[cssd(14493)]CRS-1601:CSSD Reconfiguration complete. Active nodes are host01
host02 .
2012-11-09 12:18:52.713
[crsd(14820)]CRS-2772:Server host02 has been assigned to pool Generic.
2012-11-09 12:18:52.713
[crsd(14820)]CRS-2772:Server host02 has been assigned to pool ora.orcl.
7. After the node rejoins the cluster and the instance is restarted, reexecute the
query it succeeds
SQL> conn sys/oracle@orcl as sysdba
col host_name for a20
select instance_name, host_name from gv$instance;
INSTANCE_NAME HOST_NAME
-
orcl1 host01.example.com
orcl2 host02.example.com
References:
http://www.unbreakablecloud.com/wordpress/2010/11/02/understanding-cluster-node-
eviction/
-
Related links:
Home
11G R2 RAC Index
Node Eviction Due To Missing Network Heartbeat
Node Eviction Due To Missing Disk Heartbeat
Node Eviction Due To CSSD Agent Stopping
11g R2 RAC: Reboot-less Node Fencing
11g R2 RAC :Reboot-less Fencing With Missing Disk Heartbeat
11g R2 RAC: Reboot-less Fencing With Missing Network Heartbeat
===========
11g R2 RAC: NODE
EVICTION DUE TO
MISSING NETWORK
HEARTBEAT
November 17, 201211g R2 RAC, Uncategorized
In this post, I will demonstrate node eviction due to missing netsork heartbeat i.e. a
node will be evicted from the cluster, if it cant communicate wioth other nodes in the
cluster. To simulate it, I will stop private network on one of the nodes and then scan alert
logs of the surviving nodes.
Current scenario:
No. of nodes in the cluster : 3
Names of the nodes : host01, host02, host03
Name of the cluster database : orcl
I will stop PVT. network service on host03 so that it is evicted.
Find out the pvt network name
[root@host03 ~]# oifcfg getif
eth0 192.9.201.0 global public
eth1 10.0.0.0 global cluster_interconnect
Stop pvt. network service on host03 so that it cant communicate with host01 and
host02 and will be evicted.
[root@host03 ~]# ifdown eth1
-
OCSSD log of host03
It can be seen that CSSD process of host03 cant communicate with host01 and host02
at 09:43:52
Hence votedisk timeouot is set to Short Disk Time OUT (SDTO) = 27000 ms (27 secs)
2012-11-19 09:43:52.714: [ CSSD][843736976]clssnmPollingThread: node host01 (1)
at 50% heartbeat fatal, removal in 14.880 seconds
2012-11-19 09:43:52.714: [ CSSD][843736976]clssnmPollingThread: node host01 (1)
is impending reconfig, flag 132108, misstime 15120
2012-11-19 09:43:52.714: [ CSSD][843736976]clssnmPollingThread: node host02 (2)
at 50% heartbeat fatal, removal in 14.640 seconds
2012-11-19 09:43:52.714: [ CSSD][843736976]clssnmPollingThread: node host02 (2)
is impending reconfig, flag 132108, misstime 15360
2012-11-19 09:43:52.714: [ CSSD][843736976]clssnmPollingThread: local
diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig
status(1)
2012-11-19 09:43:52.927: [ CSSD][2833247120]clssnmSendingThread: sending
status msg to all nodes
At 09:43:52, CSSD process host03 identifies that it cant communicate with CSSD on
host02 and host03
[cssd(5124)]CRS-1612:Network communication with node host01 (1) missing for 50% of
timeout interval. Removal of this node from cluster in 14.880 seconds
2012-11-19 09:43:52.714
[cssd(5124)]CRS-1612:Network communication with node host02 (2) missing for 50% of
timeout interval. Removal of this node from cluster in 14.640 seconds
2012-11-19 09:44:01.880
[cssd(5124)]CRS-1611:Network communication with node host01 (1) missing for 75% of
timeout interval. Removal of this node from cluster in 6.790 seconds
2012-11-19 09:44:01.880
[cssd(5124)]CRS-1611:Network communication with node host02 (2) missing for 75% of
timeout interval. Removal of this node from cluster in 6.550 seconds
2012-11-19 09:44:06.536
[cssd(5124)]CRS-1610:Network communication with node host01 (1) missing for 90% of
timeout interval. Removal of this node from cluster in 2.780 seconds
2012-11-19 09:44:06.536
[cssd(5124)]CRS-1610:Network communication with node host02 (2) missing for 90% of
timeout interval. Removal of this node from cluster in 2.540 seconds
2012-11-19 09:44:09.599
At 09:44:16, CSSD process of host03 reboots the node to preserve cluster integrity
[cssd(5124)]CRS-1609:This node is unable to communicate with other nodes in the
cluster and is going down to preserve cluster integrity; details at (:CSSNM00008:) in
/u01/app/11.2.0/grid/log/host03/cssd/ocssd.log.
2012-11-19 09:44:16.697
[/u01/app/11.2.0/grid/bin/orarootagent.bin(5713)]CRS-5822:Agent
/u01/app/11.2.0/grid/bin/orarootagent_root disconnected from server. Details at
(:CRSAGF00117:) in
/u01/app/11.2.0/grid/log/host03/agent/crsd/orarootagent_root/orarootagent_root.log.
2012-11-19 09:44:16.193
[ctssd(5285)]CRS-2402:The Cluster Time Synchronization Service aborted on host
host03. Details at (:ctsselect_mmg5_1: in
/u01/app/11.2.0/grid/log/host03/ctssd/octssd.log.
2012-11-19 09:44:21.177
Related links:
Home
11G R2 RAC INDEX
NODE EVICTION DUE TO MISSING DISK HEARTBEAT
NODE EVICTION DUE TO MEMBER KILL ESCALATION
NODE EVICTION DUE TO CSSDAGENT STOPPING
11g R2 RAC: Reboot-less Node Fencing
11g R2 RAC :Reboot-less Fencing With Missing Disk Heartbeat
11g R2 RAC: Reboot-less Fencing With Missing Network Heartbeat
=========
Current scenario:
No. of nodes in the cluster : 3
Names of the nodes : host01, host02, host03
Name of the cluster database : orcl
Stop ISCSI service on host03 so that it cant access shared storage and hence voting
disk
Note that ocssd process of host03 is not able to access voting disks
[cssd(5149)]CRS-1649:An I/O error occured for voting file: ORCL:ASMDISK01;
details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/host03/cssd/ocssd.log.
2012-11-17 03:32:11.310
2012-11-17 03:32:11.311
2012-11-17 03:32:11.311
2012-11-17 03:32:11.312
2012-11-17 03:32:11.310
[client(8050)]CRS-10001:6323 6363 6391 6375 6385 6383 6402 6319 6503 6361 6377
6505 6389 6369 6335 6367 6333 6387 6871 6325 6381 6327 6496 6498 6552 6373
7278 6339 6400 6357 6500 6329 6365
[client(8052)]CRS-10001:ACFS-9113: These processes will now be terminated.
[client(8127)]CRS-10001:ACFS-9114: done.
[client(8178)]CRS-10001:ACFS-9114: done.
2012-11-17 03:33:34.050
[/u01/app/11.2.0/grid/bin/orarootagent.bin(5682)]CRS-5016:Process
/u01/app/11.2.0/grid/bin/acfssinglefsmount spawned by agent
At 03:34, voting disk cant be accessed even after waiting for timeout
2012-11-17 03:34:10.718
[cssd(5149)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting
file ORCL:ASMDISK01 will be considered not functional in 99190 milliseconds
2012-11-17 03:34:10.724
[cssd(5149)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting
file ORCL:ASMDISK02 will be considered not functional in 99180 milliseconds
2012-11-17 03:34:10.724
[cssd(5149)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting
file ORCL:ASMDISK03 will be considered not functional in 99180 milliseconds
2012-11-17 03:35:10.666
[cssd(5149)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting
file ORCL:ASMDISK01 will be considered not functional in 49110 milliseconds
2012-11-17 03:35:10.666
[cssd(5149)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting
file ORCL:ASMDISK02 will be considered not functional in 49110 milliseconds
2012-11-17 03:35:10.666
[cssd(5149)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting
file ORCL:ASMDISK03 will be considered not functional in 49110 milliseconds
2012-11-17 03:35:46.654
[cssd(5149)]CRS-1613:No I/O has completed after 90% of the maximum interval. Voting
file ORCL:ASMDISK01 will be considered not functional in 19060 milliseconds
2012-11-17 03:35:46.654
[cssd(5149)]CRS-1613:No I/O has completed after 90% of the maximum interval. Voting
file ORCL:ASMDISK02 will be considered not functional in 19060 milliseconds
2012-11-17 03:35:46.654
[cssd(5149)]CRS-1613:No I/O has completed after 90% of the maximum interval. Voting
file ORCL:ASMDISK03 will be considered not functional in 19060 milliseconds
2012-11-17 03:36:10.596
2012-11-17 03:36:10.596
2012-11-17 03:36:10.596
CSSD of host03 reboots the node as no. of voting disks available(0) is less than
minimum required (2)
2012-11-17 03:36:15.645
I/O fencing for ORCL database is carried out by CSSD at 03:32 ( same time as
when host02 got the msg that orcl has failed on host03)
2012-11-17 03:32:10.356: [ CSSD][997865360]clssgmFenceClient: fencing client
(0xaa14990), member 2 in group DBORCL, no share, death fence 1, SAGE fence 0
..
..
2012-11-17 03:36:29.610
2012-11-17 03:36:43.992
2012-11-17 03:36:46.441
After network communication cant be established for timeout interval, the node
is removed form cluster
2012-11-17 03:36:46.572
Note that ocssd process of host01 discovers missing disk heartbeat from
host03 at 03:32:16
. At 03:32, CRSD process of host02 receives message that orcl database has
failed on host03 as
2012-11-17 03:32:44.303
. CRSD process of host02 receives message that acfs has failed on host03 as
2012-11-17 03:36:16.981
2012-11-17 03:36:16.981
2012-11-17 03:36:28.920
2012-11-17 03:36:43.328
2012-11-17 03:36:46.297
2012-11-17 03:36:46.470
2012-11-17 03:36:51.890
2012-11-17 03:36:51.909
note that ocssd of host02 discovers missing host03 only after it has been
rebooted at 03:36
[root@host02 ~]# tailf /u01/app/11.2.0/grid/log/host02/cssd/ocssd.log
..
References:
http://www.unbreakablecloud.com/wordpress/2010/11/02/understanding-cluster-node-
eviction/
Related links:
Home