现象描述:
12.1.0.2
集群进程
ocssd.bin
占用较高的
CPU
:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27647 grid RT 0 5002m 2.8g 86m S 117.6 0.6 209825:27 ocssd.bin
系统总体 CPU 使用并不高,造成以上单个线程的 CPU 使用率较高是由 clsdadr_bucketThread 线程。
#5 0x00007f0b76feb0bc in clsdadr_process_bucket () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#6 0x00007f0b76feac23 in
clsdadr_bucketThread
() from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
同时,查看 ocssd.trc 文件中,可以发现大量以下的提示信息:
2018-10-12 14:33:04.342374 : CSSD:3886925568: clssgmMemberPublicInfo: group OSM_ALL member 1 not found
2018-10-12 14:33:04.435152 : CSSD:3886925568: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2018-10-12 14:33:04.435164 : CSSD:3886925568: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2018-10-12 14:33:04.437429 : CSSD:3889161984: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2018-10-12 14:33:04.442236 : CSSD:3883771648: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:160:4
2018-10-12 14:33:04.552308 : CSSD:3848582912: clssnmSendingThread: sending status msg to all nodes
2018-10-12 14:33:04.553123 : CSSD:3848582912: clssnmSendingThread: sent 4 status msgs to all nodes
2018-10-12 14:33:05.353496 : CSSD:3886925568: clssgmMemberPublicInfo: group OSM_ALL member 1 not found
2018-10-12 14:33:06.346352 : CSSD:3886925568: clssgmMemberPublicInfo: group OSM_ALL member 1 not found
2018-10-12 14:33:06.488828 : CSSD:3886925568: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2018-10-12 14:33:06.488842 : CSSD:3886925568: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2018-10-12 14:33:06.492775 : CSSD:3889161984: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2018-10-12 14:33:06.499862 : CSSD:3883771648: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:160:4
2018-10-12 14:33:07.233537 :GIPCHTHR:3878508288: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30020loopCount 37
2018-10-12 14:33:07.349067 : CSSD:3886925568: clssgmMemberPublicInfo: group OSM_ALL member 1 not found
2018-10-12 14:33:08.351711 : CSSD:3886925568: clssgmMemberPublicInfo: group OSM_ALL member 1 not found
2018-10-12 14:33:08.548780 : CSSD:3886925568: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2018-10-12 14:33:08.548794 : CSSD:3886925568: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2018-10-12 14:33:08.551894 : CSSD:3889161984: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2018-10-12 14:33:08.558731 : CSSD:3883771648: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:160:4
通过查询 MOS,查到类似的情况,详细参考MOS文档 ID 2235698.1,该问题是由 Bug 26513709 造成,可通过应用补丁 20302006 与 25211209 避免,同时,官方宣布在 12.2.0.2 中已经将该 bug修复。