Mpx:Path Bus 1 Tgt 1 Lun 1 to CK200074100595 is dead

对Oracle 10g RAC数据库进行例行检查时发现两个节点在同一时间段有相同的报错信息，应该是和磁盘方面有关系，由于采用的是共享磁盘柜，怀疑是连接磁盘柜的通信出现问题。

查询Storage的相关信息，发现果然是连接Server与Storage的一台交换机断电过，冗余的两台交换机，一台接UPS, 一台接市电，防止UPS损坏导致系统异常。刚好前两天由于特殊情况停过电 。

Jun 8 04:04:08 gnd-rac01 syslogd 1.4.1: restart.
Jun 8 04:04:08 gnd-rac01 snmpd[1271]: NET-SNMP version 5.1.2
Jun 8 05:32:53 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 8 05:32:53 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 8 05:32:53 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 8 05:32:53 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 8 12:58:48 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LOOP DOWN detected (2).
Jun 8 12:59:33 gnd-rac01 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 12:59:33 gnd-rac01 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 12:59:33 gnd-rac01 kernel: SCSI error : <3 0 0 1> return code = 0x10000
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP A5.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP B5.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is dead.
Jun 8 13:00:09 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:00:09 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP occured (f8f7).
Jun 8 13:00:09 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f7f7).
Jun 8 13:00:19 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:00:20 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LOOP UP detected (4 Gbps).
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is alive.
Jun 8 13:27:15 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LOOP DOWN detected (2).
Jun 8 13:28:00 gnd-rac01 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac01 kernel: SCSI error : <3 0 0 1> return code = 0x10000
Jun 8 13:28:00 gnd-rac01 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac01 kernel: SCSI error : <3 0 1 0> return code = 0x10000
Jun 8 13:28:00 gnd-rac01 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP B5.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP A5.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is dead.
Jun 8 13:28:36 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:28:45 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:28:46 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LOOP UP detected (4 Gbps).
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is alive.
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is alive.
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is alive.
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is alive.
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is alive.
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is alive.
Jun 9 05:28:11 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 05:28:11 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 05:28:11 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 05:28:12 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 9 08:51:25 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 08:51:25 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 08:51:25 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 08:51:25 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 9 11:02:16 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 11:02:16 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 11:02:16 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 11:02:16 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 9 14:17:29 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 14:17:29 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 14:17:29 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 14:17:29 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 9 15:00:39 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 15:00:39 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 15:00:39 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 15:00:39 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 9 22:08:21 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 22:08:21 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 22:08:21 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 22:08:21 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 10 08:24:17 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 10 08:24:17 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 10 08:24:17 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 10 08:24:17 gnd-rac01 kernel: hda: ATAPI reset complete

Jun 8 04:04:13 gnd-rac02 syslogd 1.4.1: restart.
Jun 8 04:04:13 gnd-rac02 snmpd[13034]: NET-SNMP version 5.1.2
Jun 8 08:43:24 gnd-rac02 kernel: ide-cd: cmd 0x1e timed out
Jun 8 08:43:24 gnd-rac02 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 8 08:43:24 gnd-rac02 kernel: hda: irq timeout: error=0x00
Jun 8 08:43:24 gnd-rac02 kernel: hda: ATAPI reset complete
Jun 8 11:34:34 gnd-rac02 kernel: ide-cd: cmd 0x1e timed out
Jun 8 11:34:34 gnd-rac02 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 8 11:34:34 gnd-rac02 kernel: hda: irq timeout: error=0x00
Jun 8 11:34:34 gnd-rac02 kernel: hda: ATAPI reset complete
Jun 8 12:58:48 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LOOP DOWN detected (2).
Jun 8 12:59:33 gnd-rac02 kernel: SCSI error : <3 0 0 1> return code = 0x10000
Jun 8 12:59:33 gnd-rac02 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 12:59:33 gnd-rac02 last message repeated 2 times
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP A5.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP B5.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is dead.
Jun 8 13:00:09 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:00:09 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LIP occured (f8f7).
Jun 8 13:00:09 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f7f7).
Jun 8 13:00:19 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:00:20 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LOOP UP detected (4 Gbps).
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is alive.
Jun 8 13:27:15 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LOOP DOWN detected (2).
Jun 8 13:28:00 gnd-rac02 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac02 kernel: SCSI error : <3 0 0 1> return code = 0x10000
Jun 8 13:28:00 gnd-rac02 kernel: SCSI error : <3 0 1 0> return code = 0x10000
Jun 8 13:28:00 gnd-rac02 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac02 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP A5.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP B5.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is dead.
Jun 8 13:28:34 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).

from:http://tolywang.itpub.net/post/48/464085