-----Original Message----- From: cisc...@puck.nether.net
[mailto:cisc...@puck.nether.net] On Behalf Of Hank Nussbacher Sent: mercoled? 5 gennaio 2005 7.16 To: cisc...@puck.nether.net Subject: [c-nsp] Weird IOS problem

This is a swing in the dark that perhaps someone else has seen this before. Our
7513 (w/ RSP16) running 12.2(18)S7 seems to be having a bad day. One particular
VIP4-50 (slot1) started misbehaving and it causes all other VIPs to lose their
CEF and for that particular VIP to take itself off-line!

Log output: Jan 5 07:50:57: %CBUS-3-CMDTIMEOUT: Cmd timed out, CCB 0xF800FF30, slot 1, cmd
code 2 -Traceback= 404F8334 404F8BF8 404EFF44 404ECD40 404CE974 4045C4A4
403D6B94 Jan 5 07:50:58: %DBUS-3-DBUSINTERR: Slot 1, Internal Error Jan 5
07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61 0x00000008) failed (0x8010)
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000008) failed
(0x8010) Jan 5 07:50:58: %CBUS-3-ADDRFILTR: Interface FastEthernet1/0/1,
address filter write command failed, code 0x8010 -Traceback= 404FE1F8 404FEA40
404FECD4 40403F08 40651774 409C5234 409C588C 40A09674 409E5960 40420C1C 404DDDFC 404DDF80 404E4990 404CE86C Jan 5 07:50:58:
%CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x0000FFFF) failed (0x8010) Jan 5
07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000100) failed (0x8010)
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000100) failed
(0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333)
failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00003333) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1,
cmd (36 0x00003333) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1:
Controller 1, cmd (36 0x00003333) failed (0x8010) Jan 5 07:50:59:
%OSPF-5-ADJCHG: Process 378, Nbr 192.114.22.1 on FastEthernet1/0/1 from 2WAY to DOWN, Neighbor Down: Interface down or detached
Jan 5 07:50:59: %OSPF-5-ADJCHG: Process 378, Nbr 192.114.99.52 on FastEthernet1/0/1 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 5 07:51:12: %DBUS-3-SW_NOTRDY: DBUS software not ready after HARD_RESET,
elapsed 13012, status 0x0 -Traceback= 404DE264 404E5E64 404F4620 404F7E1C
404E49B4 404CE86C Jan 5 07:51:25: %DBUS-3-SW_NOTRDY: DBUS software not ready
after HARD_RESET, elapsed 13036, status 0x0 -Traceback= 404DE264 404E5E64
404F7EA0 404E49B4 404CE86C Jan 5 07:51:38: %DBUS-3-SW_NOTRDY: DBUS software not
ready after RESET, elapsed 13036, status 0x0 -Traceback= 404DE264 404E5A78
404F7EB8 404E49B4 404CE86C Jan 5 07:51:51: %DBUS-3-SW_NOTRDY: DBUS software not
ready after RESET, elapsed 13028, status 0x0 -Traceback= 404DE264 404DEA78
404E3B44 404DC470 404DC8B4 404DCD90 404F7EE4 404E49B4 404CE86C Jan 5 07:52:02: %MDS-2-RP: MDFS is disabled on some line card(s). Use "show ip
mds stats linecard" to view status and "clear ip mds linecard" to reset. Jan 5 07:54:28: %DBUS-3-SW_NOTRDY: DBUS software not ready after DOWNLOAD,
elapsed 20024, status 0x40 -Traceback= 404DE264 404E3E60 404DC470 404DC8B4
404DCD90 404F7EE4 404E49B4 404CE86C Jan 5 07:54:28: %DBUS-3-WCSLDERR: Slot 1,
error loading WCS, status 0x40 cmd/data 0xB6 pos 4818686 Jan 5 07:54:28:
%UCODE-3-LDFAIL: Unable to download ucode from system image in slot 1, trying
rom ucode Jan 5 07:54:42: %DBUS-3-SW_NOTRDY: DBUS software not ready after
HARD_RESET, elapsed 13064, status 0x0 -Traceback= 404DE264 404E1448 404E7BDC Jan
5 07:54:55: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET, elapsed
13064, status 0x0 -Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4
404DCD90 404E1450 404E7BDC Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error,
slot/cpu 2/0: No window message, LC to RP IPC is non-operational Jan 5
07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: No window message, LC to
RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal e! rror, slot/cpu 10/0: No window message, LC to RP IPC is non-operational Jan 5
07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0: No window message, LC to
RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error,
slot/cpu 8/0: No window message, LC to RP IPC is non-operational Jan 5
07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: No window message, LC
to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error,
slot/cpu 9/0: No window message, LC to RP IPC is non-operational .Jan 5
07:57:33: %DBUS-3-SW_NOTRDY: DBUS software not ready after DOWNLOAD, elapsed
20012, status 0x40 -Traceback= 404DE264 404E3E60 404DC470 404DC8B4 404DCD90
404E1450 404E7BDC .Jan 5 07:57:33: %DBUS-3-WCSLDERR: Slot 1, error loading WCS,
status 0x40 cmd/data 0xB6 pos 4818686 .Jan 5 07:57:33: %UCODE-3-LDFAIL: Unable
to download ucode from system image in slot 1, trying rom ucode .Jan 5
07:57:33: %RSP-3-NOSTART: No microcode for VIP4-50 RM5271 card, s! lot 1 .Jan 5 07:57:34: %MDS-2-LC_FAILED_IPC_ACK: RP failed in getting Ack for IPC message of size 80 to LC in slot 1 with sequence 40250, error =
retry queue flush .Jan 5 07:57:55: %SNMP-3-AUTHFAIL: Authentication failure for
SNMP req from host 132.74.1.154 .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal
error, slot/cpu 8/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE:
Fatal error, slot/cpu 4/0: keepalive failure .Jan 5 08:00:46:
%FIB-3-FIBDISABLE: Fatal error, slot/cpu 9/0: keepalive failure .Jan 5
08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: keepalive failure .Jan
5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 10/0: keepalive failure
.Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 2/0: keepalive
failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0:
keepalive failure TAU-gp1#sho cef line Slot/CPU MsgSent XDRSent Window LowQ MedQ HighQ Flags 2 5669 160596 LC wait 0 0 0 disabled 3 5669 160598 LC wait 0 0 0 disabled 4 5738 161622 LC wait 0 0 0 disabled 8 5742 161621 LC wait 0 0 0 disabled 9 5668 160577 LC wait 0 0 0 disabled 10 5668 160574 LC wait 0 0 0 disabled 11 5668 160577 LC wait 0 0 0 disabled

VRF Default, version 311259, 153409 routes Slot/CPU Version CEF-XDR I/Fs State Flags 2 310726 157274 9 Active table-disabled 3 310726 157274 8 Active table-disabled 4 310726 157484 6 Active table-disabled 8 310726 157484 6 Active table-disabled 9 310726 157274 8 Active table-disabled 10 310726 157274 18 Active table-disabled 11 310726 157274 6 Active table-disabled TAU-gp1#show ip mds stats linecard

Slot Status IPC(seq/max/window) Q(high/route) Reloads 1 disabled 40455/44286/3831 0/0 1 2 active 258 /4354 /4096 0/0 2 3 active 267 /4363 /4096 0/0 2 4 active 284 /4380 /4096 0/0 2 8 active 239 /4335 /4096 0/0 2 9 active 275 /4371 /4096 0/0 2 10 active 231 /2279 /2048 0/0 2 11 active 249 /4345 /4096 0/0 2

Taking out VIP1 from the chassis and rebooting solves the problem. We have
already replaced the VIP in slot1 yesterday when it first happened and life went
on for about 12 hours before it happened again.

Anyone seen anything like this? Cisco output intepreter was of no help since it
indicated things that we don't run at all.

Thanks, Hank