When working with the Cisco ASR router platform (or any router for that matter) you may run into situations where the router had some sort of issue and crashed.
I know what your thinking – “ROUTERS AREN”T SUPPOSED TO CRASH!!”
This is true and they usually don’t crash.
But every once in a while you will run into one that does. When this happens you will need to troubleshoot it to figure out what type of crash it was and retrieve the relevant information about the crash to open a support case on the issue.
Types of ASR-1006 Crashes
There are several types of crashes in an ASR-1006. Crashes can occur on the different modules RP (route processor), SIP (SPA interface processor) and ESP (embedded services processor).
The table below lists out some of the common crashes you can experience.
|
Type of Crashes |
Module |
Description |
|
IOSD Crash |
RP |
Cisco IOS Software runs as IOSD on a Linux kernel on RP. |
|
SPA Driver Crash |
SIP |
Limited Cisco IOS Software runs to control SPA on SIP. |
|
Cisco IOS XE Process Crash |
RP, ESP, SIP |
Several Cisco IOS XE Processes run on a Linux kernel. For example, the chassis manager, the forwarding manager, interface manager, and so on run on RP. |
|
Cisco Quantum Flow Processor Microcode Crash |
ESP |
The microcode runs on QFP. QFP is a packet forwarding ASICs on ESP. |
|
Linux Kernel Crash |
RP |
Linux kernel runs on RP, ESP, and SIP |
Getting the Crash Information
We can retrieve information related to a crash using this:
- Logs
- Syslogs
- Show platform (Provide health of the different modules in the ASR)
- Crash info file
- Core dump file
Crash info file and core dump file are stored in bootflash and hard disk.
- Get a show tech and look for the word “crashinfo” to locate the file.
- You can also use the below commands to look for crashes and pay attention on current dates that match with the time of the event and also it is necessary to identify the core dump files. These are important when working with cisco TAC.
sh bootflash:
sh harddisk:
sh stby-bootflash: (when having ASR running SSO)
sh stby-harddisk: (when having ASR running SSO)
Crash files are visible under this nomenclature:
For bootflash:
Router#sh bootflash: | inc crash
17 308367 Jun 25 2011 11:11:35 +00:00 /bootflash/crashinfo_RP_00_00_20110625-111135-GMT
18 340647 Sep 25 2011 22:34:04 +00:00 /bootflash/crashinfo_RP_00_00_20110925-223403-GMT
19 314308 Oct 24 2012 20:49:56 +00:00 /bootflash/crashinfo_RP_00_00_20121024-204955-GMT
Router#sh stby-bootflash:
Router#sh stby-bootflash: | inc crash
7 302084 Jul 28 2011 12:24:33 +00:00 /bootflash/crashinfo_RP_01_00_20110728-122433-GMT
8 314783 Sep 09 2011 04:42:46 +00:00 /bootflash/crashinfo_RP_01_00_20110909-044246-GMT
9 306378 Sep 27 2011 21:03:57 +00:00 /bootflash/crashinfo_RP_01_00_20110927-210357-GMT
For hardisk:
Router#sh stby-harddisk: | inc core
363 4096 Sep 27 2011 21:06:02 +00:00 /harddisk/core
364 106324543 Jul 28 2011 12:26:34 +00:00 /harddisk/core/Router_RP_1_linux_iosd-imag_25039.core.gz
365 105225466 Sep 09 2011 04:44:48 +00:00 /harddisk/core/Router_RP_1_linux_iosd-imag_24883.core.gz
366 104067438 Sep 27 2011 21:06:02 +00:00 /harddisk/core/Router_RP_1_linux_iosd-imag_25480.core.gz
Router#sh harddisk: | inc core
313 4096 Oct 24 2012 20:52:19 +00:00 /harddisk/core
314 94310102 Jun 25 2011 11:13:33 +00:00 /harddisk/core/Router_RP_0_linux_iosd-imag_26418.core.gz
315 111456840 Sep 25 2011 22:36:10 +00:00 /harddisk/core/Router_RP_0_linux_iosd-imag_27115.core.gz
316 148005287 Oct 24 2012 20:52:19 +00:00 /harddisk/core/Router_RP_0_linux_iosd-imag_26432.core.gz
Identifying IOSD Crashes
When the IOSD crashes, the crash info file and core dump file are generated on the RP.
Router#dir bootflash:
Directory of bootflash:/
11 drwx 16384 Jun 8 2011 12:39:34 +00:00 lost+found
12 -rw- 403260776 Jun 8 2011 12:41:45 +00:00 asr1000rp2-adventerprisek9.03.02.00.S.151-1.S.bin
48193 drwx 4096 Oct 25 2012 09:32:50 +00:00 .prst_sync
160641 drwx 4096 Oct 25 2012 10:13:47 +00:00 .rollback_timer
64257 drwx 4096 Oct 25 2012 10:00:38 +00:00 .installer
13 -rw- 308367 Jun 25 2011 11:11:35 +00:00 crashinfo_RP_00_00_20110625-111135-GMT
14 -rw- 340647 Sep 25 2011 22:34:04 +00:00 crashinfo_RP_00_00_20110925-223403-GMT
15 -rw- 314308 Oct 24 2012 20:49:56 +00:00 crashinfo_RP_00_00_20121024-204955-GMT
16 -rw- 443599048 Oct 24 2012 23:28:33 +00:00 asr1000rp2-adventerprisek9.03.05.02.S.152-1.S2.bin
Router#dir harddisk:core
Directory of harddisk:/core/
475138 -rw- 94310102 Jun 25 2011 11:13:33 +00:00 Router_RP_0_linux_iosd-imag_26418.core.gz
475139 -rw- 111456840 Sep 25 2011 22:36:10 +00:00 Router_RP_0_linux_iosd-imag_27115.core.gz
475140 -rw- 148005287 Oct 24 2012 20:52:19 +00:00 Router_RP_0_linux_iosd-imag_26432.core.gz
78704144384 bytes total (74333220864 bytes free)
Identifying SPA Driver Crashes
Router#dir harddisk:
Directory of harddisk:/
14 -rw- 224579 Aug 28 2008 08:52:06 +00:00
crashinfo_SIP_00_00_20080828-085206-UTC
Router#dir harddisk:core
Directory of harddisk:/core/
4653060 -rw- 1389762 Aug 28 2008 08:52:12 +00:00
Router_SIP_0_mcpcc-lc-ms_6985.core.gz
Identifying IOS XE Process Crashes
It is necessary to look for the below messages on the router’s logs:
Jan 24 23:37:06.644 JST: %PMAN-3-PROCHOLDDOWN: F0: pman.sh: The process cpp_cp_svr has been helddown (rc 134)
Jan 24 23:37:06.727 JST: %PMAN-0-PROCFAILCRIT: F0: pvp.sh: A critical processcpp_cp_svr has failed (rc 134)
Jan 24 23:37:11.539 JST: %ASR1000_OIR-6-OFFLINECARD: Card (fp) offline in slot F0
Router#dir harddisk:core
Directory of harddisk:/core/
1032194 -rw- 38255956 Jan 24 2009 23:37:06 +09:00
Router_ESP_0_cpp_cp_svr_4714.core.gz
Router#dir harddisk:tracelogs/cpp_cp*
Directory of harddisk:tracelogs/
4456753 -rwx 24868 Jan 24 2009 23:37:15 +09:00
cpp_cp_F0-0.log.4714.20090124233714
Identifying Cisco Quantum Flow Processor Microcode Crashes
It is necessary to look for the below messages on the router’s logs:
Dec 17 05:50:26.417 JST: %IOSXE-3-PLATFORM: F0: cpp_cdm: CPP crashed, core file /tmp/corelink/Router_ESP_0_cpp-mcplo-ucode_121708055026.core.gz
Dec 17 05:50:28.206 JST: %ASR1000_OIR-6-OFFLINECARD: Card (fp) offline in slot F0
You can find the core dump file.
Router#dir harddisk:core
Directory of harddisk:core/
3719171 -rw- 1572864 Dec 17 2008 05:50:31 +09:00
Router_ESP_0_cpp-mcplo-ucode_121708055026.core.gz
Identifying Linux Kernel Crashed
Router#dir harddisk:core
Directory of harddisk:/core/
393230 —- 137389415 Dec 19 2008 01:19:40 +09:00
Router_RP_0_kernel_20081218161940.core
List of useful troubleshooting commands
- show platform
- show environment all
- show redundancy
- show redundancy states
- show redundancy history
- show facility-alarm status
- clear facility-alarm
There is never a good time for a router to crash. But when it does, you need to be able to find out as much information about the crash as possible into order to help you understand the crash and to make sure the crash doesn’t happen again!
What is your experience with router crashes? Do you have any tips of your own on how to handle router crashes? Leave a comment below and share your story…
