How To Troubleshoot Crashes on an ASR-1006 Router

how to troubleshoot a crash

how to troubleshoot a crashWhen working with the Cisco ASR router platform (or any router for that matter) you may run into situations where the router had some sort of issue and crashed.

I know what your thinking – “ROUTERS AREN”T SUPPOSED TO CRASH!!”

This is true and they usually don’t crash.

But every once in a while you will run into one that does.  When this happens you will need to troubleshoot it to figure out what type of crash it was and retrieve the relevant information about the crash to open a support case on the issue.

Types of ASR-1006 Crashes

There are several types of crashes in an ASR-1006. Crashes can occur on the different modules RP (route processor), SIP (SPA interface processor) and ESP (embedded services processor).

The table below lists out some of the common crashes you can experience.

Type of Crashes

Module

Description

IOSD Crash

RP

Cisco IOS Software runs as IOSD on a Linux kernel on RP.

SPA Driver Crash

SIP

Limited Cisco IOS Software runs to control SPA on SIP.

Cisco IOS XE Process Crash

RP, ESP, SIP

Several Cisco IOS XE Processes run on a Linux kernel. For example, the chassis manager, the forwarding manager, interface manager, and so on run on RP.

Cisco Quantum Flow Processor Microcode Crash

ESP

The microcode runs on QFP. QFP is a packet forwarding ASICs on ESP.

Linux Kernel Crash

RP

Linux kernel runs on RP, ESP, and SIP

 

Getting the Crash Information

We can retrieve information related to a crash using this:

  • Logs
  • Syslogs
  • Show platform (Provide health of the different modules in the ASR)
  • Crash info file
  • Core dump file

Crash info file and core dump file are stored in bootflash and hard disk.

  • Get a show tech and look for the word “crashinfo” to locate the file.
  • You can also use the below commands to look for crashes and pay attention on current dates that match with the time of the event and also it is necessary to identify the core dump files. These are important when working with cisco TAC.

sh bootflash:

sh harddisk:

sh stby-bootflash: (when having ASR running SSO)

sh stby-harddisk: (when having ASR running SSO)

 

Crash files are visible under this nomenclature:

For bootflash:

Router#sh bootflash: | inc crash

17     308367 Jun 25 2011 11:11:35 +00:00 /bootflash/crashinfo_RP_00_00_20110625-111135-GMT

18     340647 Sep 25 2011 22:34:04 +00:00 /bootflash/crashinfo_RP_00_00_20110925-223403-GMT

19     314308 Oct 24 2012 20:49:56 +00:00 /bootflash/crashinfo_RP_00_00_20121024-204955-GMT

Router#sh stby-bootflash:

 

Router#sh stby-bootflash: | inc crash

7     302084 Jul 28 2011 12:24:33 +00:00 /bootflash/crashinfo_RP_01_00_20110728-122433-GMT

8     314783 Sep 09 2011 04:42:46 +00:00 /bootflash/crashinfo_RP_01_00_20110909-044246-GMT

9     306378 Sep 27 2011 21:03:57 +00:00 /bootflash/crashinfo_RP_01_00_20110927-210357-GMT

 

For hardisk:

Router#sh stby-harddisk: | inc  core

363       4096 Sep 27 2011 21:06:02 +00:00 /harddisk/core

364  106324543 Jul 28 2011 12:26:34 +00:00 /harddisk/core/Router_RP_1_linux_iosd-imag_25039.core.gz

365  105225466 Sep 09 2011 04:44:48 +00:00 /harddisk/core/Router_RP_1_linux_iosd-imag_24883.core.gz

366  104067438 Sep 27 2011 21:06:02 +00:00 /harddisk/core/Router_RP_1_linux_iosd-imag_25480.core.gz

 

Router#sh harddisk: | inc  core

313       4096 Oct 24 2012 20:52:19 +00:00 /harddisk/core

314   94310102 Jun 25 2011 11:13:33 +00:00 /harddisk/core/Router_RP_0_linux_iosd-imag_26418.core.gz

315  111456840 Sep 25 2011 22:36:10 +00:00 /harddisk/core/Router_RP_0_linux_iosd-imag_27115.core.gz

316  148005287 Oct 24 2012 20:52:19 +00:00 /harddisk/core/Router_RP_0_linux_iosd-imag_26432.core.gz

 

Identifying IOSD Crashes

When the IOSD crashes, the crash info file and core dump file are generated on the RP.

Router#dir bootflash:

Directory of bootflash:/

 

11  drwx       16384   Jun 8 2011 12:39:34 +00:00  lost+found

12  -rw-   403260776   Jun 8 2011 12:41:45 +00:00  asr1000rp2-adventerprisek9.03.02.00.S.151-1.S.bin

48193  drwx        4096  Oct 25 2012 09:32:50 +00:00  .prst_sync

160641  drwx        4096  Oct 25 2012 10:13:47 +00:00  .rollback_timer

64257  drwx        4096  Oct 25 2012 10:00:38 +00:00  .installer

13  -rw-      308367  Jun 25 2011 11:11:35 +00:00  crashinfo_RP_00_00_20110625-111135-GMT

14  -rw-      340647  Sep 25 2011 22:34:04 +00:00  crashinfo_RP_00_00_20110925-223403-GMT

15  -rw-      314308  Oct 24 2012 20:49:56 +00:00  crashinfo_RP_00_00_20121024-204955-GMT

16  -rw-   443599048  Oct 24 2012 23:28:33 +00:00  asr1000rp2-adventerprisek9.03.05.02.S.152-1.S2.bin

 

Router#dir harddisk:core

Directory of harddisk:/core/

 

475138  -rw-    94310102  Jun 25 2011 11:13:33 +00:00  Router_RP_0_linux_iosd-imag_26418.core.gz

475139  -rw-   111456840  Sep 25 2011 22:36:10 +00:00  Router_RP_0_linux_iosd-imag_27115.core.gz

475140  -rw-   148005287  Oct 24 2012 20:52:19 +00:00  Router_RP_0_linux_iosd-imag_26432.core.gz

 

78704144384 bytes total (74333220864 bytes free)

 

Identifying SPA Driver Crashes

Router#dir harddisk:

Directory of harddisk:/

 

14  -rw-      224579  Aug 28 2008 08:52:06 +00:00

crashinfo_SIP_00_00_20080828-085206-UTC

 

Router#dir harddisk:core

Directory of harddisk:/core/

 

4653060  -rw-     1389762  Aug 28 2008 08:52:12 +00:00

Router_SIP_0_mcpcc-lc-ms_6985.core.gz

 

Identifying IOS XE Process Crashes

It is necessary to look for the below messages on the router’s logs:

Jan 24 23:37:06.644 JST: %PMAN-3-PROCHOLDDOWN: F0: pman.sh:  The process cpp_cp_svr has been helddown (rc 134)

Jan 24 23:37:06.727 JST: %PMAN-0-PROCFAILCRIT: F0: pvp.sh: A critical processcpp_cp_svr has failed (rc 134)

Jan 24 23:37:11.539 JST: %ASR1000_OIR-6-OFFLINECARD: Card (fp) offline in slot F0

 

Router#dir harddisk:core

Directory of harddisk:/core/

 

1032194  -rw-    38255956  Jan 24 2009 23:37:06 +09:00

Router_ESP_0_cpp_cp_svr_4714.core.gz

 

Router#dir harddisk:tracelogs/cpp_cp*

Directory of harddisk:tracelogs/

 

4456753  -rwx       24868  Jan 24 2009 23:37:15 +09:00

cpp_cp_F0-0.log.4714.20090124233714

 

Identifying Cisco Quantum Flow Processor Microcode Crashes

It is necessary to look for the below messages on the router’s logs:

Dec 17 05:50:26.417 JST: %IOSXE-3-PLATFORM: F0: cpp_cdm: CPP crashed, core file /tmp/corelink/Router_ESP_0_cpp-mcplo-ucode_121708055026.core.gz

Dec 17 05:50:28.206 JST: %ASR1000_OIR-6-OFFLINECARD: Card (fp) offline in slot F0

 

You can find the core dump file.

Router#dir harddisk:core

Directory of harddisk:core/

 

3719171  -rw-     1572864  Dec 17 2008 05:50:31 +09:00

Router_ESP_0_cpp-mcplo-ucode_121708055026.core.gz

 

 

Identifying Linux Kernel Crashed

Router#dir harddisk:core

Directory of harddisk:/core/

 

393230  —-   137389415  Dec 19 2008 01:19:40 +09:00

Router_RP_0_kernel_20081218161940.core

 

List of useful troubleshooting commands

  • show platform
  • show environment all
  • show redundancy
  • show redundancy states
  • show redundancy history
  • show facility-alarm status
  • clear facility-alarm

 

There is never a good time for a router to crash. But when it does, you need to be able to find out as much information about the crash as possible into order to help you understand the crash and to make sure the crash doesn’t happen again!

What is your experience with router crashes?  Do you have any tips of your own on how to handle router crashes?  Leave a comment below and share your story…

 

Leave a Reply

Your email address will not be published. Required fields are marked *

Share this article.

Recommended
Noction Ad
Popular Articles

More Articles