Cisco Prime Nerk 43 User Guide

							  
11-15
Cisco Prime Network 4.3.2 User Guide
Chapter 11      Managing Tickets with the Vision Client
  Troubleshooting a Ticket
The Advanced tab provides the ticket’s totals for the same information:
For more information about how Prime Network processes flapping events, see How Prime Network 
Correlates Incoming Events, page 10-4.
Viewing a Ticket’s Affected Parties Tab (Resource Pairs)
The Affected Parties tab lists service resources (pairs) that are affected by an event, alarm, or ticket. This 
information is only populated for events that calculate impact analysis. If it is calculated for the event, 
the tab lists of all the endpoints that are affected in the Source area and a Destination areas. This includes 
business tags and IP addresses. If the NE is an IP interface, the subinterface IP address is displayed.
The tab also reports affected status, which represents the degree of certainty that the pair will be 
impacted. Affected Status can be one of the following: 
Potential—The service might be affected (for example, rerouting may prevent any problem).
Real—The service is affected.
Recovered—The service has recovered. This state applies only to entries that were marked 
previously as potentially affected. It indicates only the fact that there is an alternate route to the 
service, regardless of the service quality level.
If any entries begin with the word Misconfigured, it means the flow has stopped unexpectedly between 
the source and destination points. (An unexpected termination point can be a routing entity, bridge, or 
VC switching entity.) Because the link does not terminate as expected, the link is not actually impacted. 
Check the configuration and status of the affected termination points to make sure there are no errors. 
As time progresses and more information is accumulated from the network, Prime Network updates the 
information to indicate which of the potentially affected parties are real or recovered.
For more information on impact analysis, including how Prime Network reports events and alarms that 
affect the same resource pairs, see How Prime Network Calculates and Reports Affected Parties (Impact 
Analysis), page 10-11. Advanced Tab Field Description
Duplication Count (For flapping) Sum of the duplication counts for all events and alarms in the 
ticket.
For example, a ticket with the following Link Down Flapping alarms on three 
different network elements would have a duplication count of 9:
NE 1: Link down -> link up -> link down -> link up -> link down -> link up
NE 2: Link down -> link up -> link down -> link up -> link down -> link up
NE 3: Link down -> link up -> link down -> link up -> link down -> link up
Reduction Count (For flapping) Sum of the reduction counts for all events and alarms in the 
ticket.
Using the previous example, the ticket would have a reduction count of 18.
Affected Devices Total number of devices affected by the ticket. To view the devices in a map, 
see the procedure in Troubleshooting a Ticket, page 11-12.
Alarm Count Total number of alarms associated with the ticket (includes the root alarm) 
						

							  
11-16
Cisco Prime Network 4.3.2 User Guide
Chapter 11      Managing Tickets with the Vision Client
  Troubleshooting a Ticket
Viewing a Ticket’s Root Cause and Associated Events (Correlation Information)
The Correlation tab displays the root cause and all of the events it caused presented in a hierarchy with 
the ticket’s root cause at the top. The ticket takes the name of the root cause. If you want to view the 
historical chronology for the events and alarms in a ticket, click the ticket’s History tab.
From here you can also launch windows for individual alarms and events. Each alarm and event will have 
its own set of tabs: Details, History, Correlation, Affected Parties, and so forth. 
Figure 11-6 shows a correlation tree for a Card Down ticket.
Figure 11-6 Correlation Tree for Card Out Ticket
Figure 11-7 illustrates the same correlation tree. The Link Down Due to Card Out alarm is the cause of 
the Interface Status Down event, and the Card Out alarm is the cause of the Link Down Due to Card Out 
alarm. The Card Out alarm is also the root cause for all of the events; thus this is a Card Out ticket. 
						

							  
11-17
Cisco Prime Network 4.3.2 User Guide
Chapter 11      Managing Tickets with the Vision Client
  Troubleshooting a Ticket
Figure 11-7 Alarms and Events in the Card Out Ticket
Finding Out How Many Devices Are Affected by a Ticket
When a fault occurs, Prime Network automatically calculates the affected devices and embeds this 
information in the ticket in the Advanced tab. If you want to get a quick visual representation of the 
affected NEs, use the Find Affected Elements feature, which is launched from the tickets table.
Right-click a ticket in the ticket pane and choose Find Affected Elements. 
If only one element is affected, it is highlighted in the Vision client map and navigation pane. 
If multiple devices are affected, they are highlighted in a list view. 
Viewing User-Entered Ticket Notes and Finding Out Who Changed the Ticket
The Notes tab can contain any free text entered by other users. Once a user has added some notes, the 
notes cannot be deleted. If used correctly, it can contain helpful information that is not automatically 
collected by Prime Network. 
The User Audit tab lists any users who have acknowledged, deacknowledged, cleared the ticket, or added 
notes to the ticket, and when the operation was performed. This is a good way to find out who may have 
already worked with this ticket.
Card Out
Link Down Due
to Card OutPort Down Due
to Card Out
Interface Status
DownInterface Status
DownInterface Status
Down
Port Down Due
to Card OutPort Down Due
to Card OutLink Down Due
to Card OutCard OutCard OutCard OutCard Out
Ticket
Alarm
361760EventInterface Status
DownInterface Status
DownAll IP Interfaces
DownInterface Status
DownInterface Status
DownInterface Status
Down 
						

							  
11-18
Cisco Prime Network 4.3.2 User Guide
Chapter 11      Managing Tickets with the Vision Client
  Troubleshooting a Ticket
Checking the Online Documentation for Ticket Troubleshooting Information
In addition to checking the Troubleshooting field in the Details tab, you may find additional information 
as follows:.
Check the correlation examples in Event Correlation Examples, page C-1. Even if the event you are 
experiencing is not described, these examples can help you understand how faults are correlated.
Check the event-specific reference documentation on Cisco.com:
Using Built-in Troubleshooting Scripts and Tools
The NE may have some built-in troubleshooting scripts that can be launched by right-clicking the NE 
and choosing Commands. Helpful scripts and commands may be available from that menu. These 
commands are documented throughout this guide under the technology or topology they apply to. For 
information on which devices and device software support the commands, see the Addendum: Additional 
VNE Support for Cisco Prime Network 4.3.2.
For example, to check device reachability, you can use the NE right-click Tools menu to run a ping or 
Telnet. These tools contact the device from the client machine.The devices that support the following 
commands are listed in the Addendum: Additional VNE Support for Cisco Prime Network 4.3.2. 
Whether you can run these commands depends on your permissions. See Permissions for Vision Client 
NE-Related Operations, page B-4.
If you are using Windows 7, you must enable the windows telnet client before you can use the Prime 
Network telnet option. The telnet communicates with the device using the telnet window from the client 
station.
To enable the windows telnet client:
Step 1From the Start menu, choose Control Panel > Turn Windows features on or off. 
The Turn Windows features on or off dialog box appears.
Step 2Check the Telnet Client check box.
Step 3Click OK. Event Types Document on Cisco.com
Notifications that are generated by Prime Network; normally you will 
find the information you need in this document.Cisco Prime Network 
Supported Service Alarms
Syslogs received from devices (IOS syslogs, ACE syslogs, Nexus 
syslogs, ASR syslogs, UCS syslogs, and so forth) and handled by 
Prime Network.Cisco Prime Network 
Supported Syslogs
SNMPv1, v2, and v3 traps received from devices (ASR traps, IOS, 
traps, MIB 2 traps, Nexus traps, CPT traps, and so forth) and handled 
by Prime Network.Cisco Prime Network 
Supported Traps
Client login and user activities related to manage the system and the 
environment (user accounts, device scopes, logging in and out, 
password issues, unit changes. 
Events concerning Prime Network components; for example, 
reachability events, database-related events, system overload 
prevention steps, and so forth. Cisco Prime Network 
Supported Security and 
System Events 
						

							  
11-19
Cisco Prime Network 4.3.2 User Guide
Chapter 11      Managing Tickets with the Vision Client
  Troubleshooting a Ticket
The devices that support the following commands are listed in the Addendum: Additional VNE Support 
for Cisco Prime Network 4.3.2. Whether you can run these commands depends on your permissions. See 
Permissions for Vision Client NE-Related Operations, page B-4.
Troubleshooting Device Reachability and Performance Issues
These topics provide some guidance for responding to problems with reachability and performance:
Checking the Device State, page 11-19
Checking Device Memory and CPU Usage, page 11-24
Checking the Device State
These topics explain how to troubleshoot reachability issues and identifying the source of a 
communication problem:
Checking the VNE Management State Badge, page 11-19
Checking the VNE Status to See If It Is an Internal Prime Network Problem, page 11-21
Checking the Communication Between the VNE and the Device, page 11-23
Checking the VNE Management State Badge
Tickets can result from device connectivity issues, or if Prime Network cannot fully discovery a device 
for various reasons. These kinds of problems are signaled by a badge at the bottom right of the device 
icon. For example, a router that is partially reachable by the Vision client is displayed as illustrated in 
Figure 11-8.  Command Navigation Description
OAM> Trace Route 
from DeviceNE> CommandsPerforms a traceroute to a destination address, 
showing how many hops were required and how 
long each hop takes.
OAM> Ping> 
Destination From DevicePings a specified IP address to see if the 
IP address is accessible.
OAM> Traceroute 
VRF
1
1. Not applicable for Cisco UBR10K and RFGW10 cards.
Logical Inventory> 
VRFs> VRF> 
CommandsPerforms a traceroute from selected VRF to a 
destination address, showing how many hops 
were required and how long each hop takes.
OAM> Ping VRF
1Pings a specified VRF to see if the VRF is 
accessible. 
						

							  
11-20
Cisco Prime Network 4.3.2 User Guide
Chapter 11      Managing Tickets with the Vision Client
  Troubleshooting a Ticket
Figure 11-8 Element with Device Partially Reachable Badge
This badge represents the VNE management state. In the Prime Network model, each device is 
represented by one Virtual Network Element (VNE) that contains a complete model of the device. VNEs 
are created by system administrators using the Administration client. After a VNE is created and started, 
Prime Network investigates the network element and automatically builds a live model of it including its 
physical and logical inventory, configuration, and status. As different VNEs build their model, a 
complete model of the network is created.
This VNE managements state badge represents:
VNE communication state, which represents the status of connectivity between the device and Prime 
Network.
VNE investigation state, which represents the extent to which the device and its components were 
discovered. Table 11-3 lists the VNE communication states and their icons. The table describes the 
default behavior. (Administrators can change the settings that determine when a device is considered 
partially or fully unreachable; refer to the Cisco Prime Network 4.3.2 Administrator Guide.) In most 
cases, rectifying these problems will require the support of your system administrator.
Table 11-4 lists the VNE investigation states, which describe the degree to which Prime Network could 
discover and model the device.
370907
Table 11-3 VNE Communication States
Badge State Name Description
Device Unreachable The connection between the VNE and the device is down because all of the protocols 
are down (though the device might be sending traps or syslogs). 
Device Partially Reachable The VNE is not fully reachable because at least one protocol is not operational.
VNE/Agent Unreachable The VNE is not responding to the gateway. This can happen if a Prime Network 
component is overutilized, connections between Prime Network components were 
lost, or the VNE is not responding in a timely fashion. (A VNE in this state does not 
mean the device is down; it might still be processing network traffic.) 
None Connecting The VNE is starting and the initial connection has not yet been made to the device. 
This is a momentary state. 
None Device Reachable All element protocols are enabled and connected. 
None Tracking Disabled The reachability detection process is not enabled for any of the protocols used by the 
VNE. The VNE will not perform reachability tests nor will Prime Network generate 
reachability-related events. (In some cases this is desirable; for example, tracking for 
Cloud VNEs should be disabled because Cloud VNEs represent unmanaged network 
segments.)
None Agent Not Loaded The VNE is not responding because it was stopped, or it was just created.  
						

							  
11-21
Cisco Prime Network 4.3.2 User Guide
Chapter 11      Managing Tickets with the Vision Client
  Troubleshooting a Ticket
Checking the VNE Status to See If It Is an Internal Prime Network Problem
To check the status of the device’s VNE, open the inventory window and click VNE Status in the 
properties pane. Figure 11-9 provides an example of a VNE properties window. This VNE is modeling 
a Cisco 3620 router. 
Table 11-4 VNE Investigation States
Badge State Name Description
Unsupported The device type is either not supported by Prime Network or is misconfigured.
Partially Discovered The VNE model is inconsistent with the device because a required device command 
failed, even after repeated retries. A common cause of this state is that the device 
contains an unsupported module. 
Currently  Unsynchronized The VNE model is inconsistent with the device; however, this is often recoverable, or 
may indicated a small inconsistency (such as a minor inventory component not being 
properly modeled). It could also be due to a more serious issue, such as an inability 
to reach a configured protocol on the device. Because this state can be due to a variety 
of reasons, check the VNE Status Details window for more information (see 
Checking the Communication Between the VNE and the Device, page 11-23). 
Discovering The VNE is building the model of the device (the device type was found and is 
supported by Cisco Prime Network). A VNE remains in this state until all device 
commands are successfully executed at least once, or until there is a discovery 
timeout. 
Maintenance VNE polling was suspended because it was manually moved to this state by an 
Administration client user. The VNE remains in this state until it is manually 
restarted. A VNE in the maintenance state has the following characteristics:
Does not poll the device or process traps and syslogs.
Maintains the status of any existing links. 
Responds to VNE reachability requests.
Passively participates in correlation flow issues (but is not an initiator). 
The VNE is moved to the Stopped state if there are changes in other Prime Network 
components (for example, Prime Network is restarted).
Shutting Down The VNE has been stopped or deleted by the user, and the VNE is terminating its 
connection to the device.
None Operational The VNE has a stable model of the device. Modeling may not be fully complete, but 
there is enough information to monitor the device and make its data available to other 
applications, such as transactions (activation workflows). A VNE remains in this state 
unless it is stopped or moved to the maintenance state, or there are device errors.
None Stopped The VNE process has terminated (it will immediately move to Defined Not Started).
None Initializing The VNE is managed and support of its device type is being validated. 
None Defined Not Started A new VNE was created (and is starting); or an existing VNE was stopped. A VNE 
remains in this state until it is started (or restarted). 
						

							  
11-22
Cisco Prime Network 4.3.2 User Guide
Chapter 11      Managing Tickets with the Vision Client
  Troubleshooting a Ticket
NoteVNE status is not the same as device status. A device may be fully reachable and operating even though 
a VNE status is Down, Unreachable, or Disconnected.
If the VNE status is down, the device may still be fully operational. This indicates a problem in Prime 
Network, not the device.
Figure 11-9 VNE Properties Window 
						

							  
11-23
Cisco Prime Network 4.3.2 User Guide
Chapter 11      Managing Tickets with the Vision Client
  Troubleshooting a Ticket
Checking the Communication Between the VNE and the Device
To check the status of the communication between the device and the VNE that is modeling it, open the 
inventory window and click VNE Details in the properties pane. Figure 11-10 provides an example of a 
VNE Status Details window.
Figure 11-10 VNE Status Details Window
The VNE Status Details window provides this information about the VNE: 
Its management connectivity state, which has to do with how the VNE was configured
The protocols the VNE is using to communicate with the device and the status of each
Whether the device is generating syslogs or traps
In the Management State area, if the Reduced Polling field is true, this means updates are driven by 
incoming events. If the Investigation State is Currently Unsynchronized, perform a manual device poll 
by clicking Poll Now in the inventory window.
For more information on this topic, see the Cisco Prime Network 4.3.2 Administrator Guide.  
						

							  
11-24
Cisco Prime Network 4.3.2 User Guide
Chapter 11      Managing Tickets with the Vision Client
  Troubleshooting a Ticket
Using Ping, Telnet, and Trace Route
To check device reachability, you can use the NE right-click Tools menu to run a ping or Telnet. These 
tools contact the device from the client machine. The devices that support these commands are listed in 
the Addendum: Additional VNE Support for Cisco Prime Network 4.3.2. Whether you can run these 
commands depends on your permissions. See Permissions for Vision Client NE-Related Operations, 
page B-4.
NoteIf you are using Windows 7, you must enable the windows telnet client before you can use the Prime 
Network telnet option. See Using Built-in Troubleshooting Scripts and Tools, page 11-18.
Checking Device Memory and CPU Usage 
The Vision client provides a tool that displays memory and CPU usage information for a device or 
network element, including its history. To open the CPU usage graph:
Step 1Right-click a network element in the navigation tree and choose Tools > CPU Usage. 
The CPU Usage dialog box displays the following information: 
CPU Usage—The CPU usage rate as a percentage.
CPU Usage History—The CPU usage rate history is graphically displayed.
Memory Usage—The memory usage rate as a percentage.
Memory Usage History—The memory usage rate history is graphically displayed.
Step 2If desired, click Save to CSV File to export the displayed data. 
Step 3Click the upper right corner to close the CPU Usage dialog box.
Prime Network also provides a web-based Monitoring tool for administrators that tracks how the 
gateway, units, and individual AVMs are operating—Java heap, dropped messages, CPU usage, and so 
forth. This information is provided in graphical form and you can use it to locate and diagnose problems. 
This tool is described in the Cisco Prime Network 4.3.2 Administrator Guide. Command Navigation Description
OAM> Trace Route 
from DeviceNE> CommandsPerforms a traceroute to a destination address, 
showing how many hops were required and how 
long each hop takes.
OAM> Ping> 
Destination From DevicePings a specified IP address to see if the 
IP address is accessible.
OAM> Traceroute 
VRF
1
1. Not applicable for Cisco UBR10K and RFGW10 cards.
Logical Inventory> 
VRFs> VRF> 
CommandsPerforms a traceroute from selected VRF to a 
destination address, showing how many hops 
were required and how long each hop takes.
OAM> Ping VRF
1Pings a specified VRF to see if the VRF is 
accessible.