Points to remember while troubleshooting

Network maintenance and troubleshooting methods.

Network maintenance includes:

•Troubleshooting network problems.

• Hardware and software installation/configuration.

• Monitoring and improving network performance.

• Planning for future network growth.

• Creating network documentation and keeping it up-to-date.

• Ensuring compliance with company policies.

• Ensuring compliance with legal regulations.

• Securing the network against all kind of threats.

Tasks can be performed in following ways:

1. Structured tasks.

2. Interrupt-driven tasks.

Network Maintenance models:

FCAPS:

 Fault management.

 Configuration management.

 Accounting management.

 Performance management.

 Security management.

ITIL: IT Infrastructure Library is a set of practices for IT services management that

focuses on aligning IT services with the needs of a business.

TMN: Telecommunications Management Network is another maintenance model that

was created by the ITU-T (Telecommunications Standardization Sector) and is a

variation of the FCAPS model. TMN targets management of telecommunications

networks.

Cisco Life Cycle Services: Of course Cisco has it's own network maintenance model

which defines the different phases in the life of a Cisco network:

 Prepare

 Plan

 Design

 Implement

 Operate

 Optimize

Tools for Troubleshooting

NTP

Syslog

SNMP

Wireshark

SPAN(Switched Port Analyzer)

RITE(Route IP Traffic Export)

Here are the severity levels:

0. Emergencies

1. Alerts

2. Critical

3. Errors

4. Warnings

5. Notifications

6. Informational

7. Debugging

Switching :

Check your interfaces and see if they show as up/up.

Check if an interface is in err-disabled and if so:

A) check why this happened and

B) solve the problem.

Not seeing err-disabled doesn’t automatically mean there are no port-security issues.

The default violation mode for port security is shutdown which will put the

interface in err-disabled mode. The restrict mode will keep the interface up but

shows a log message on the console. Protect mode also keeps the interface up but

doesn’t show any console messages. It’s not a bad idea to take a quick look to see if port

security is active or not…it’s also a good idea to use show mac address-table to see if the

switch learned the MAC addresses on the interfaces.

If everything else seems to be ok, make sure there’s no VACL!

Make sure you use the same encapsulation protocol when configuring trunks.

Always check if a trunk allows all VLANs or not.

If you use a multilayer switch for inter-VLAN routing make sure the SVI interfaces are configured correctly and that routing is enabled.

Make sure the interface you want to be the root port has the lowest cost path.

Check if spanning-tree is enabled or disabled.

Make sure BPDUs are not blocked or filtered between switches.

Make sure the VLAN is active on the interface before looking at spanning-tree related issues.

Make sure you use the same EtherChannel protocol on both sides.

When using PAgP make sure at least one of the switches is using desirable mode or in case of LACP make sure one switch is in active mode.

Make sure all interfaces that will be added to the port-channel have the exact same configuration!

Routing :

EIGRP:

Make sure both routers are on the same subnet.

Make sure the K-values are the same on all EIGRP routers within the same autonomous system.

Make sure the AS number is the same if you want an EIGRP neighbor adjacency.

Don’t enable passive interface if you want to establish an EIGRP neighbor adjacency.

Check if your frame-relay network supports broadcast or not. Configure EIGRP to use unicast or change your frame-relay configuration to support broadcast traffic.

If the network commands are correct, check if you have a distribute-list that is preventing prefixes from being advertised or installed in the routing table.

If EIGRP auto-summary is enabled you might end up with discontiguous networks.

EIGRP auto-summary creates an entry to the null0 interface which might prevent the installation of summaries you receive from neighbor routers.

You can’t advertise what you don’t have in your routing table.

In order for a summary route to be advertised at least one prefix that falls

within the summary, range has to be in the routing table of the advertising router!

Split horizon prevents the advertisement of a prefix out of the interface where we learned it on.

Make sure the next hop IP address is reachable and if needed add additional frame-relay map statements.

Change the administrative distance to change traffic patterns but be aware; In more complex topologies this can also cause routing loops.

When you redistribute something into EIGRP you need to configure seed metrics.

OSPF:

Make sure you have the configured correct network address, wildcard bits, and area.

Make sure OSPF is sending hello packets on an interface because otherwise, you won’t be able to become neighbors.

Don’t block OSPF multicast addresses 224.0.0.5 and 224.0.0.6 (DR/BDR).

Make sure you use the same subnet mask on routers that are directly connected to each other

Make sure you use the same OSPF authentication type and password between routers.

Make sure your OSPF routers agree on the same area number.

Make sure your OSPF routers use the same area type.

Most people have learned that OSPF always requires at least area 0 (the backbone

area). It’s perfectly fine to configure single area OSPF and to use a different area

number. Only when you connect different OSPF areas to each other you’ll need a

backbone area.

The broadcast and non-broadcast network types require a DR/BDR election. Make sure one of the routers gets elected.

The keyword broadcast means we can send broadcast and multicast traffic down the PVC.

Make sure you understand all the OSPF network types and their requirements.

Make sure you configure the correct network address, wildcard bits, and area.

You can not advertise what you don’t have!

Be aware of distribute-lists that prevent the advertising and/or installing of prefixes in the routing table.

Stub areas do not allow external prefixes (LSA Type 5). Either change the area to NSSA to stop redistributing.

If you want to advertise a default route with OSPF you need to have a default route in your routing table or use the “always” keyword.

Make sure you use the correct OSPF network type on both routers.

Add the “subnets” keyword when using redistribution or only classful networks are redistributed.

Use the administrative distance to prevent or allow the installation of prefixes in your routing table(s) and use the metric to select the best path.

Use the correct command for OSPF summarization.

BGP:

Make sure your interfaces are up and running.

Make sure the BGP routers can reach each other, that BGP packets are sourced from the correct interface and in case of EBGP don’t forget to use the multihop command.

Its common practice to configure IBGP between loopback interfaces. Make sure these loopbacks are reachable and that the BGP updates are sourced from the loopback interface.

Type in the exact correct subnet mask

If you see classful networks in your BGP table you might have auto-summary enabled.

Make sure there are no route-maps blocking the advertisement of prefixes.

IBGP neighbor adjacencies have to be full mesh! Another solution would be by using a route-reflector or confederation.

Make sure the next hop IP address is reachable so routes can be installed in the routing table and that all required networks are reachable.

Network Services:

Make sure you have the correct inside and outside interfaces.

Make sure you use the correct access-list to match your inside hosts.

Make sure your routers know how to reach the translated networks.

If everything is OK, make sure the DHCP service is running.

If you use IP helper make sure the DHCP server knows how to reach the subnet where the client is located.

Make sure preemption is enabled for HSRP if you use interface tracking.

Make sure the VRRP routers are able to reach each other.

IPv6:

Make sure IPv6 unicast-routing is enabled if you want to use router advertisements or IPv6 routing protocols.

Make sure you activate RIPNG on all interfaces if they have prefixes that you want to see advertised.

Make sure you configure a router-ID for OSPFv3.

OSPFv3 for IPv6 has the same requirements to form a neighbor adjacency as OSPFv2 for IPv4. Apply your “IPv4 OSPF” knowledge to solve neighbor adjacency issues.

Check the OSPFv3 network type and configure the neighbors using the link-local addresses. Also, make sure you have the correct frame-relay maps.

Apply the same IPv4 OSPF troubleshooting techniques to OSPFv3 after the neighbor

adjacency has been established.

Make sure you use the correct 6to4 tunnel IPv6 addresses.

TSHOOT ospf mind-map

Cisco IOS access lists: Things you should know

What is an access control list?
In the Cisco IOS, an access control list is a record that identifies and manages traffic. After identifying that traffic, an administrator can specify various events that can happen to that traffic.

What's the most common type of ACL?
IP ACLs are the most popular type of access lists because IP is the most common type of traffic. There are two types of IP ACLs: standard and extended. Standard IP ACLs can only control traffic based on the SOURCE IP address. Extended IP ACLs are far more powerful; they can identify traffic based on source IP, source port, destination IP, and destination port.

What are the most common numbers for IP ACLs?
The most common numbers used for IP ACLs are 1 to 99 for standard lists and 100 to 199 for extended lists. However, many other ranges are also possible.
Standard IP ACLs: 1 to 99 and 1300 to 1999
Extended IP ACLs: 100 to 199 and 2000 to 2699

How can you filter traffic using ACLs?
You can use ACLs to filter traffic according to the "three P's"—per protocol, per interface, and per direction. You can only have one ACL per protocol (e.g., IP or IPX), one ACL per interface (e.g., FastEthernet0/0), and one ACL per direction (i.e., IN or OUT).

How can an ACL help protect my network from viruses?
You can use an ACL as a packet sniffer to list packets that meet a certain requirement. For example, if there's a virus on your network that's sending out traffic over IRC port 194, you could create an extended ACL (such as number 101) to identify that traffic. You could then use the debug ip packet 101 detail command on your Internet-facing router to list all of the source IP addresses that are sending packets on port 194.

What's the order of operations in an ACL?
Routers process ACLs from top to bottom. When the router evaluates traffic against the list, it starts at the beginning of the list and moves down, either permitting or denying traffic as it goes. When it has worked its way through the list, the processing stops.
That means whichever rule comes first takes precedence. If the first part of the ACL denies traffic, but a lower part of the ACL allows it, the router will still deny the traffic. Let's look at an example:
Access-list 1 permit any Access-list 1 deny host 10.1.1.1 Access-list 1 deny any

What does this ACL permit?
The first line permits anything. Therefore, all traffic meets this requirement so the router will permit all traffic, and processing will then stop.

What about traffic you don't specifically address in an ACL?
At the end of an ACL is an implicit deny statement. Whether you see the statement or not, the router denies all traffic that doesn't meet a condition in the ACL. Here's an example:
Access-list 1 deny host 10.1.1.1 Access-list 1 deny 192.168.1.0 0.0.0.255

What traffic does this ACL permit?
None: The router denies all traffic because of the implicit deny statement. In other words, the ACL really looks like this:
Access-list 1 deny host 10.1.1.1 Access-list 1 deny 192.168.1.0 0.0.0.255 Access-list 1 deny ANY

Can I name an ACL?
Numbers—who needs numbers? You can also name your ACLs so you can more easily identify their purpose. You can name both standard and extended ACLs. Here's an example of using a named ACL:
router(config)# ip access-list ?   extended  Extended Access List   log-update      Control access list log updates   logging         Control access list logging   resequence      Resequence Access List   standard        Standard Access List
router(config)# ip access-list extended test
router(config-ext-nacl)# 10 deny ip any host 192.168.1.1
router(config-ext-nacl)# exit
router(config)# exit
router# show ip access-list
Extended IP access list test
10 deny ip any host 192.168.1.1

What's a numbering sequence?
In the "old days," you couldn't edit an ACL—you could only copy it to a text editor (such as Notepad), remove it, edit it in notepad, and then re-create it. In fact, this is still a good way to edit some Cisco configurations.
However, this approach can also create a security risk. During the time you've removed the ACL to modify it, the router isn't controlling traffic as needed. But it's possible to edit a numbered ACL with commands. Here's an example:
router(config)# access-list 75 permit host 10.1.1.1
router(config)#^Z
router# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
 router(config)# ip access-list standard 75
 router(config-std-nacl)# 20 permit any
router(config-std-nacl)# no 10 permit 10.1.1.1
router(config-std-nacl)#^Z
 router# show ip access-lists 75 Standard IP access list 75
    20 permit any router#

How else can I use an ACL?
ACLs aren't just for filtering traffic. You can also use them for a variety of operations. Let's look at some of their possible other uses:
To control debug output: You can use the debug list X command to control debug output. By using this command before another debug command, the command only applies to what you've defined in the list.
To control route access: You can use a routing distribute-list ACL to only permit or deny certain routes either into or out of your routing protocol.
As a BGP AS-path ACL: You can use regular expressions to permit or deny BGP routes.
For router management: You can use an ACL to control which workstation or network manages your router with an ACL and an access-class statement to your VTY lines.
For encryption: You can use ACLs to determine how to encrypt traffic. When encrypting traffic between two routers or a router and a firewall, you must tell the router what traffic to encrypt, what traffic to send unencrypted, and what traffic to drop.

What is the difference between show access-list vs show ip access-list ?
There may be more than just IPv4 access-lists on the router.   If so, the command show access-lists would show them all, including IPX or other types that may exist.
On that same router, the command show ip access-lists would only show the IP access-lists, and not the IPX or other types.

HR Interview ebook

   Interview ebook

Ethernet

#Ethernet frame calculation:
http://www.erg.abdn.ac.uk/~gorry/eg3567/lan-pages/enet-calc.html

  Ethernet

CCNP TSHOOT TIPS !!

IOS tools to monitor and maintain the network:
Show ip route 10.1.50.9
It does not show default route in output

Show  ip route 10.1.50.0 255.255.255.0 longer-prefixes
Show processes
Show processes cpu | include IP Input
Show proc cpu | excl 0.00%  0.00%  0.00%
Show ip int br | exclude unassigned
Show run | begin vty
Show run | section interface
Show run | section archive
Show processes cpu | include ^CPU|Ip Input
Show interfaces | include fastethernet | error
Show tech-support | redirect flash:showtech.txt
Dir flash:
More flash: showtech.txt
Show ip int br | tee flash: showipint.txt
Show version | append flash: show version.txt
Ping 4.4.4.4
Ping 4.2.2.2 source loopback 0
Ping 4.4.4.4 size 1500 repeat 10
Ping 4.2.2.2 size 1500 df-bit
Ping
Target address :
Repeat count :

telnet 3.3.3.3
telnet mail.ciscoblog.com 25

show memory
IOS bug leads memory leak in router
Show ip int br
Show int fa0/0
Show int fa0/0 | include drop|error
Late collision è cause duplex mismatch
Input drops : cause processor busy/utilization
Output drops : cause MTU size issue .its normal
Input and output errors are due to bad cablings and interfaces
Show inventory (to see models of cards installed)
Show diag (more detail information about each card)

SPAN: switch port analyzer
S(config)#monitor session 1(identify this instance) source int fa0/1
#monitor session 1 dest int fa0/4

RSPAN: Remote switch port analyzer
SYSLOG : allows u pipe the output to outside syslog server
R(config)# logging x.x.x.x (ip of syslog server)
#logging trap ?

SNMP
R(config)#snmp-server community cisco1  
#snmp-server ifindex persist (interface index will always remain the same even after reboot)

Netflow :
Traffic flows going in and out of device
To enable netflow
R(config)# int f0/0
#ip flow ingress
R#show ip cache flow
Netflow is kind of push system
R(config)# ip flow-export version 9
#ip flow-export destination 10.1.1.101 999 (some random UDP port)

EEM : Embedded event manager
Scripting language for your router
To sent message/notification in case somebody access your router.

VLAN and Spanning Tree Review:
Vlans are a constraint to the wiring clause. Cisco recommends of using local vlan instead of using vlan across the campus network. By doing this you are introducing the L3 device router and forwarding information through routing protocol is much faster compare to stp.
Show mac address-table
Show vlan
Show interface switchport
Show interface trunk
Traceroute mac

RSTP remembers the blocked port but STP doesn’t so once the port comes up , it puts that blocking port to forwarding state instead of going through listening, learning as in STP.

MST: grouping STP instances together.
If you have 50 vlan. one instance is running for 25 vlan and other instance is running for rest 25 vlan so we have 2 root bridge for each instance.
Show spanning-tree
Show spanning-tree interface detail
Show process cpu

TT:
Show interface status
Show vlan
Show run | inc cdp
Show run interface port-channel 1
Check both the switch are in same mode.
Show run int …

L3 switching and Redundancy protocols concepts:
In L2 :
Switch (config)# vlan 10
#name Sales

In L3 switching(router in a switch) we create the switch virtual interface
Switch (config)# interface vlan 10
# ip adddrerss 10.1.1.1
L3 Switching (vlan + routing)
Commands
Show standby brief (subst VRRP/GLBP)
Show standby (subst vrrp/glbp)
Debug standby terse (subst glbp) => shows everything except hello messages . not implemented for vrrp

L3 Switching and Redundancy protocols.
Show glbp
Show run | inc ip def
Show run | keychain
Show standby vlan 44

Route redistribution and ospf
-show ip ospf neighbors(to check routes)
-show ip ospf interface(for hello and dead timers)
-show ip ospf database(show link state database)
-Debug ip ospf adj/packets(neighbor forming process)
-Clear ip ospf proc
-debug ip routing
-seed metrics
-proper filtering
Show run | s router eigrp

Two scenarios in Ping:

1.      Request timedout: means your default gateway has forwarded the packet but some next hop router on the way doesn’t know path to a destination . So this router will drop the packet and sends icmp unreachable to default gateway but gateway never forwards it back to the sender .Hence occurs the request timed out .

2.      Icmp unreachable ie destination unreacheable: means your default gateway doesn’t know how to reach the destination so it will reply to sender with icmp unreachable message.So it is one hop story.

Basic issues that needs to be taken care while troubleshooting ospf

-          Area should match in both routers

-          Authentication key should match on both routers

-          There should not be passive-interface default

-          Be sure about the trailing spaces after the keys

-          Stub area flag should be matched

-          Network should present in the config

-          First apply the ospf authentication and then apply the key.

BGP concept overview :
-one of the slowest routing protocol
-routing protocol for internet
-meant for external use
-outbound traffic is simple and manageable but inbound traffic is complex
-used for controlling inbound and outbound traffic
-bgp runs on top of tcp (port 179)
-TCP used for reliability and keepalives
-Updates (of course) are incremental and triggered
-metric is the biggest you’ve ever seen.
-slowest routing protocol on the planet to converge.

Commands:
-show ip bgp summary
-show ip bgp
-show ip bgp neighbors
-debug ip bgp
-debug ip bgp upsates
No logging console (to get rid of console messgaes)
Show run | s prefix-list
Clear ip bgp *

Note: local pref is used for outgoing traffic and MED is used for incoming traffic.

Router Performance Issues :
Processor
Memory
Throughput

Key processes:

-          Arp input process

-          Net background process

-          Ip background process

-          Tcp timer process

Areas to check:
-Default route pointed to interface
-interface throttles, overruns, ignores
- show tcp statistics/brief
-show process cpu (history, excl 0.00%)

Troubleshooting Memory Overload
Key Symptoms:
-syslog message: sys-2-mallocfail
-show commands return blank output
-console: “unable to create exec – no memory or too many processes”
Areas to check:
-wrong IOS image (not enough memory to run)
-memory leak due to bad IOS Image (reload in)
-worm/virus focused on IOS
-BGP (show process memory)

Troubleshooting Interface utilization:
Key Symptoms:
-high cpu/memory utilization
-packet drops
- unreachable destinations

Areas to check :

-          Verify switching mode

-          Verify routing table

-          Verify cef/arp cache

Router performance:
Ping 10.1.1.1 –l 500 –t (increasing the byte size to 500) normal ping packets are 32 bytes
Show proc cpu
98%/23% means 98% is the cpu utilization out of which 23 % is caused by packets
no ip-route cache => disables fast switching
So better turn on ip-route cache under all interfaces
And enable ip cef too in global config mode.
Cef precache every destination in the routing table and makes the switching fast

Control plane-----data plane
Packets first arrives at data plane and they are moved to control plane. Router will do the processing and add a new headers and sent out of the interface.

How to summarize bgp routes
Router bgp 54000
Aggregate-address 10.16.0.0 255.240.0.0 summary-only
Access-list concept review :
Adding access list capabilities :

-          Standard

-          Extended

-          Dynamic (authentication based)

-          Established (reflexive): used specifically to restrict some traffic from internet

-          Time based: based on time period, traffic is filtered.

-          Context-based access control (CBAC) also known as IOS based firewall. It is same as reflexive but filter more specific traffic and works with UDP too.

Rules :
-list is read from top to bottom; stops at first match
-invisible implicit deny at the bottom
-ACL is applied to an interface inbound or outbound

ACL troubleshooting keys:
-show access-list
-show run interface
-know the security policy of the network
-temporarily disable security (if safe)
-verify security policy before making changes

Security TSHOOT:
To remove console session from getting terminated :
Line console 0
No exec-timeout
Show start | section line con
Not able to access the router due to authentication failed
Do the password recovery
Reload the router
Do break
Rommon 2 > confreg 0x2142
Rommon3 > reset
Would you go to initital dialog : no
Now restore original config
Copy start run
Do no shut on all the interfaces
And move back to previous config-register 0x2102
Now remove authentication on console :
Config# aaa authentication login CONSOLE none

Ipv6 tshoot:
Ipv6 addressing
-address size moved from 32 bit to 128bit (ipv6)
-to make addresses more manageable, divided into 8 groups of 4 hex characters each

Rule1 :
-eliminate groups of consecutive zeros 2001:0050::AB4:1E2B:98AA
-drop leading zeros 2001:50::AB4:1E2B:98AA

Types of Communication:
-unicast : one to one
- multicast : one to many
-anycast : one to closest

Link-local scope address : layer 2 domain
Unique /site-local scope address : organization
Global scope address : internet

Link Local address :
-assigned automatically as an IPv6 host comes online
-its similar to 169.254.X.X address of Ipv4 (when dhcp server is down, hosts will take this ip address )
This address is used to communicate with hosts in same subnets. Used in ospf neighbor discovery, cdp etc
Always begin with ‘FE80” (first 10 bits : 1111111010) followed by 54 bits of zeros
-last 64 bits is the 48-bit mac address with “FEEE” squeezed in the middle
(FE80)1111 1110 1000 0000 0000 0000 00.. |   0019.D1FF.FE22.DCF3
With Ipv6 we don’t have private ip address , all the address are global . Private address are known as site local but they it is removed as of now.

Global Address:
-have their high-level 3 bits set to 001 (2000::/3)
Nbits (global routing prefix 001..) + (64-Nbits) subnet id + 64 bits(interface id)
-global routing prefix is 48 bits or less
-subnet id is comprised of whatever bits are left over after global routing prefix
-the primary addresses expected to comprise the ipv6 internet form the 2001::/16 subnet

IPv6 static routes:
Turn on ipv6 routing by
-ipv6 unicast-routing

For static route :
-ipv6 route
IPv6 RIping:
Ipv6 unicast-routing
(global) ipv6 rip enable
(interface) ipv6 router rip
Here no need to go under rip routing mode . all the other commands are configured under interface.

OSPFv3:
Ipv6 unicast-routing
(global) ipv6 router ospf
(router) router-id
(interface) ipv6 ospf area

203-Bad number of parameters (1 with min/max=2/2)

This video shows you how to solve the 203-Bad number of parameters (1 with min/max=2/2) error which you might encounter when importing prebuilt GNS3 labs.