 | Start a set with this search |
 | Include this search in one of my sets |
 | Exclude this search from one of my sets |
 | Permalink to these results Paste this link in email or IM: |
 | Atom feed for tracking future search results Paste this URL into your reader: |
16 messages in net.nether.puck.cisco-nsp[c-nsp] Better way of finding out the...| From | Sent On | Attachments |
|---|---|---|
| Dave Temkin | Jan 27, 2005 7:42 am | |
| Joe Maimon | Jan 27, 2005 8:17 am | |
| Dave Temkin | Jan 27, 2005 8:27 am | |
| Jason Ackley | Jan 27, 2005 8:43 am | |
| Dave Temkin | Jan 27, 2005 8:44 am | |
| Rodney Dunn | Jan 27, 2005 8:54 am | |
| Dave Temkin | Jan 27, 2005 8:57 am | |
| Rodney Dunn | Jan 27, 2005 9:09 am | |
| Dave Temkin | Jan 27, 2005 9:14 am | |
| Dave Temkin | Jan 27, 2005 9:17 am | |
| Rodney Dunn | Jan 27, 2005 9:21 am | |
| Dave Temkin | Jan 27, 2005 9:28 am | |
| Rodney Dunn | Jan 27, 2005 9:33 am | |
| Dave Temkin | Feb 1, 2005 6:47 pm | |
| Rodney Dunn | Feb 1, 2005 7:30 pm | |
| Dave Temkin | Feb 1, 2005 8:51 pm |
| Permalink for this message Paste this link in email or IM: |
| Permalink for this thread Paste this link in email or IM: |
| Atom feed for this thread Paste this URL into your reader: |
| Subject: | [c-nsp] Better way of finding out the source of process switched traffic? | Actions... |
|---|---|---|
| From: | Dave Temkin (da...@ordinaryworld.com) | |
| Date: | Jan 27, 2005 9:17:47 am | |
| List: | net.nether.puck.cisco-nsp | |
As far as what's getting punted: bala-choke-1#sh int stat Load for five secs: 21%/12%; one minute: 18%; five minutes: 18% Time source is NTP, 09:16:33.362 EST Thu Jan 27 2005
FastEthernet0/0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 11 713 15 976 Route cache 13 6882 13 11586 Total 24 7595 28 12562 FastEthernet1/0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 0 0 2 122 Route cache 9 8603 13 6830 Total 9 8603 15 6952 FastEthernet2/0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 0 0 0 0 Route cache 2 1651 0 0 Total 2 1651 0 0 bala-choke-1#sh int | incl protocol|bits/sec FastEthernet0/0 is up, line protocol is up 30 second input rate 2785000 bits/sec, 1379 packets/sec 30 second output rate 7066000 bits/sec, 1396 packets/sec FastEthernet0/0.101 is up, line protocol is up FastEthernet0/0.103 is up, line protocol is up FastEthernet0/0.110 is up, line protocol is up FastEthernet0/0.111 is up, line protocol is up FastEthernet0/0.800 is up, line protocol is up FastEthernet0/0.801 is up, line protocol is up FastEthernet0/0.802 is up, line protocol is up FastEthernet1/0 is up, line protocol is up 30 second input rate 4767000 bits/sec, 977 packets/sec 30 second output rate 2730000 bits/sec, 1362 packets/sec FastEthernet2/0 is up, line protocol is up 5 minute input rate 2096000 bits/sec, 359 packets/sec 5 minute output rate 2000 bits/sec, 4 packets/sec
bala-choke-1#sh int | incl protocol|bits/sec FastEthernet0/0 is up, line protocol is up 30 second input rate 2491000 bits/sec, 1353 packets/sec 30 second output rate 7686000 bits/sec, 1407 packets/sec FastEthernet0/0.101 is up, line protocol is up FastEthernet0/0.103 is up, line protocol is up FastEthernet0/0.110 is up, line protocol is up FastEthernet0/0.111 is up, line protocol is up FastEthernet0/0.800 is up, line protocol is up FastEthernet0/0.801 is up, line protocol is up FastEthernet0/0.802 is up, line protocol is up FastEthernet1/0 is up, line protocol is up 30 second input rate 5413000 bits/sec, 1029 packets/sec 30 second output rate 2440000 bits/sec, 1340 packets/sec FastEthernet2/0 is up, line protocol is up 5 minute input rate 2250000 bits/sec, 379 packets/sec 5 minute output rate 4000 bits/sec, 7 packets/sec bala-choke-1#sh int stat Load for five secs: 10%/6%; one minute: 17%; five minutes: 18% Time source is NTP, 09:17:54.241 EST Thu Jan 27 2005
FastEthernet0/0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 19640 1331558 19330 3660910 Route cache 90014 22901439 96587 75927787 Total 109654 24232997 115917 79588697 FastEthernet1/0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 12372 2158585 18889 1202885 Route cache 68798 49543259 89661 22469401 Total 81170 51701844 108550 23672286 FastEthernet2/0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 6499 1398433 341 23855 Route cache 27813 25999911 353 71982 Total 34312 27398344 694 95837
On Thu, 27 Jan 2005, Dave Temkin wrote:
Oddly enough - it's all HTTP traffic - and it seems perfectly normal.. And it's not going to one of the NAT'ed addresses.
bala-choke-1#show buff input-interface f2/0 packet
Public particle pools:
Private particle pools:
bala-choke-1#show buff input-interface f2/0 packet
Buffer information for Small buffer at 0x63FCBB14 data_area 0xF25E824, refcount 1, next 0x63FC7374, flags 0x280 linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1 if_input 0x638875AC (FastEthernet2/0), if_output 0x0 (None) inputtime 31w5d (elapsed 00:00:04.648) outputtime 00:00:00.000 (elapsed never), oqnumber 65535 datagramstart 0xF25E86A, datagramsize 60, maximum size 260 mac_start 0xF25E86A, addr_start 0xF25E86A, info_start 0x0 network_start 0xF25E878, transport_start 0xF25E88C, caller_pc 0x608957C4
source: 166.90.137.22, destination: y.y.y.y, id: 0x1C71, ttl: 56, TOS: 0 prot: 6, source port 80, destination port 47965
0F25E860: 0000 0C07AC66 ....,f 0F25E870: 00021630 A5810800 45000028 1C714000 ...0%...E..(.q at . 0F25E880: 380603B6 A65A8916 8DA26596 0050BB5D 8..6&Z..."e..P;] 0F25E890: 0EDE0396 BA2C0A36 50111B48 DF5D0000 .^..:,.6P..H_].. 0F25E8A0: 00000000 000005 .......
bala-choke-1#show buff input-interface f2/0 packet
Buffer information for Big buffer at 0x6420A0CC data_area 0xF292BC4, refcount 1, next 0x0, flags 0x280 linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1 if_input 0x638875AC (FastEthernet2/0), if_output 0x0 (None) inputtime 00:00:00.000 (elapsed never) outputtime 00:00:00.000 (elapsed never), oqnumber 65535 datagramstart 0xF292C0A, datagramsize 1253, maximum size 1692 mac_start 0xF292C0A, addr_start 0xF292C0A, info_start 0x0 network_start 0xF292C18, transport_start 0xF292C2C, caller_pc 0x608957C4
source: 63.211.153.3, destination: y.y.y.x, id: 0x66D2, ttl: 119, TOS: 0 prot: 6, source port 80, destination port 61477
0F292C00: 0000 0C07AC66 ....,f 0F292C10: 00021630 A5810800 450004D7 66D24000 ...0%...E..WfR at .
-Dave
On Thu, 27 Jan 2005, Rodney Dunn wrote:
Tidbit: When you are collecting command outputs to help troubleshoot enable this for your terminal session:
terminal exec prompt timestamp
Then do:
clear counters sh int stat sh int | incl protocol|bits/sec
wait 30 seconds
clear counters sh int stat sh int | incl protocol|bits/sec
what percentage of the traffic is being punted to process level?
Run the 'sh buff' commands like I gave you and let's see a few of the packets.
Rodney
tOn Thu, Jan 27, 2005 at 08:58:05AM -0500, Dave Temkin wrote:
Thanks Rodney.
The one thing I'm hesitant to blame it on is the fact that the actual NAT'ed traffic is very very little (it's AIM conversations, that's it). So I'm wondering why on a box that's as big as this one (NPE-400) it'd choke on that...
-Dave
-- David Temkin
On Thu, 27 Jan 2005, Rodney Dunn wrote:
Your problem is almost surely that the packets being punted are TCP control packets where we punt to create/tear down the translations. SYN, FIN, RST.
If you want to see the packets at process level you can either turn on: debug ip packet <acl> to limit the granularity of the debug since that only prints packets at process level. /*not true for 12.2S with the right commands*/
You can also do "sh buffers input-interface <name> packet" a few times and catch some packets in the buffer and manually decode the TCP header to see if the flags are set in the header.
Now, in 12.3(4)T and later we made some NAT enhancements where we create the flows in the CEF path without punting traffic. That is the suggested way to go if you are seeing a high number of process switched traffic with NAT enabled.
Rodney
On Thu, Jan 27, 2005 at 07:42:33AM -0500, Dave Temkin wrote:
I've got an internet-facing router that's seeing a very high rate of process switched traffic. Nothing too crazy is configured on this router - a little bit of NAT, a couple of route maps, BGP. That's about it. Aside from doing a debug ip packet and killing the router (it's passing about 30-40mbit of traffic), are there any other options for tracking down what's in the process queue? Router is running 12.3.6a
FastEthernet0/0 Throttle count 4 Drops RP 5 SP 0 SPD Flushes Fast 3103 SSE 0 SPD Aggress Fast 0 SPD Priority Inputs 83215964 Drops 0
Protocol IP Switching path Pkts In Chars In Pkts Out Chars Out Process 1803602701 4025634609 1661069368 456573125 Cache misses 0 - - - Fast 2713542052 1802705001 3837108389 304620460 Auton/SSE 0 0 0 0
FastEthernet1/0 Outside Throttle count 0 Drops RP 0 SP 0 SPD Flushes Fast 1796 SSE 0 SPD Aggress Fast 0 SPD Priority Inputs 6927146 Drops 0
Protocol IP Switching path Pkts In Chars In Pkts Out Chars Out Process 543622379 2397426796 317919218 1743487367 Cache misses 0 - - - Fast 3071349692 1923264716 1211037578 2505497398 Auton/SSE 0 0 0 0
FastEthernet2/0 Outside 2 Throttle count 0 Drops RP 0 SP 0 SPD Flushes Fast 1480 SSE 0 SPD Aggress Fast 0 SPD Priority Inputs 42435822 Drops 0
Protocol IP Switching path Pkts In Chars In Pkts Out Chars Out Process 1056561152 1934756549 1414955036 1939153311 Cache misses 0 - - - Fast 819752907 318162363 1502399074 1302221329 Auton/SSE 0 0 0 0
! interface FastEthernet0/0.101 encapsulation dot1Q 101 ip address x.x.x.x x.x.x.x.x no ip redirects no ip proxy-arp ip nat inside ip policy route-map RM101 no cdp enable standby 101 ip x.x.x.y standby 101 timers 1 3 standby 101 priority 250 standby 101 preempt standby 101 name HSRP101 !
! interface FastEthernet1/0 description Outside 1 ip address x.x.x.x x.x.x.x ip access-group Yipes-Outside in ip nat outside load-interval 30 duplex full ntp disable hold-queue 300 in hold-queue 300 out
-Dave
_______________________________________________ cisco-nsp mailing list cisco-nsp at puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/