 | Start a set with this search |
 | Include this search in one of my sets |
 | Exclude this search from one of my sets |
 | Permalink to these results Paste this link in email or IM: |
 | Atom feed for tracking future search results Paste this URL into your reader: |
16 messages in edu.merit.nanogRe: impossible circuit| From | Sent On | Attachments |
|---|---|---|
| Jon Lewis | Aug 10, 2008 8:15 pm | |
| George Carey | Aug 10, 2008 10:24 pm | |
| Laurence F. Sheldon, Jr. | Aug 11, 2008 6:27 am | |
| Justin Shore | Aug 11, 2008 1:16 pm | |
| Jay R. Ashworth | Aug 11, 2008 1:22 pm | |
| list...@pwns.ms | Aug 12, 2008 4:36 am | |
| Jon Lewis | Aug 12, 2008 7:37 am | |
| Andy Johnson | Aug 13, 2008 7:41 am | |
| Justin Shore | Aug 13, 2008 9:02 am | |
| Jon Lewis | Aug 13, 2008 9:29 am | |
| Andy Johnson | Aug 13, 2008 11:27 am | |
| Jared Mauch | Aug 13, 2008 11:33 am | |
| Jon Lewis | Aug 16, 2008 11:07 pm | |
| list...@pwns.ms | Aug 16, 2008 11:36 pm | |
| Jay Hennigan | Aug 16, 2008 11:56 pm | |
| Paul Wall | Aug 18, 2008 1:46 pm |
| Permalink for this message Paste this link in email or IM: |
| Permalink for this thread Paste this link in email or IM: |
| Atom feed for this thread Paste this URL into your reader: |
| Subject: | Re: impossible circuit | Actions... |
|---|---|---|
| From: | Jon Lewis (jle...@lewis.org) | |
| Date: | Aug 16, 2008 11:07:46 pm | |
| List: | edu.merit.nanog | |
On Tue, 12 Aug 2008, Jon Lewis wrote:
What would happen if you pinged the Ocala router such that the TTL was 1 when travelling over the DS3? From your traceroute it seems it travelled two IP hops that did not send ICMP error messages, but it might just be that the ICMP errors from the Ocala router are arriving first.
Based on where the dupes are coming from, I assume pinging across the DS3 with TTL tuned to expire at the Ocala side would result in TTL exceeded messages from both Ocala and the Sprint router where the packets are injected into Sprint's network. It doesn't look as if IOS gives the option to set TTL on ping...so I'd try this from a Linux machine in our data center.
I just went ahead and "re-broke" the circuit for a bit by turning it back to hdlc to see if the issue is still there and to run some additional tests. Someone is still cross connecting our Orlando->Ocala traffic over to Sprint.
I did your suggested ping with short TTL and the result was close to what I expected.
$ traceroute ocalflxa-br-1 traceroute to ocalflxa-br-1.atlantic.net (209.208.6.229), 30 hops max, 38 byte packets 1 209.208.25.165 (209.208.25.165) 0.539 ms 0.426 ms 0.388 ms 2 69.28.72.162 (69.28.72.162) 0.246 ms 0.351 ms 0.223 ms 3 andc-br-3-f2-0 (209.208.9.138) 0.559 ms 0.435 ms 0.471 ms 4 ocalflxa-br-1-s1-0 (209.208.112.98) 2.735 ms * 2.656 ms
So, I need a TTL of 4 to get there from this machine.
$ ping -t4 ocalflxa-br-1
PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
time=2.68 ms
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
time=2.72 ms
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252
time=2.88 ms
Decrease ttl by one, and I get the expected ttl exceeded from the Orlando side of the circuit.
$ ping -t 3 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. From andc-br-3-f2-0.atlantic.net (209.208.9.138) icmp_seq=0 Time to live exceeded
Now, here's a mild surprise. You'll notice that in the above -t4 trace, I didn't hear back from Sprint.
$ ping -t 5 ocalflxa-br-1
PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
time=2.89 ms
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
time=3.10 ms
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252
time=2.97 ms
hmm...still no ttl exceeded from Sprint?
$ ping -t 6 ocalflxa-br-1
PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
time=2.95 ms
From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=0 Time to live
exceeded
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
time=2.78 ms
From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=1 Time to live
exceeded
$ ping -t 7 ocalflxa-br-1
PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
time=2.88 ms
From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=0 Time to live
exceeded
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
time=2.84 ms
From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=1 Time to live
exceeded
Is it just coincidence that there are 2 private IP hops in some traceroutes between us and Sprint? i.e. Look at this trace from cogent:
Tracing the route to 209.208.33.1
1 fa0-8.na01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 0 msec 4 msec
4 msec
2 gi3-9.3507.core01.dca01.atlas.cogentco.com (66.28.67.225) 160 msec 4 msec 8
msec
3 te3-1.ccr02.dca01.atlas.cogentco.com (154.54.3.158) 0 msec 0 msec 4 msec
4 vl3493.mpd01.dca02.atlas.cogentco.com (154.54.7.230) 28 msec 4 msec
te4-1.mpd01.dca02.atlas.cogentco.com (154.54.2.182) 52 msec
5 vl3494.mpd01.iad01.atlas.cogentco.com (154.54.5.42) 4 msec 4 msec
vl3497.mpd01.iad01.atlas.cogentco.com (154.54.5.66) 4 msec
6 timewarner.iad01.atlas.cogentco.com (154.54.13.250) 4 msec
peer-01-ge-3-1-2-13.asbn.twtelecom.net (66.192.252.217) 4 msec 12 msec
7 66-194-200-202.static.twtelecom.net (66.194.200.202) 28 msec 28 msec 32
msec
8 66-194-200-202.static.twtelecom.net (66.194.200.202) 32 msec 32 msec 28
msec
9 andc-br-3-f2-0.atlantic.net (209.208.9.138) 32 msec 32 msec 32 msec
10 172.22.122.1 32 msec 32 msec 32 msec
11 10.247.28.205 32 msec 32 msec 32 msec
12 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 32 msec 32 msec 28 msec
13 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 28 msec 32 msec 32 msec
14 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 32 msec 32 msec 28
msec
15 vlan79.csw2.Washington1.Level3.net (4.68.17.126) 28 msec
vlan89.csw3.Washington1.Level3.net (4.68.17.190) 32 msec
vlan79.csw2.Washington1.Level3.net (4.68.17.126) 40 msec
16 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 28 msec
ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 28 msec
ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 32 msec
17 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 48 msec 48 msec 56 msec
18 ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 44 msec 48 msec
ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52 msec
19 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 56 msec 104 msec 56 msec
20 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 52 msec 52 msec 56 msec
21 unknown.Level3.net (63.209.98.66) 52 msec 52 msec 56 msec
22 andc-br-3-f2-0.atlantic.net (209.208.9.138) 52 msec 52 msec 56 msec
23 172.22.122.1 52 msec 56 msec 52 msec
24 10.247.28.205 52 msec 52 msec 56 msec
25 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 52 msec 56 msec 52 msec
26 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 56 msec 56 msec 56 msec
27 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 52 msec 52 msec 52
msec
28 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 52 msec
vlan69.csw1.Washington1.Level3.net (4.68.17.62) 56 msec
vlan89.csw3.Washington1.Level3.net (4.68.17.190) 56 msec
29 ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 64 msec
ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 52 msec 56 msec
30 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 76 msec 72 msec 72 msec
I've seen the 172.22.122.1 & 10.247.28.205 hops before. They occasionally show up in traces when the traffic is jumping over to Sprint. Sometimes they don't show up though. i.e. Tracing from my house:
traceroute to 209.208.33.1 (209.208.33.1), 30 hops max, 40 byte packets
1 172.31.0.1 (172.31.0.1) 0.336 ms 0.272 ms 0.268 ms
2 10.210.160.1 (10.210.160.1) 10.109 ms 11.719 ms 14.265 ms
3 gig7-0-4-101.orldflaabv-rtr1.cfl.rr.com (24.95.232.100) 15.302 ms 15.324
ms 16.687 ms
4 198.228.95.24.cfl.res.rr.com (24.95.228.198) 16.688 ms 18.812 ms 18.816
ms
5 te-3-3.car1.Orlando1.Level3.net (4.79.116.145) 20.084 ms 19.946 ms
te-3-1.car1.Orlando1.Level3.net (4.79.116.137) 21.328 ms
6 unknown.Level3.net (63.209.98.66) 19.900 ms 14.714 ms 14.689 ms
7 andc-br-3-f2-0.atlantic.net (209.208.9.138) 104.058 ms 11.932 ms 13.584
ms
8 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 15.872 ms 15.886 ms
17.238 ms
9 * * *
10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 41.277 ms 41.964 ms
41.955 ms
11 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 43.360 ms 44.578 ms
35.635 ms
12 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 37.035 ms 37.062 ms
33.185 ms
13 vlan89.csw3.Washington1.Level3.net (4.68.17.190) 44.060 ms 44.057 ms
vlan99.csw4.Washington1.Level3.net (4.68.17.254) 39.603 ms
14 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 38.123 ms
ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141) 39.546 ms
ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 38.115 ms
15 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 46.284 ms 46.275 ms 46.274 ms
16 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52.523 ms
ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 53.338 ms
ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 53.299 ms
17 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 34.964 ms 39.582 ms 38.088
ms
18 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 36.701 ms 38.144 ms 36.949
ms
19 unknown.Level3.net (63.209.98.66) 36.902 ms 37.750 ms 37.717 ms
20 andc-br-3-f2-0.atlantic.net (209.208.9.138) 37.729 ms 35.812 ms 35.048 ms
21 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 37.485 ms 37.601 ms
36.495 ms
22 * * *
23 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 56.459 ms 56.449 ms
57.709 ms
24 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 57.694 ms 57.692 ms
60.243 ms
25 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 103.257 ms 100.829 ms
82.571 ms
26 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 70.401 ms
vlan89.csw3.Washington1.Level3.net (4.68.17.190) 69.262 ms
vlan99.csw4.Washington1.Level3.net (4.68.17.254) 82.700 ms
27 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 74.132 ms
ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 74.135 ms
ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 75.540 ms
28 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 58.656 ms 60.838 ms 54.346 ms
29 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 59.323 ms
ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 59.336 ms
ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 63.323 ms
30 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 127.652 ms 57.884 ms
57.851 ms
From the traces I've seen, it seems if the first Sprint hop is sl-bb20-dc, the private IP hops don't show up. If the first Sprint hop is sl-crs2-dc, then the private IP hops are there. I wonder if anyone from Sprint can shed some light on that?
Unfortunately, the Sprint engineer I intitially made contact with who was helpful and seemed curious about the issue seems to have vanished and isn't returning my calls or emails. Anyone else from Sprintlink care to play?
---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________