I was recently asked to help troubleshoot a FTP problem that a customer was having. They have a FTP transaction that they do once a day, and a few days ago, it stopped working. So, of course, the remote end says that everyone else is working and that it must be my customer. So, I have to prove that the traffic is going out to the remote end. Afterall, when you are dealing with larger companies like this one, you always seem to have to do the troubleshooting and proving that your side is ok.
On our ASA, I want to capture the packets actually leaving the firewall. Thats the only way Im going to be able to prove that its leaving our site. Here we go.
First, I have to come up with a capture. I use my ACL to define the traffic I want to see, and then apply it to the OUTSIDE interface.
ASA(config)# access-list 194 permit ip host 20.20.20.194 40.40.40.0 255.255.255.0
ASA(config)# capture capin interface outside access-list 194
ASA(config)#
Now that the config is in, lets have my customer test.
First Try, gets to login but nothing after:
ASA# sh capture capin
3 packets captured
1: 10:44:39.319960 20.20.20.194.33201 > 40.40.40.12.21: S 2922611106:2922611106(0) win 8192 <mss 1380,nop,wscale 0,nop,nop,sackOK>
2: 10:44:39.822894 20.20.20.194.33201 > 40.40.40.12.21: S 2086380860:2086380860(0) win 8192 <mss 1380,nop,wscale 0,nop,nop,sackOK>
3: 10:44:40.327940 20.20.20.194.33201 > 40.40.40.12.21: S 2023625103:2023625103(0) win 8192 <mss 1380,nop,nop,sackOK>
3 packets shown
ASA#
So far, its looking good. Even though the FTP failed, I see my customer's FTP packets going out. Next, my customer tried a different FTP program, Filezilla:
Tested filezilla:
ASA# sh capture capin
9 packets captured
1: 10:44:39.319960 20.20.20.194.33201 > 40.40.40.12.21: S 2922611106:2922611106(0) win 8192 <mss 1380,nop,wscale 0,nop,nop,sackOK>
2: 10:44:39.822894 20.20.20.194.33201 > 40.40.40.12.21: S 2086380860:2086380860(0) win 8192 <mss 1380,nop,wscale 0,nop,nop,sackOK>
3: 10:44:40.327940 20.20.20.194.33201 > 40.40.40.12.21: S 2023625103:2023625103(0) win 8192 <mss 1380,nop,nop,sackOK>
-----
4: 10:47:26.243227 20.20.20.194.25433 > 40.40.40.12.21: S 2515756725:2515756725(0) win 64240 <mss 1380,nop,nop,sackOK>
5: 10:47:26.646923 20.20.20.194.25433 > 40.40.40.12.21: S 2417835985:2417835985(0) win 64240 <mss 1380,nop,nop,sackOK>
6: 10:47:27.193913 20.20.20.194.25433 > 40.40.40.12.21: S 1514131771:1514131771(0) win 64240 <mss 1380,nop,nop,sackOK>
7: 10:47:32.196660 20.20.20.194.57968 > 40.40.40.12.21: S 2153025465:2153025465(0) win 64240 <mss 1380,nop,nop,sackOK>
8: 10:47:32.663173 20.20.20.194.57968 > 40.40.40.12.21: S 1936097903:1936097903(0) win 64240 <mss 1380,nop,nop,sackOK>
9: 10:47:33.210636 20.20.20.194.57968 > 40.40.40.12.21: S 1691218731:1691218731(0) win 64240 <mss 1380,nop,nop,sackOK>
-----
9 packets shown
ASA#
Ok, so now I see more traffic leaving my firewall. Looks good, but the FTP still failed. Upon the recommendation of the larger provider, they want us to send a transaction. Below, they test.
Tested to the remote endpoint a regular transaction:
ASA# sh capture capin
12 packets FTP captured
1: 10:44:39.319960 20.20.20.194.33201 > 40.40.40.12.21: S 2922611106:2922611106(0) win 8192 <mss 1380,nop,wscale 0,nop,nop,sackOK>
2: 10:44:39.822894 20.20.20.194.33201 > 40.40.40.12.21: S 2086380860:2086380860(0) win 8192 <mss 1380,nop,wscale 0,nop,nop,sackOK>
3: 10:44:40.327940 20.20.20.194.33201 > 40.40.40.12.21: S 2023625103:2023625103(0) win 8192 <mss 1380,nop,nop,sackOK>
4: 10:47:26.243227 20.20.20.194.25433 > 40.40.40.12.21: S 2515756725:2515756725(0) win 64240 <mss 1380,nop,nop,sackOK>
5: 10:47:26.646923 20.20.20.194.25433 > 40.40.40.12.21: S 2417835985:2417835985(0) win 64240 <mss 1380,nop,nop,sackOK>
6: 10:47:27.193913 20.20.20.194.25433 > 40.40.40.12.21: S 1514131771:1514131771(0) win 64240 <mss 1380,nop,nop,sackOK>
7: 10:47:32.196660 20.20.20.194.57968 > 40.40.40.12.21: S 2153025465:2153025465(0) win 64240 <mss 1380,nop,nop,sackOK>
8: 10:47:32.663173 20.20.20.194.57968 > 40.40.40.12.21: S 1936097903:1936097903(0) win 64240 <mss 1380,nop,nop,sackOK>
9: 10:47:33.210636 20.20.20.194.57968 > 40.40.40.12.21: S 1691218731:1691218731(0) win 64240 <mss 1380,nop,nop,sackOK>
----
10: 10:48:32.160407 20.20.20.194.58543 > 40.40.40.12.21: S 2448264669:2448264669(0) win 65535 <mss 1380,nop,nop,sackOK>
11: 10:48:32.655849 20.20.20.194.58543 > 40.40.40.12.21: S 2167152525:2167152525(0) win 65535 <mss 1380,nop,nop,sackOK>
12: 10:48:33.093439 20.20.20.194.58543 > 40.40.40.12.21: S 3432259079:3432259079(0) win 65535 <mss 1380,nop,nop,sackOK>
-----
12 packets shown
ASA#
(Connection was refused by endpoint)
There is goes, packets 10-12. Looks like its getting out of our firewall. Now the girl on the other end decides to reset the FTP service. Even though everyone else is working, she still takes this step, which proved to get this resolved. After she restarted the FTP service on the server, below is what I saw leaving the firewall (packets 13-34).
FTP test AFTER FTP service was restarted on remote end:
ASA# sh capture capin
34 packets captured
1: 10:44:39.319960 20.20.20.194.33201 > 40.40.40.12.21: S 2922611106:2922611106(0) win 8192 <mss 1380,nop,wscale 0,nop,nop,sackOK>
2: 10:44:39.822894 20.20.20.194.33201 > 40.40.40.12.21: S 2086380860:2086380860(0) win 8192 <mss 1380,nop,wscale 0,nop,nop,sackOK>
3: 10:44:40.327940 20.20.20.194.33201 > 40.40.40.12.21: S 2023625103:2023625103(0) win 8192 <mss 1380,nop,nop,sackOK>
4: 10:47:26.243227 20.20.20.194.25433 > 40.40.40.12.21: S 2515756725:2515756725(0) win 64240 <mss 1380,nop,nop,sackOK>
5: 10:47:26.646923 20.20.20.194.25433 > 40.40.40.12.21: S 2417835985:2417835985(0) win 64240 <mss 1380,nop,nop,sackOK>
6: 10:47:27.193913 20.20.20.194.25433 > 40.40.40.12.21: S 1514131771:1514131771(0) win 64240 <mss 1380,nop,nop,sackOK>
7: 10:47:32.196660 20.20.20.194.57968 > 40.40.40.12.21: S 2153025465:2153025465(0) win 64240 <mss 1380,nop,nop,sackOK>
8: 10:47:32.663173 20.20.20.194.57968 > 40.40.40.12.21: S 1936097903:1936097903(0) win 64240 <mss 1380,nop,nop,sackOK>
9: 10:47:33.210636 20.20.20.194.57968 > 40.40.40.12.21: S 1691218731:1691218731(0) win 64240 <mss 1380,nop,nop,sackOK>
10: 10:48:32.160407 20.20.20.194.58543 > 40.40.40.12.21: S 2448264669:2448264669(0) win 65535 <mss 1380,nop,nop,sackOK>
11: 10:48:32.655849 20.20.20.194.58543 > 40.40.40.12.21: S 2167152525:2167152525(0) win 65535 <mss 1380,nop,nop,sackOK>
12: 10:48:33.093439 20.20.20.194.58543 > 40.40.40.12.21: S 3432259079:3432259079(0) win 65535 <mss 1380,nop,nop,sackOK>
------
13: 10:56:12.340070 20.20.20.194.17009 > 40.40.40.12.21: S 3548665372:3548665372(0) win 65535 <mss 1380,nop,nop,sackOK>
14: 10:56:12.346341 20.20.20.194.17009 > 40.40.40.12.21: . ack 723165591 win 65535
15: 10:56:12.355633 20.20.20.194.17009 > 40.40.40.12.21: P 3548665373:3548665389(16) ack 723165636 win 65490
16: 10:56:12.401361 20.20.20.194.17009 > 40.40.40.12.21: P 3548665389:3548665408(19) ack 723165656 win 65470
17: 10:56:12.601195 20.20.20.194.17009 > 40.40.40.12.21: . ack 723165712 win 65414
18: 10:56:12.601455 20.20.20.194.17009 > 40.40.40.12.21: P 3548665408:3548665414(6) ack 723165732 win 65394
19: 10:56:12.607970 20.20.20.194.17009 > 40.40.40.12.21: P 3548665414:3548665419(5) ack 723165742 win 65384
20: 10:56:12.617598 20.20.20.194.17009 > 40.40.40.12.21: P 3548665419:3548665447(28) ack 723165787 win 65339
21: 10:56:12.637708 20.20.20.194.17009 > 40.40.40.12.21: P 3548665447:3548665453(6) ack 723165813 win 65313
22: 10:56:12.681651 20.20.20.194.53132 > 40.40.40.12.20: S 3213895457:3213895457(0) ack 2441606499 win 16384 <mss 1380,nop,wscale 0,nop,nop,sackOK>
23: 10:56:12.745139 20.20.20.194.17009 > 40.40.40.12.21: . ack 723165865 win 65261
24: 10:56:12.745170 20.20.20.194.53132 > 40.40.40.12.20: . ack 2441606803 win 65232
25: 10:56:12.745170 20.20.20.194.53132 > 40.40.40.12.20: F 3213895458:3213895458(0) ack 2441606803 win 65232
26: 10:56:12.789769 20.20.20.194.17009 > 40.40.40.12.21: P 3548665453:3548665481(28) ack 723165865 win 65261
27: 10:56:12.796376 20.20.20.194.17009 > 40.40.40.12.21: P 3548665481:3548665487(6) ack 723165891 win 65235
28: 10:56:12.802860 20.20.20.194.41603 > 40.40.40.12.20: S 3126237913:3126237913(0) ack 666517918 win 16384 <mss 1380,nop,wscale 0,nop,nop,sackOK>
29: 10:56:12.824099 20.20.20.194.41603 > 40.40.40.12.20: . ack 666518222 win 65232
30: 10:56:12.824130 20.20.20.194.41603 > 40.40.40.12.20: F 3126237914:3126237914(0) ack 666518222 win 65232
31: 10:56:12.824145 20.20.20.194.17009 > 40.40.40.12.21: . ack 723165943 win 65183
32: 10:56:12.831866 20.20.20.194.17009 > 40.40.40.12.21: P 3548665487:3548665493(6) ack 723165943 win 65183
33: 10:56:12.838594 20.20.20.194.17009 > 40.40.40.12.21: . ack 723165953 win 65174
34: 10:56:12.838625 20.20.20.194.17009 > 40.40.40.12.21: F 3548665493:3548665493(0) ack 723165953 win 65174
--------
34 packets shown
ASA#
Now, here is the frustrating part. After all was said and done, one of the guys on the remote end made the statement that this customer stopped working on a particular night and that happened to be the same time when THEY had restarted the FTP service on their side on their server. I guess it doesnt occur to everyone that when a customer calls with a problem, that "Oh yeah, we restarted the FTP service about that same time" just doesnt come to mind for some reason.
This is the retired Shane Killen personal blog, an IT technical blog about configs and topics related to the Network and Security Engineer working with Cisco, Brocade, Check Point, and Palo Alto and Sonicwall. I hope this blog serves you well. -- May The Lord bless you and keep you. May He shine His face upon you, and bring you peace.
Subscribe to:
Post Comments (Atom)
Nice troubleshooting. Once again we always have to prove that the problem isn't the network...been doing it for 20 years now, and guess I will continue doing it. Ugh...
ReplyDelete