Packet Capture: How To Graph Time Delay Between Packets

If you dont like to look at the time column in Wireshark, sometimes its helpful to graph it out.  Some people are just visual, and it helps to actually see a graph.  I went to a website and captured it.  Here is how I could read the graphs.

Here is the total graph, as I saw it:

So, I filtered on the packets I wanted to see.  Notice in my filter "tcp.stream eq 21".  When I click on the first dot, notice that it highlights the packet as well.

Then, there is a .06 ms delay to the next packet:

Next, there is about a .02 ms delay to the next packet.  Also, notice the size of the packet in the graph (and the "length" in the column of the capture):

Next packet is microseconds behind:

Next packet:

And the last:

Good stuff.

Smartphones And The Fun Things We Use Them For

I have to say that I was one of the last ones to move into the smartphone world.  I really didnt want to.  I was just fine with my little flip phone that I had.  It did all I wanted it to: Text and Talk.
BUT, I have had to move into the real current world (for now).  I have been thinking about what all I actually use my HTC phone for.  I have come up with a list below of things I use regularly.  I'd like to hear your list too.
1. First and foremost, its a clock.
2. Phone.
3. Texting.
4. Alarm.
5. Bible.
6. Email.
7. Wifi Analyzer.
8. SSH client.
9. Internet (general).
10. Camera (which my pic of the week comes from)
11. MP3 player.
12. Reach my stored files when not behind computer.
13. Notepad (for remembering things).
14. Controls my music in my house.
15. Calendar and appointments.
16. Flashlight.
17. Calculator.
18. GPS for getting me places.
19. Camcorder.
20. LinkedIn.  I closed my LinkedIn account, so this no longer applies.
20. Study for certifications and learning new technologies.

Cisco IP Phone: What Does The Circle With A Dot In The Middle Mean

Honestly, I have no idea what that circle with a dot means.  But, here is what I do know.  The port on the switch was not configured correctly for the voice vlan.  I sent out another engineer, and he moved that patch cable to another port that was configured with the voice vlan, and the phone came up at that point.  Chalk this one up to improper configuration of the voice vlan.

Sunday Thought: "Be Born In Me"

More December music/worship.
https://www.youtube.com/watch?v=yOPHkvbdOpw&index=3&list=LLg7WaNtHz6oyXJpsDRzFxzQ

Pic Of The Week: Sanctuatry

Someone's blanket and cardboard padding at one of the downtown churches.

Merry Christmas To Everyone!

Merry Christmas!  I think this song really sums it up.  Take a few minutes, listen to the lyrics and reflect on your own life.  Think about what the Savior coming into the world really means.  Remember, He is the reason we celebrate this season:
https://www.youtube.com/watch?v=O3wujkozv9E

Christmas Eve :)

Thought I would share a Christmas picture from my own home.

Palo Alto: Agentless User-ID And Windows 2000 Server Integration

Yes, its 2015.  But, there are some companies out there with Windows 2000 server as their domain controller still.  So at this company, I have installed a Palo Alto (ver 6.0.5-h3) firewall, and one of the last things I implemented (in this case), is user-ID.

Does the Palo Alto agentless user-ID integrate with AD on Windows 2000?  Short answer, NO.  Its supported only from Windows 2008 and above.  How about the agent client version?  Yes, but you have to be running client version 4.0 or earlier.  

So, no worries.  Its just time to upgrade the Windows 2000 server.  

Wireshark: Firewall ACL Rule Help

Have you ever taken a wireshark capture, found the packets that you need to find, and wonder how to block that at your firewall?  Wireshark does make this easy for you, if you know where to go.  In my example, I select the packet I dont want, and go to Tools --> Firewall ACL Rules and it will show you what you should type in for your ACL for denying traffic.  See the screenshot below.  Play around with it if you are the firewall admin.  It can help you out.

Sunday Thought: Real

You all know that I like Christian music.  Maybe you will like this one.  Its a good December song.  Take a few minutes and carefully listen.

https://www.youtube.com/watch?v=aLQgnYxxMcM&index=4&list=LLg7WaNtHz6oyXJpsDRzFxzQ

Pic Of The Week: Into The Sun

Cisco 6509-E Upgrade

I have been working through a problem with a customer that has involved a lot of people in the IT department.  Ill make this story short and really only relate to the "network" portion of this particular day.  I really want to concentrate this post around upgrading the core switches, which happens to be Cisco 6509-E switches (two of them).

On the first core 6509-E, there were no issues at all.  Both blades upgraded and all was done pretty quickly.  We put the code on a compact flash, changed the boot statement in the config, reboot and all was good.  Things should really always go this good, right?

On the second core 6509-E, there were issues.  First, it did continued to boot up on the old code.  It turned out that we had to format the new compact flash card that we received. It never saw the code on the new compact flash card, so it reverted back to the sup-bootflash.  We formatted the flash card, got the new image on, changed the boot statement again, and we were good to go.  Once we booted to the new code, we noticed that one of the fiber modules wouldn't power up after that upgrade.  Why?  Turns out the new IOS that we put on would not support the old fiber module that wouldn't power up.  This 6509-E was older than the first 6509-E that we upgraded.  So, we got on Cisco and downloaded a different (older than we got, but newer than we originally had) and put it on the second 6509-E.  All things seemed to work fine at that point, with the exception that I had to put the config for that module (the fiber module that wouldn't power up) back in.  It seemed to get deleted out when I had the issues on the module (only one module was affected).  Once I cut and pasted that back in, all was good.

It just goes to show that even though an upgrade should be painless, sometimes it can be painful.

Quick Network Analysis Tool

One tool that I really like using is called Capsa, by Colasoft.  Now please dont think Im advertising for them.  Im not, but if Im being honest, it IS one of my tools in my tool pouch (my laptop).  Its really helpful for quick troubleshooting.
So, with that said, I had a customer ask me about a 97% utilization on the remote site MPLS link.  I told him that I would go figure it out and be right back.
10 minutes later, I come back with the source and destination of the troublemaker (not really).  We found out that a guy was doing legitimate work traffic, but not the way he was supposed to.  Either way, my point is that having the right tools as a network guy can help you troubleshoot problems quickly and effectively.  And yes, sometimes it does require spending some money to get those tools.

Cisco ASA: How To View Your Captured Packets In Wireshark From The ASA

Some people just like GUIs.  That is fine.  I can understand that, I guess.  So what can you do when you want to see packet captures on the ASA in Wireshark instead of CLI?  Well, two things.  First, make sure "http server enable" and "http X.X.X.X X.X.X.X inside" is configured.  Then, take your packet capture.  (Click on this link on how to enable a packet capture in CLI.)  Once you have your packet capture going and you have the traffic you want to see, then do the following in your web browser:
https://10.10.2.2/capture/capin/pcap  <-- where 'capin' is the name of the capture I am taking

Rename the file and save it to what you want to save it as.  Then open it up in Wireshark.  You have the .pcap file that you want to see in Wireshark now.

Cisco Nexus: FEX Phases After Install

Dont forget that when you install a FEX into a Nexus, its going to check to make sure the image matches the 5k/7k.  If not, it will download the image to make sure it matches.  It goes through the process: Image Download --> Offline (reboot) --> Online

N5K-1# sh fex
  FEX         FEX           FEX                       FEX
Number    Description      State            Model            Serial
------------------------------------------------------------------------
112        FEX0112               Image Download    N2K-C2232PP-10GE   SSI17XXXXX

N5K-1# sh fex
  FEX         FEX           FEX                       FEX
Number    Description      State            Model            Serial
------------------------------------------------------------------------
112        FEX0112               Offline    N2K-C2232PP-10GE   SSI17XXXXX

N5K-Backup-1# sh fex
  FEX         FEX           FEX                       FEX
Number    Description      State            Model            Serial
------------------------------------------------------------------------
112        FEX0112                Online    N2K-C2232PP-10GE   SSI17XXXXX

Cisco ASA: Packet Capture On The ASA In CLI

I had to do some remote troubleshooting on an ASA that, according to the customer, was not allowing SIP traffic in on their new SIP services.  In this scenario, CLI was the only option, and really, I just glad about that.  Although, I do also like the GUI form of the packet capture that Cisco has in the ASDM.  Its easy.  Easier than CLI in this case, but I like CLI, so Im OK with it.  Here is the ACL I configured to capture traffic to their phone system's external IP:
CiscoASA# config t
CiscoASA(config)# access-list 188 permit ip any host 5.5.5.250
CiscoASA(config)# exit

Now, lets enable the capture on the outside interface:
CiscoASA# capture capin interface outside access-list 188

So now I run a ping to that 5.5.5.250 IP address.  Then, I make the phone call to see if SIP traffic came to the ASA.  Then, I HTTP'ed to the phone system.  So, how many bytes are captured?

CiscoASA# sho capture

capture capin type raw-data access-list 188 interface outside [Capturing - 360 bytes]

Now, what is in the packet capture log?

See the capture below:

CiscoASA#show capture capin

29 packets captured

   1: 00:41:49.017668 33.33.33.128 > 5.5.5.250: icmp: echo request

   2: 00:41:50.068218 33.33.33.128 > 5.5.5.250: icmp: echo request

   3: 00:41:54.843233 33.33.33.128 > 5.5.5.250: icmp: echo request

   4: 00:41:55.874863 33.33.33.128 > 5.5.5.250: icmp: echo request

   5: 00:45:23.107217 33.33.33.128.2098 > 5.5.5.250.80: S 2929358780:2929358780(0) win 8192 <mss 1260,nop,wscale 8,nop,nop,sackOK>

   6: 00:45:23.107523 33.33.33.128.2097 > 5.5.5.250.80: S 2605027608:2605027608(0) win 8192 <mss 1260,nop,wscale 8,nop,nop,sackOK>

   7: 00:45:23.331632 33.33.33.128.2098 > 5.5.5.250.80: . ack 1980576723 win 68

   8: 00:45:23.331846 33.33.33.128.2097 > 5.5.5.250.80: . ack 1928612590 win 68

   9: 00:45:23.335599 33.33.33.128.2097 > 5.5.5.250.80: P 2605027609:2605028002(393) ack 1928612590 win 68

  10: 00:45:23.413201 33.33.33.128.2099 > 5.5.5.250.8080: S 1012239204:1012239204(0) win 8192 <mss 1260,nop,wscale 8,nop,nop,sackOK>

  11: 00:45:23.413476 33.33.33.128.2100 > 5.5.5.250.8080: S 4230440435:4230440435(0) win 8192 <mss 1260,nop,wscale 8,nop,nop,sackOK>

  12: 00:45:23.489537 33.33.33.128.2099 > 5.5.5.250.8080: . ack 1452080160 win 68

  13: 00:45:23.491628 33.33.33.128.2100 > 5.5.5.250.8080: . ack 1273284172 win 68

  14: 00:45:23.495350 33.33.33.128.2099 > 5.5.5.250.8080: P 1012239205:1012239603(398) ack 1452080160 win 68

  15: 00:45:23.635495 33.33.33.128.2097 > 5.5.5.250.80: . ack 1928612939 win 67

  16: 00:45:23.663829 33.33.33.128.2099 > 5.5.5.250.8080: P 1012239603:1012240043(440) ack 1452080463 win 67

  17: 00:45:23.667690 33.33.33.128.2100 > 5.5.5.250.8080: P 4230440436:4230440774(338) ack 1273284172 win 68

  18: 00:45:23.750037 33.33.33.128.2099 > 5.5.5.250.8080: P 1012240043:1012240578(535) ack 1452080637 win 67

  19: 00:45:23.937359 33.33.33.128.2100 > 5.5.5.250.8080: . ack 1273285310 win 64

  20: 00:45:23.943371 33.33.33.128.2099 > 5.5.5.250.8080: . ack 1452083157 win 68

  21: 00:45:24.017333 33.33.33.128.2099 > 5.5.5.250.8080: . ack 1452085677 win 68

  22: 00:45:24.201329 33.33.33.128.2099 > 5.5.5.250.8080: . ack 1452088197 win 68

  23: 00:45:24.201390 33.33.33.128.2099 > 5.5.5.250.8080: . ack 1452090717 win 68

  24: 00:45:24.355709 33.33.33.128.2099 > 5.5.5.250.8080: . ack 1452093237 win 68

  25: 00:45:24.355740 33.33.33.128.2099 > 5.5.5.250.8080: . ack 1452098277 win 68

  26: 00:45:24.438881 33.33.33.128.2099 > 5.5.5.250.8080: . ack 1452103317 win 68

  27: 00:45:24.735542 33.33.33.128.2099 > 5.5.5.250.8080: . ack 1452104344 win 64

  28: 00:45:35.432396 33.33.33.128.2098 > 5.5.5.250.80: F 2929358781:2929358781(0) ack 1980576723 win 68

  29: 00:45:35.621702 33.33.33.128.2098 > 5.5.5.250.80: . ack 1980576724 win 68

29 packets shown

CiscoASA#

So, as you can see, no SIP traffic.  Its not making it to the ASA.  Now lets disable the capture:

CiscoASA# no capture capin interface outside access-list 188

Sunday Thought: Bless Israel

You know, it seems that every belief causes a separation of people.  In IT, if you believe that Windows is the thing, then the Unix guys are going to think you are full of it.  And probably vice versa.  Same goes for our beliefs in other areas of our lives.
As a Christian, Im fully aware that not everyone believes the same way I do.  People choose to believe what they want to.  Im really only responsible for me and what I believe.  I cant make you believe anything that you dont want to believe.  Although, I am responsible for at least telling you, at least once, about the Messiah (Jesus) and what he has done for us.  Beyond that, its on you.
Same goes for supporting Israel.  I DO support Israel.  I think God is clear that if you bless Israel, you will be blessed.  If you curse Israel, you will be cursed.  I think that Genesis 12:3 is clear about that.  Look it up.
I'd like to have the blessing of God in my life.  So I choose to bless Israel.

Pic Of The Week: Sign I Saw

Cisco Nexus 7000 Module Replacement

I wanted to just share a few pictures of an RMA I had to do on an Nexus 7000 blade.  The blade came in bad from the start, so time for an RMA.
The customer told me that he troubleshot as much as possible and asked me to do the RMA.  So I reseated the module myself, put it in another slot, and even put it in another Nexus 7000.  Still same issue, RED light on front and not show up as powered up.
Nexus-1# sh hardware

Module9  powered-dn
  Module type is : 10/100/1000 Mbps Ethernet XL Module
  1 submodules are present
  Model number is N7K-M148GT-11L
  H/W version is 2.1
  Part Number is 73-15230-01
  Part Revision is C0
  Manufacture Date is Year 17 Week 22
  Serial number is XXXXX
  CLEI code is XXXX

So obviously, its not looking good.  So I called up Cisco and got the RMA going.
Blade comes in and I got it replaced.

Good to go!

Technical Training

I have recently done some 'virtual' technical training on a certain technology.  It always seems to be the same to me.  Now Im not a fan of 'virtual' training.  I need to be there with someone and able to ask questions and interact.  Virtual training does work for some, but not for me.  But one thing I do think is that training should actually prepare you for installing equipment, from start to finish.  It seems like what I typically see is that the trainer will go over what each feature is instead of what it is AND how to implement it.  I mean the labs that they give you just do this 'step 1, do this. step 2, do this'.  It just seems like there is a better way of teaching this stuff.  Maybe this works well for some people, but it doesnt for me.  Im thinking in the future, I might want to be a technical trainer.

Palo Alto: In Initial Configuration, Commit Fails Due To Virtual-Wire Config

In the initial configuration of the Palo Alto, I notice that if you do not want to do virtual wire, you have to go into Network --> Virtual Wires and delete the default that is configured to get the commit to succeed.  I dont love that really, but if you are configuring for a Layer 3 firewall, then I have found I have to go delete this out.

ShoreTel Backups: Be Careful To Watch The Services Stop

One thing I notice on a 14.1 to 14.2 upgrade I was doing that I wanted to share with you all.  I notice that when I ran the stop hq services bat file (to stop all services so you can do a backup of the Shoreline Data folder), one service in particular didnt stop.  Make sure you check that all services stop before you copy your Shoreline Data folder to another location.

Sunday Thought: Will You Be Known?

Matthew 7:15-23

“Beware of false prophets who come disguised as harmless sheep but are really vicious wolves. You can identify them by their fruit, that is, by the way they act. Can you pick grapes from thornbushes, or figs from thistles? A good tree produces good fruit, and a bad tree produces bad fruit. A good tree can’t produce bad fruit, and a bad tree can’t produce good fruit. So every tree that does not produce good fruit is chopped down and thrown into the fire. Yes, just as you can identify a tree by its fruit, so you can identify people by their actions.

“Not everyone who calls out to me, ‘Lord! Lord!’ will enter the Kingdom of Heaven. Only those who actually do the will of my Father in heaven will enter. On judgment day many will say to me, ‘Lord! Lord! We prophesied in your name and cast out demons in your name and performed many miracles in your name.’ But I will reply, ‘I never knew you. Get away from me, you who break God’s laws.’”

Pic Of The Week: Family

Windows: How To Reset The Administrator Password On A PC ~By Babak Hoseini

Babak Hoseini is a friend of mine that I have the privilege of posting some of his IT knowledge on my blog.  He was kind enough to write another post about resetting the Windows administrator password when you dont know it.  Thank you Babak.  ~~Shane Killen

Windows: How To Reset The Administrator Password On A PC
There are several ways to reset the administrator password such as “ERDcommander”, “Hiren’s BootCD and the other repairing CD’s but sometimes you may not have such CD’s.
There is an exploit in Microsoft Windows that you can reset the administrator password without using these CD’s. I strongly advise you to fix this exploit on servers. I’m going to explain how to reset the administrator password and then to close this security breach.

First of all you need to boot your PC with Windows CD in recovery mode. Put the Windows 7 or 2008 CD into the CD Rom and boot your system with the CD and select the “Recovery mode” option.

Then select the time and language and then click next. Click on “Repair your computer” and select “Command prompt”.

It’s better to back up the “Sethc.exe” file in another location. Type this command:
copy  d:\windows\system32\sethc.exe  d:\

Then you must copy the “cmd.exe” file instead of “sethc.exe
copy  d:\windows\system32\cmd.exe  d:\windows\system32\sethc.exe
Type “exit” and restart your system normally. At the “log on” screen, press the “shift key” 5 times. A “Command prompt” will open and you can reset administrator password with the “net user” command.

net user  administrator  123456

And now what should be done to close this exploit? It’s so easy and simple! You can disable “Sticky keys” via this path:
Control Panel –> Ease of Access Center –> Make the keyboard easier to use

Uncheck “Turn on Sticky keys” and from the “Setup Stick keys” section, uncheck the “Turn on Sticky keys when SHIFT is pressed five times”.

Runt Packets

Have you ever seen runts on your switch interfaces?  What is that?  A runt is a packet that is less than 64 bytes long.  So, really, this is a problem.  In my case, on the server (which I just inform the server guys at this particular customer).  See below.  Notice that the runts and the input errors line up.  In this case, its a server issue.  But keep in mind, you dont really want to see these runts on the switch.  Call the server guys and get them to figure out what is going on.  They probably want to know about it anyway.

GigabitEthernet1/0/9 is up, line protocol is up (connected)
...edited
     5962915515 packets input, 2654018058090 bytes, 0 no buffer
     Received 1305694 broadcasts (1253405 multicasts)
     60677 runts, 0 giants, 0 throttles
     60677 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1253405 multicast, 0 pause input
     0 input packets with dribble condition detected
     1181023696 packets output, 913878577075 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets

Cisco Nexus: How To Add A FEX Module To Redundant 5000s

Below is a configuration example for adding a FEX to a pair Nexus 5000 for redundancy.  Two 5000s  while adding two 2000 FEX devices.  You have to do basically the same config on both 5000s:

Nexus5000 #1 config:
fex 106
  pinning max-links 1
  description "FEX0106"

interface port-channel106
  switchport mode fex-fabric
  fex associate 106
  vpc 106
speed 10000

interface Ethernet2/15
  description **FEX 106**
  switchport mode fex-fabric
  fex associate 106
  channel-group 106
speed 10000

fex 107
  pinning max-links 1
  description "FEX0107"

interface port-channel107
  switchport mode fex-fabric
  fex associate 107
  vpc 107

interface Ethernet 2/16
  description **FEX 107**
  switchport mode fex-fabric
  fex associate 107
  channel-group 107


Nexus5000 #2 config:

fex 106
  pinning max-links 1
  description "FEX0106"

interface port-channel106
  switchport mode fex-fabric
  fex associate 106
  vpc 106
speed 10000

interface Ethernet2/15
  description *** To FEX 106 ****** To FEX 106 ***
  switchport mode fex-fabric
  fex associate 106
  channel-group 106

fex 107
  pinning max-links 1
  description "FEX0107"

interface port-channel107
  switchport mode fex-fabric
  fex associate 107
  vpc 107
speed 10000

interface Ethernet2/16
  description *** To FEX 107 ****** To FEX 107 ***
  switchport mode fex-fabric
  fex associate 107
  channel-group 107
speed 10000

Brocade ICX Switch: Verifying Stack Communication

When you build a stack, you should always check to make sure the stack is communicating the way it should.  Make sure the topology looks the way it should (which I edited out here), but also make sure the communication between CPU to CPU looks good.  What that means is communication across the stacking cables from one CPU to another.  What that boils down to is when a packet traverses from one switch in the stack to the other, switch one processes the packet, forwards across the stacking cable to switch two and on to its CPU for processing.  You need to make sure communication is good, and the way to do that is highlighted below.  Check it using the 'show stack connection' command, as show below:
CORE#show stack connection
edited...

trunk probe results: 4 links
Link 1: u4 -- u1, num=5
  1: 1/2/1 (T0) <---> 4/2/6 (T1)
  2: 1/2/2 (T0) <---> 4/2/7 (T1)
  3: 1/2/3 (T0) <---> 4/2/8 (T1)
  4: 1/2/4 (T0) <---> 4/2/9 (T1)
  5: 1/2/5 (T0) <---> 4/2/10(T1)
Link 2: u2 -- u1, num=5
  1: 1/2/6 (T1) <---> 2/2/1 (T0)
  2: 1/2/7 (T1) <---> 2/2/2 (T0)
  3: 1/2/8 (T1) <---> 2/2/3 (T0)
  4: 1/2/9 (T1) <---> 2/2/4 (T0)
  5: 1/2/10(T1) <---> 2/2/5 (T0)
Link 3: u3 -- u2, num=5
  1: 2/2/6 (T1) <---> 3/2/1 (T0)
  2: 2/2/7 (T1) <---> 3/2/2 (T0)
  3: 2/2/8 (T1) <---> 3/2/3 (T0)
  4: 2/2/9 (T1) <---> 3/2/4 (T0)
  5: 2/2/10(T1) <---> 3/2/5 (T0)
Link 4: u4 -- u3, num=5
  1: 3/2/6 (T1) <---> 4/2/1 (T0)
  2: 3/2/7 (T1) <---> 4/2/2 (T0)
  3: 3/2/8 (T1) <---> 4/2/3 (T0)
  4: 3/2/9 (T1) <---> 4/2/4 (T0)
  5: 3/2/10(T1) <---> 4/2/5 (T0)
CPU to CPU packets are fine between 4 units.

Here is what one looks like when CPU to CPU communication doesnt look good:

*** Error! should have 4 links, but 2: missing u1-u2, u2-u3, .
Link 1: u4 -- u1, num=1
  1: 1/2/1 (T0) <---> 4/2/6 (T1)
Link 2: u4 -- u3, num=1
  1: 3/2/6 (T1) <---> 4/2/1 (T0)
*** Error! no CPU to CPU:  u1 -x- u2,
*** Error! only one directional CPU to CPU: u3 --> u1
*** Error! no CPU to CPU:  u2 -x- u3,
*** Error! no CPU to CPU:  u2 -x- u4,

*** Error! one directional CPU to CPU:  u3 --> u4,

Make sure you check to verify what you think you have is correct.