Friday, January 30, 2015

Even More Carrier Problems

Thank you to all the Telcos out there that make me keep you honest.  One of my customers told me that they came into work this morning to find that they could not make voice calls and could not receive calls.  So I get onsite and I can see that the PRI module has an alarm on it.  I pull out my trusty T1 loopback and I get green (meaning it loops up) on the csu/dsu module.  Green is good.  So I go back to the AT&T smartjack where I find no power on the HTU-R that is inside the case.  Ok, that is a problem for sure.
Well, the responsible carrier says that they can not get to smartjack.  I tell them they are correct, because it doesnt have power on it.  They say they will call AT&T.  AT&T tells them everything looks good.  Good?  No power is good?  I tell the responsible carrier that its not good.  We still cant call out and make calls.  Still no power on the smartjack.  How can things be good?  The responsible carrier says Im right, that they cant reach the smartjack.  I tell them that is because there is NO POWER ON THE SMARTJACK!  Responsible carrier says they will have to contact AT&T again.
You kind of get the idea on how this is going.  Anyway, eventually, they fix it.

Look AT&T, no power!

Thursday, January 29, 2015

Proving Telco Problems

Why are Voice carriers always able to get by telling you whatever they want, without ever having to prove anything to you???  Doesn't anyone think that is just not right?  Well, its ok.  I can prove things for them.  So I have a DID number that I cant call into.  I mean, the customer internal phone doesn't ring anymore when you call its external DID number.  A little frustrating for the customer for sure.  We looked into the Cisco CUCM and the translation pattern looks ok.  Lets debug on the voice gateway, to make sure the call is getting that far.  I mean, it has to come into the gateway to get to the customer CUCM, right?
So I get on the Cisco router (the voice gateway), and I do a "debug isdn q931".  That turns on debugging for incoming/outgoing calls on the PRI.  Lets make a test call into one we know works.  Ive changed the numbers for customer privacy.

003768: *Jan  6 20:07:25.958: ISDN Se0/0/1:23 Q931: RX <- SETUP pd = 8  callref = 0x406C 
Bearer Capability i = 0x8090A2 
Standard = CCITT 
Transfer Capability = Speech  
Transfer Mode = Circuit 
Transfer Rate = 64 kbit/s 
Channel ID i = 0xA98385 
Exclusive, Channel 5 
Facility i = 0x9F8B0100A11702011C020100800F4A4343454F20202020202020202020 
Protocol Profile =  Networking Extensions 
0xA11702011C020100800F4A4343454F20202020202020202020 
Component = Invoke component 
Invoke Id = 28 
Operation = CallingName 
Name Presentation Allowed Extended
Name = XYZ Company           
Display i = 'XYZ Company ' 
Calling Party Number i = 0x2180, '2055558511' 
Plan:ISDN, Type:National 
Called Party Number i = 0xC1, '3337546' 
Plan:ISDN, Type:Subscriber(local)
003769: *Jan  6 20:07:26.146: ISDN Se0/0/1:23 Q931: TX -> CALL_PROC pd = 8  callref = 0xC06C 
Channel ID i = 0xA98385 
Exclusive, Channel 5


Yeah, looks like we see that call coming in from the Telco.  Now lets try the number we question, that the Telco says is fine:



Oh look, nothing.  Well, Im no genius, but if my voice gateway doesnt see it, it must not be getting to it.  Which means its a Telco issue.

Wednesday, January 28, 2015

Keep Your Friends Close, And Your Packet Captures Closer...

Ok, have you ever had to deal with another technical 'professional' that drove you to this point below?  This is me, below, after dealing with someone who didn't understand that in order for packets to get to your internal web server (from the Internet), that they have to hit the firewall first (the one I configured).  And if the packets are not hitting the firewall, they aren't going to get to the web server!!!  So, logically, Miss Techie Support, I don't need your tech guy to come onsite to troubleshoot!  I need you to send your data to the right public address!
Packet captures are the network guy's best friend.  Keep your friends close, and your packet captures closer.  On the Cisco ASA, see this link for how to do a packet capture in CLI.

Tuesday, January 27, 2015

Check Point: How To Get The Right Migrate Export/Import Tools When Upgrading To A Different Version

I really hate the Check Point doesn't document this well, at least as far as I have seen.  Check Point says that when you go from one major version to another (like R76 to R77), they say that you have to use the version of migrate export tools of the version you are going to.  That is mostly correct.  I have seen this also be the case when going from R77.10 to R77.20, which is not a major version change.  In fact, I got myself in trouble one night when I relied on the documentation, only to be told by Check Point TAC that I had to use the R77.20 upgrade export tools instead.  This meant, for me, re-installing R77.10 and re-importing again just to get back to my starting point.  Then go through the process.  So keep that in mind.  But, how, exactly, do you get the right version?
There are two ways to do this.  First, you can download the migration tools from the Check Point site, based on the version you are going to.  If you are going to R77.20, then you have to download those tools (and since their website change recently, its anything but fast on the front end).  I downloaded "Check_Point_migration_tools_R77.20_B020.Gaia_SecurePlatform_Linux.gz" for my upgrade export.  You have to get this file on the management station, gtar it, and then run THIS "migrate export" from the upgrade_tools folder (in /var partition) instead of the one of the current version install.  This is how you get the right version that you will need when going to do an upgrade.  Its probably good just to do this anytime you do an upgrade.
The second way, which I have not done, is to download the upgrade_tools folder from another management station to the box you are trying to get an export from.  This seems like too much trouble to me, but do what you like.
Hope this is helpful, as I have not really seen this documented in Check Point.

Monday, January 26, 2015

Cisco ASA Firewall Cluster Member Replacement ~~By Justin Jocewicz

I have the privilege of having Justin Jocewicz as a guest poster on the Network Fun!!! blog today.  He brings some very nice experience to us all today, and I want to thank him for posting.  Very nice job Justin! ~~Shane Killen

Cisco ASA Firewall Cluster Member Replacement
So one of your firewalls in your highly available cluster died.  It happens.  It’s not your fault.  But, you have to put humpty dumpty back together again.  Do it the wrong way, and you can erase your configuration and bring the cluster down!

Prepare for Success
1.      Backup current configuration:
          a.      Use the more system:running-config command
     b.      Certificates (if required)
2.      No network connectivity:
     a.      Logically shutdown switchports
3.      Matching:
     a.      Exact same hardware, software version, and license as the other cluster member
4.      Rack & stack new hardware.
5.      Connect all cables.
6.      Console connectivity.
7.      Commands:
     a.      failover lan unit <primary|secondary>
          failover lan interface <interface name> <physical interface>
          failover link <interface name> <physical interface>
          failover interface ip <interface name> <IP> <SUBNET> standby <IP>
          interface <physical interface>
          no shut
          exit
         failover


The Main Event
1.      Login to the replacement firewall via console.
2.      Paste your prepared commands.


3.      Verify failover status.


4.      Unshut switchports.
5.       Verify connectivity, failover, connections, VPNs, xlate.

6.       Celebrate flawless replacement with coffee.

Sunday, January 25, 2015

Sunday Thought: Jude

Have you ever taken the time to read Jude?  In the Bible, Jude is toward the end.  I find it interesting to me, that Jude was going to write about salvation.  Instead, he wrote a warning to us.  But, what was he going to say about salvation?  I just think that would have been an interesting topic from Jude.  What would you say about salvation to others?  Or, a better question, what DO you say?
Its one chapter.  Its a good read.

Saturday, January 24, 2015

Pic Of The Week: Trust

More of a screenshot than a picture today, but worth reading in my opinion.

Friday, January 23, 2015

Which Comes First, The Chicken Or The Egg?

What I meant to say was, what comes first in a router when a packet hits, PBR or the routing table?  So think of it this way: If you want to manipulate traffic, like setting the next hop, can the routing table come first?  The answer is no, since it will take PBR to make the change BEFORE decision is made to look at the routing table.  Just FYI.  That will be the case no matter what vendor equipment you are configuring.  Check Point, Cisco, Brocade, Palo Alto etc.  Its all the same on this one.

Thursday, January 22, 2015

Dont Oversell To Your Customer

There have been plenty of times, as a consultant, where I go into a customer to help with an environment, only to think to myself that some sales guy really oversold to the customer.  It really aggravates me to some extent.  Its just not necessary.  Yes, have a five year growth plan.  Yes, expect some unexpected growth.  But dont sell the top of the line equipment when in reality, the customer only needs something middle of the road.  Switching gear is where I see this the most.
Properly size the gear with the current network needs, plus the five year growth plan, plus some extra for unexpected growth.  Not the twenty year growth plan.  Be honest and fair to your customers.

Wednesday, January 21, 2015

ShoreTel Voice: Translations Table Steps

I put this together for a customer of mine.  Here is how you can do some voice translations when you need to have one number automatically go to another.  Digit manipulation.

Tuesday, January 20, 2015

Cisco ASA: Packet Capture To Prove Network Traffic Is Making It Through

I have a customer that I had to go onsite and prove that the firewall was not blocking SIP traffic. The phone guy there was saying that the firewall was blocking the SIP traffic going out to the carrier.  Now, I know it wasn't the firewall that was the problem.  After all, I configured it.  But, as you know, the network guy has to prove everything.  So, I went onsite and did a packet capture.  The internal IP address of 10.11.1.250 is actually hitting the inside interface of the ASA.  Here is how I know:

PACKETS COMING INTO THE INSIDE THE INTERFACE:
ASA(config)# access-list 189 permit ip host 10.11.1.250 host 50.50.50.38
ASA(config)# capture capin interface inside access-list 189
ASA(config)# show capture capin
...
 41: 16:20:17.844484 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  42: 16:20:22.025969 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  43: 16:20:50.807422 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  44: 16:20:51.367931 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  45: 16:20:52.463721 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  46: 16:20:54.561479 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  47: 16:20:58.749808 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  48: 16:21:02.904631 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  49: 16:21:07.055569 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  50: 16:21:11.196935 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  51: 16:21:15.346310 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  52: 16:21:19.507923 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  53: 16:21:23.668056 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394
  54: 16:21:52.539492 10.11.1.250.5060 > 50.50.50.38.5060:  udp 394

Now, I want to see the traffic being sent out the ASA on the outside interface.  I mean, I need to know that the traffic is actually getting THROUGH the firewall, right?  Looks like the NAT'ed IP address of 40.40.40.250 is making it out to the destination of IP 50.50.50.38:5060.  Lets see below:

PACKETS GOING OUT THE OUTSIDE INTERFACE:
ASA(config)# access-list 191 permit ip any host 50.50.50.38
ASA(config)# capture capin interface outside access-list 191
ASA(config)# sh capture capin

26 packets captured

   1: 17:08:02.706521 40.40.40.194 > 50.50.50.38: icmp: echo request
   2: 17:08:03.706506 40.40.40.194 > 50.50.50.38: icmp: echo request
   3: 17:08:04.706659 40.40.40.194 > 50.50.50.38: icmp: echo request
   4: 17:08:04.881820 40.40.40.250.5060 > 50.50.50.38.5060:  udp 394
   5: 17:08:05.440544 40.40.40.250.5060 > 50.50.50.38.5060:  udp 394
   6: 17:08:05.706765 40.40.40.194 > 50.50.50.38: icmp: echo request
   7: 17:08:06.528354 40.40.40.250.5060 > 50.50.50.38.5060:  udp 394
   8: 17:08:06.707818 40.40.40.194 > 50.50.50.38: icmp: echo request
   9: 17:08:07.707894 40.40.40.194 > 50.50.50.38: icmp: echo request
  10: 17:08:08.642926 40.40.40.250.5060 > 50.50.50.38.5060:  udp 394
  11: 17:08:08.707894 40.40.40.194 > 50.50.50.38: icmp: echo request
  12: 17:08:09.709100 40.40.40.194 > 50.50.50.38: icmp: echo request
  13: 17:08:10.709130 40.40.40.194 > 50.50.50.38: icmp: echo request
  14: 17:08:11.709222 40.40.40.194 > 50.50.50.38: icmp: echo request
  15: 17:08:12.710305 40.40.40.194 > 50.50.50.38: icmp: echo request
  16: 17:08:12.794285 40.40.40.250.5060 > 50.50.50.38.5060:  udp 394
  17: 17:08:13.710580 40.40.40.194 > 50.50.50.38: icmp: echo request
  18: 17:08:14.710458 40.40.40.194 > 50.50.50.38: icmp: echo request
  19: 17:08:15.711556 40.40.40.194 > 50.50.50.38: icmp: echo request
  20: 17:08:16.711663 40.40.40.194 > 50.50.50.38: icmp: echo request
  21: 17:08:16.926755 40.40.40.250.5060 > 50.50.50.38.5060:  udp 394
  22: 17:08:17.712792 40.40.40.194 > 50.50.50.38: icmp: echo request
  23: 17:08:18.713876 40.40.40.194 > 50.50.50.38: icmp: echo request
  24: 17:08:19.714028 40.40.40.194 > 50.50.50.38: icmp: echo request
  25: 17:08:20.714043 40.40.40.194 > 50.50.50.38: icmp: echo request
  26: 17:08:21.085521 40.40.40.250.5060 > 50.50.50.38.5060:  udp 394
26 packets shown

Looks like it is being NAT'ed correctly and also being sent out to the appropriate external IP address.  Looks like the firewall is OK.

Monday, January 19, 2015

More ISP Troubleshooting: A Solid Technique For Determining Intermittent Issues

I think this is important for the network guy to really get good at.  Its a solid, proven technique.  I recommend this to anyone troubleshooting intermittent ISP issues.  I realize that I just wrote about this Friday, but this is an important topic.  Its just too common.
Use two different methods of ICMP for testing.  As I mentioned Friday, I use ping plotter pro and a DOS prompt for ICMP.  I NEVER rely on just one method when troubleshooting issues like this.  You will have some false positives, but you will learn which ones are false and which ones are not over time.  If they both fail (ping plotter pro and ping through DOS prompt), then you got what you need.  Capture several time outs to prove it THEM, not you.
Here is another example I just came across.  Customer complained about dropped calls between sites, across the VPN.






Here is a different customer that I did some troubleshooting for.  Same thing, different ISP.  Again, a solid, proven technique.

Sunday, January 18, 2015

Sunday Thought: The Story

There is an interesting book called The Story.  It's selections of the Bible in chronological order, without the Bible chapter/verse numbers. I find that I can't stop reading it. It's like a novel, or at least written like one. I suggest it to you if you don't like the chapter and verse numbers in the Word. This isn't a substitute for reading the Bible, but a companion. I highly encourage you to take a look at it.

Saturday, January 17, 2015

New Page On This Blog

I have added a "About Me - Career Story" page to this blog.  If you are interested in reading it, click on the link across the top or Click Here.  I hope that this is an encouragement to young IT professionals.

Pic Of The Week: Sun


Friday, January 16, 2015

Finding Intermittent ISP Connectivity Issues

How do you do it?  You know something isnt right, but you can still get on the Internet.  Everything works ok, but sometimes it gets slow.  Sometimes you get a "page not found".  But if you refresh, it seems to work.  What is going on???
Well, here is a pretty reliable way to find unreliable ISP connectivity.  Get yourself a program that will do a graph of ICMP.  I use ping plotter pro.  I also use DOS with it.  You will see what I mean below.  Start a ping to a remote site IP address like google or norton or somewhere (not the URL). Also, get your graphing program (ping plotter pro in my case) and ping the same IP within that program also.  Then, watch them both closely.

Now keep in mind this, one sporatic "time out" is really not enough for reliably saying that there is a problem.

Now in the below, the speed test starts dropping out when a consecutive "time out" occurs.


Anyway, if you suspect problems with your Internet connectivity, try this above.  Its pretty reliable in finding these sorts of issues.  I have even found VPN drops this same way.  That turned out to be dirty fiber, which I found just like this above.

Thursday, January 15, 2015

Check Point: "Read-only file system" "cannot create regular file" / Cant Write To External 2T Hard Drive

I have been getting prepared to do an upgrade of a Check Point management station to R77.20.  One of the things I am getting ready to do is to move the log files over to an external hard drive, so that we can import them back in and no logs are lost.  Ill do the migrate import in after I do a fresh install of R77.20 on the new management station.  Then Ill import in the logs so that no logs are lost.
One problem I have run into is when I connect the 2T byte drive in the USB connection, I can mount the drive just fine, and read it contents.  However, I can not write to it.  On read access is available.  This drive is formatted via NTFS.
After a while of trying to figure this out, it turns out that I have to have some other install package (Fuse, I think) to make it work.  Well, I dont want to do that on my Check Point management station, or any enforcement module for that matter.  So, I formatted my 2T drive to be exFAT instead.  This seems to have worked for me.  I can now write to this drive.
Here are some things that I ran into that you might recognize if you are running into this issue:
One of the error messages I got when trying to write to the external drive:
cp: cannot create regular file `/usb-storage/2014-10-01_235900.logptr': Read-only file system

[Expert@CP:0]# cat /proc/mounts
...
/dev/sdb1 /usb-storage ntfs ro,uid=0,gid=0,fmask=0177,dmask=077,nls=utf8,errors=continue,mft_zone_multiplier=1 0 0

Notice the RO on the mounted drive above.
So now I have found a SK on Check Point that actually looks like is the fix.  Its SK61081.  Here is what I did in CLI.

[Expert@CP1:0]# cat /proc/partitions
major minor  #blocks  name

   8     0  245175336 sda
   8     1     305203 sda1
   8     2   10514542 sda2
   8     3  234348187 sda3
 253     0   33554432 dm-0
 253     1    5242880 dm-1
 253     2    1048576 dm-2
 253     3   62914560 dm-3
 253     4    6127616 dm-4
   8    16 1953481728 sdb
   8    17 1953480704 sdb1

[Expert@CP1:0]# parted sdb1
Error: Could not stat device sdb1 - No such file or directory.
Retry/Cancel? cancel
[Expert@CP1:0]# parted /dev/sdb1
GNU Parted 1.8.1
Using /dev/sdb1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
(parted) print

Model: Unknown (unknown)
Disk /dev/sdb1: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

(parted) mkpart primary 0.000 2000000.000
(parted) print

Model: Unknown (unknown)
Disk /dev/sdb1: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      17.4kB  2000GB  2000GB               primary

(parted) quit
[Expert@CP1:0]# mkfs -t ext3 /dev/sdb1
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
244187136 inodes, 488370176 blocks
24418508 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
14904 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 30 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[Expert@CP1:0]# mount -t ext3 /dev/sdb1 /usb-storage
[Expert@CP1:0]# mount
/dev/mapper/vg_splat-lv_current on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/vg_splat-lv_log on /var/log type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/sdb1 on /usb-storage type ext3 (rw)   <--- Notice it says (rw) now!
[Expert@CP1:0]#
[Expert@CP1:0]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_splat-lv_current
                       31G  5.8G   24G  20% /
/dev/sda1             289M   64M  211M  24% /boot
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/mapper/vg_splat-lv_log
                       59G  4.6G   51G   9% /var/log
/dev/sdb1             1.8T  196M  1.7T   1% /usb-storage

Finally got this working.  

Wednesday, January 14, 2015

Brocade ICX6610: Switch Stack Configuration Template

I happened to have a template ready to go for configuring a stack for the ICX6610s.  So this is the steps for configuration below.

1.  On Each 6610:
config t
stack enable
exit
wr mem

2.  Connect the stack together, per this link here: Stacking Cables Configuration

3.  Let the stack build.  Verify with "show stack" command.
Then do the following commands:
config t
hitless-failover enable
stack mac XXXX.XXXX.XXXX   <--- (mac of the primary switch when you do a show stack)
exit
wr mem


Tuesday, January 13, 2015

Check Point: Live Captures On The Enforcement Module

Sometimes you just need to see more that what SmartTracker will give you.  Well, I do.  To effectively troubleshoot, you need to know where things are failing in the firewall, IF its failing there.  I've written about packet captures on the Check Point before, but if you can do this in CLI in any firewall, its preferable.  In this case, I needed to troubleshoot if a packet was actually making it through or not.  Sure, I could see this in Tracker, but I wanted to see the whole picture.  See the packets from 192.168.50.10 destined to 172.16.2.59?  Also, in Check Point, you can tell what stage you are in by the lettering (in orange).  Refer to this post for more info on that.

CheckPoint> fw monitor -e "host (172.16.2.59) or host (4.4.4.231) and port (25), accept;"
 monitor: getting filter (from command line)
 monitor: compiling
monitorfilter:
Compiled OK.
 monitor: loading
 monitor: monitoring (control-C to stop)
[vs_0][fw_1] eth1:o[60]: 192.168.50.10 -> 172.16.2.59 (TCP) len=60 id=4464
TCP: 55909 -> 25 .S.... seq=107657ca ack=00000000
[vs_0][fw_1] eth1:O[60]: 192.168.50.10 -> 172.16.2.59 (TCP) len=60 id=4464
TCP: 55909 -> 25 .S.... seq=107657ca ack=00000000
[vs_0][fw_1] eth1:i[60]: 172.16.2.59 -> 192.168.50.10 (TCP) len=60 id=0
TCP: 25 -> 55909 .S..A. seq=a227cfab ack=107657cb
[vs_0][fw_1] eth1:I[60]: 172.16.2.59 -> 192.168.50.10 (TCP) len=60 id=0
TCP: 25 -> 55909 .S..A. seq=a227cfab ack=107657cb
[vs_0][fw_1] eth1:o[52]: 192.168.50.10 -> 172.16.2.59 (TCP) len=52 id=4465
TCP: 55909 -> 25 ....A. seq=107657cb ack=a227cfac
[vs_0][fw_1] eth1:O[52]: 192.168.50.10 -> 172.16.2.59 (TCP) len=52 id=4465
TCP: 55909 -> 25 ....A. seq=107657cb ack=a227cfac
[vs_0][fw_1] eth1:i[89]: 172.16.2.59 -> 192.168.50.10 (TCP) len=89 id=55140
TCP: 25 -> 55909 ...PA. seq=a227cfac ack=107657cb
[vs_0][fw_1] eth1:I[89]: 172.16.2.59 -> 192.168.50.10 (TCP) len=89 id=55140
TCP: 25 -> 55909 ...PA. seq=a227cfac ack=107657cb
[vs_0][fw_1] eth1:o[52]: 192.168.50.10 -> 172.16.2.59 (TCP) len=52 id=4466
TCP: 55909 -> 25 ....A. seq=107657cb ack=a227cfd1
[vs_0][fw_1] eth1:O[52]: 192.168.50.10 -> 172.16.2.59 (TCP) len=52 id=4466
TCP: 55909 -> 25 ....A. seq=107657cb ack=a227cfd1
[vs_0][fw_1] eth1:o[77]: 192.168.50.10 -> 172.16.2.59 (TCP) len=77 id=4467
TCP: 55909 -> 25 ...PA. seq=107657cb ack=a227cfd1
[vs_0][fw_1] eth1:O[77]: 192.168.50.10 -> 172.16.2.59 (TCP) len=77 id=4467
TCP: 55909 -> 25 ...PA. seq=107657cb ack=a227cfd1
[vs_0][fw_1] eth1:i[52]: 172.16.2.59 -> 192.168.50.10 (TCP) len=52 id=55141
TCP: 25 -> 55909 ....A. seq=a227cfd1 ack=107657e4
[vs_0][fw_1] eth1:I[52]: 172.16.2.59 -> 192.168.50.10 (TCP) len=52 id=55141
TCP: 25 -> 55909 ....A. seq=a227cfd1 ack=107657e4
[vs_0][fw_1] eth1:i[75]: 172.16.2.59 -> 192.168.50.10 (TCP) len=75 id=55142
TCP: 25 -> 55909 ...PA. seq=a227cfd1 ack=107657e4
[vs_0][fw_1] eth1:I[75]: 172.16.2.59 -> 192.168.50.10 (TCP) len=75 id=55142
TCP: 25 -> 55909 ...PA. seq=a227cfd1 ack=107657e4
[vs_0][fw_1] eth1:o[84]: 192.168.50.10 -> 172.16.2.59 (TCP) len=84 id=4468
TCP: 55909 -> 25 ...PA. seq=107657e4 ack=a227cfe8
[vs_0][fw_1] eth1:O[84]: 192.168.50.10 -> 172.16.2.59 (TCP) len=84 id=4468
TCP: 55909 -> 25 ...PA. seq=107657e4 ack=a227cfe8
[vs_0][fw_1] eth1:i[66]: 172.16.2.59 -> 192.168.50.10 (TCP) len=66 id=55143
TCP: 25 -> 55909 ...PA. seq=a227cfe8 ack=10765804
[vs_0][fw_1] eth1:I[66]: 172.16.2.59 -> 192.168.50.10 (TCP) len=66 id=55143
TCP: 25 -> 55909 ...PA. seq=a227cfe8 ack=10765804
[vs_0][fw_1] eth1:o[74]: 192.168.50.10 -> 172.16.2.59 (TCP) len=74 id=4469
TCP: 55909 -> 25 ...PA. seq=10765804 ack=a227cff6
[vs_0][fw_1] eth1:O[74]: 192.168.50.10 -> 172.16.2.59 (TCP) len=74 id=4469
TCP: 55909 -> 25 ...PA. seq=10765804 ack=a227cff6
[vs_0][fw_1] eth1:i[66]: 172.16.2.59 -> 192.168.50.10 (TCP) len=66 id=55144
TCP: 25 -> 55909 ...PA. seq=a227cff6 ack=1076581a
[vs_0][fw_1] eth1:I[66]: 172.16.2.59 -> 192.168.50.10 (TCP) len=66 id=55144
TCP: 25 -> 55909 ...PA. seq=a227cff6 ack=1076581a
[vs_0][fw_1] eth1:o[58]: 192.168.50.10 -> 172.16.2.59 (TCP) len=58 id=4470
TCP: 55909 -> 25 ...PA. seq=1076581a ack=a227d004
[vs_0][fw_1] eth1:O[58]: 192.168.50.10 -> 172.16.2.59 (TCP) len=58 id=4470
TCP: 55909 -> 25 ...PA. seq=1076581a ack=a227d004
[vs_0][fw_1] eth1:i[89]: 172.16.2.59 -> 192.168.50.10 (TCP) len=89 id=55145
TCP: 25 -> 55909 ...PA. seq=a227d004 ack=10765820
[vs_0][fw_1] eth1:I[89]: 172.16.2.59 -> 192.168.50.10 (TCP) len=89 id=55145
TCP: 25 -> 55909 ...PA. seq=a227d004 ack=10765820
[vs_0][fw_1] eth1:o[78]: 192.168.50.10 -> 172.16.2.59 (TCP) len=78 id=4471
TCP: 55909 -> 25 ...PA. seq=10765820 ack=a227d029
[vs_0][fw_1] eth1:O[78]: 192.168.50.10 -> 172.16.2.59 (TCP) len=78 id=4471
TCP: 55909 -> 25 ...PA. seq=10765820 ack=a227d029
[vs_0][fw_1] eth1:i[52]: 172.16.2.59 -> 192.168.50.10 (TCP) len=52 id=55146
TCP: 25 -> 55909 ....A. seq=a227d029 ack=1076583a
[vs_0][fw_1] eth1:I[52]: 172.16.2.59 -> 192.168.50.10 (TCP) len=52 id=55146
TCP: 25 -> 55909 ....A. seq=a227d029 ack=1076583a
[vs_0][fw_1] eth1:o[162]: 192.168.50.10 -> 172.16.2.59 (TCP) len=162 id=4472
TCP: 55909 -> 25 ...PA. seq=1076583a ack=a227d029
[vs_0][fw_1] eth1:O[162]: 192.168.50.10 -> 172.16.2.59 (TCP) len=162 id=4472
TCP: 55909 -> 25 ...PA. seq=1076583a ack=a227d029
[vs_0][fw_1] eth1:i[52]: 172.16.2.59 -> 192.168.50.10 (TCP) len=52 id=55147
TCP: 25 -> 55909 ....A. seq=a227d029 ack=107658a8
[vs_0][fw_1] eth1:I[52]: 172.16.2.59 -> 192.168.50.10 (TCP) len=52 id=55147
TCP: 25 -> 55909 ....A. seq=a227d029 ack=107658a8
[vs_0][fw_1] eth1:i[89]: 172.16.2.59 -> 192.168.50.10 (TCP) len=89 id=55148
TCP: 25 -> 55909 ...PA. seq=a227d029 ack=107658a8
[vs_0][fw_1] eth1:I[89]: 172.16.2.59 -> 192.168.50.10 (TCP) len=89 id=55148
TCP: 25 -> 55909 ...PA. seq=a227d029 ack=107658a8
[vs_0][fw_1] eth1:o[56]: 192.168.50.10 -> 172.16.2.59 (TCP) len=56 id=4473
TCP: 55909 -> 25 ...PA. seq=107658a8 ack=a227d04e
[vs_0][fw_1] eth1:O[56]: 192.168.50.10 -> 172.16.2.59 (TCP) len=56 id=4473
TCP: 55909 -> 25 ...PA. seq=107658a8 ack=a227d04e
[vs_0][fw_1] eth1:o[52]: 192.168.50.10 -> 172.16.2.59 (TCP) len=52 id=4474
TCP: 55909 -> 25 F...A. seq=107658ac ack=a227d04e
[vs_0][fw_1] eth1:O[52]: 192.168.50.10 -> 172.16.2.59 (TCP) len=52 id=4474
TCP: 55909 -> 25 F...A. seq=107658ac ack=a227d04e
[vs_0][fw_1] eth1:i[52]: 172.16.2.59 -> 192.168.50.10 (TCP) len=52 id=55149
TCP: 25 -> 55909 F...A. seq=a227d04e ack=107658ad
[vs_0][fw_1] eth1:I[52]: 172.16.2.59 -> 192.168.50.10 (TCP) len=52 id=55149
TCP: 25 -> 55909 F...A. seq=a227d04e ack=107658ad
[vs_0][fw_1] eth1:o[52]: 192.168.50.10 -> 172.16.2.59 (TCP) len=52 id=4475
TCP: 55909 -> 25 ....A. seq=107658ad ack=a227d04f
[vs_0][fw_1] eth1:O[52]: 192.168.50.10 -> 172.16.2.59 (TCP) len=52 id=4475
TCP: 55909 -> 25 ....A. seq=107658ad ack=a227d04f
 monitor: caught sig 2
 monitor: unloading

Monday, January 12, 2015

Brocade ICX: Remove ICX6610 Switch From Stack

I have, on occasion, had to unconfigure a Brocade ICX out of a stack before.  Sometimes, you just have to re-purpose a switch.  Here is the command that you will want to run, on the ICX, when you have disconnected the stacking cables off the back (of an ICX 6610)(front of an ICX 6450).
6610#stack unconfigure clean
This command is not available on standalone or Active Controller
6610#stack unconfigure me
This unit will recover pre-stacking startup config. Are you sure? (enter 'y' or 'n'): y
remove stack bootup flash
6610#

Saturday, January 10, 2015

Friday, January 9, 2015

Brocade ICX: Can You Monitor Multiple Ports In One Capture?

There are times when you just need to monitor multiple ports at the same time.  One example is when you have and active/active firewall setup, and you need to monitor the traffic going to both internal interfaces.  Can you do this on the Brocade ICX switches?  Yes, you can.  Here is a piece of the config as an example:
...
mirror-port ethernet 1/1/37
!
interface ethernet 1/1/46
 mon ethe 1/1/37 both
!
interface ethernet 1/1/47
 mon ethe 1/1/37 both

Thursday, January 8, 2015

The "Title" Game In IT

I'm an IT consultant, and I do go around from customer to customer on a daily basis.  Sometimes, I see multiple customers in one day.  I guess you can say that in the IT services business, you kind of get around.
One of the things I think is interesting is the titles that people get.  Anything from Network Analyst to VP of IT.  But have you ever wondered how a lot of times, IT titles don't necessarily match the skillset of the guy holding that title?  I think this is just misleading.
For example, I met a guy once in the education sector that had a title of Network Analyst.  But if the truth was told, he couldn't tell you what a switch really did.  He could tell you it connects the network together, and I guess that is true.  But he couldn't tell you what a CAM table is.  He couldn't tell you what ARP is.  And I'm sure he couldn't configure a port on the switch to be an access port.  Would you call that guy a Network Analyst?  I wouldn't.  I would really consider this guy a PC tech, based on what I know.
Now I'm not knocking the guy.  He has experience that is valuable to the company, but not network experience.  His title really should have been something like IT technician I, PC technician, or something like that.  I mean, you set the guy up for failure one day when he applies for a real network position at another company when he really doesn't have any real world experience in network related things.
I also know people who are very talented in specific areas, like security.  But they fall under the network group and therefore, have a network title.  How is that fair or accurate?  I mean, if you have a guy whose focus is security in the company, you would expect he might have a title like Security Engineer, Security Analyst, etc.  This could affect his money making, meaning he should be making more.
On the flip side, I actually worked for a guy who was a VP of IT.  His background was a cable puller.  I think he might have touched a few servers at one time, but he was absolutely worthless as a "VP of IT".  He made terrible decisions that affected the way IT worked in that area.  He would get the opinions of the real technical people, only to ignore them and make a decision on his own or based on pressure from upper management, who also had no clue.  In fact, not one person had any respect for this guy.  I certainly did not.  (I still laugh at this one.)
I suppose you can find this all over the place.  I see it all the time.  I even know people who have an Architect title, but rarely do they do any architecture of a network.
I think companies need to really consider the titles of the employees they have.  Its not like HR has any real clue as to what a real title should be for someone with particular skillset.  Its really the IT manager who should give this consideration.  I mean, what does HR know about different areas of IT?
Its just interesting how you can go into companies and find the high powered titles, when in reality, their skillset doesn't match.  It sometimes leave you, after a first impression of a technical guy, with questions like "Shouldn't you know this?"  Anyway, I just find this interesting.  Maybe all fields are like this, but I know for sure IT is like this.

Wednesday, January 7, 2015

Fiber Checks

Im pretty sure I have posted about this before, but one thing I had to do recently was to check some fiber.  Being a network guy and not a cable guy, I really dont have any good electronic tools to check fiber runs.  But, what I do have is a cell phone with a camera on it.  I personally never look into a GBIC or a fiber where a laser could be coming out.  I know there are arguments that say its ok, but I just like to NOT do it.  My cell phone is much more forgiving than my eyes.  So I use my cell phone to check for the laser.  You can see below that I do have a laser coming out of the fiber run in this case (circled).  Anyway, glad that was not the problem in this case.

Tuesday, January 6, 2015

Check Point: Who Am I Logged In As?

If you have access into the Check Point, and for whatever reason, you need to figure out who you are logged in as (which has happened to me once before), then you can always run the following command in Expert mode:

[Expert@CheckPoint:0]# whoami
admin
[Expert@CheckPoint:0]#

Ok, Im logged in as admin.  Good to know.

Monday, January 5, 2015

Broadcast Storms And The Havoc (And Headache) They Reek

Sometimes, what looks like one problem, can actually be another.  I got a call from a school system saying that their network was down for one particular school.  No Internet.  No server access.  Nothing.
So when I got there, things seemed to be ok.  They could get to the Internet, servers, etc.  Everything appeared to be ok.  Then it went down again.  This problem was very inconsistent.  You couldnt even ping anything.  But then, after some time, it would be fine.
Well, when I was there, I really couldn't find anything wrong.  Except once, when the problem actually happened.  It appeared to me, at that moment, like the layer 3 core was acting up.  When the problem happened, I couldnt even ping the next hop MPLS router (on the same subnet).  This problem showed some really odd symptoms.  So, I broke the switch stack (Brocade ICX 6610s) and left the last switch in place to run as the core by itself.  The problem seemed to go away.  Until the next day, when the problem happened again.
I showed up and guess what.  The problem was gone again.  So I sit down and just start looking at configs, spanning tree, interface statistics, cpu utilization, etc.  What in the world is going on here.  Then it happened right in front of me.  CPU shot up to 33% (from 1%).  Internet was down and network was hosed again.  So I ran a packet capture.  Nothing on the first 4 VLANs that I looked at.  Then I found it.  The 5th VLAN I looked at was flooded with broadcasts.
6610#sho cpu-utilization
32 percent busy, from 9 sec ago
1   sec avg: 32 percent busy
5   sec avg: 32 percent busy
60  sec avg: 32 percent busy
300 sec avg: 32 percent busy

So, I disconnected the fiber to the closet that VLAN went to, and CPU dropped back down to 1% on the core.  Ok, leave that disconnected, and go to that closet.
I got to that closet and plugged in my packet analysis tool (seen above).  Again, flooded when I hit the right VLAN, but no other VLAN.  So, what do I do?  Its a stack of 2 ICX6450s.  I start unplugging one port at a time until I find the flooding settles down.  It just so happens the second port I unplugged calmed the network down right away.  It was a PC that was connected at the other end.  It appears the NIC was flaking out.  Intermittent broadcast storms.  Not a good situation, but with the right tools, I found the problem as it was happening.  Packet captures are your friend!

Sunday, January 4, 2015

Sunday Thought: Read x20

I read an article about reading each book of the Bible twenty times in a row. That's right. x20!  I'm going to start with a small book first, like Colossians. Read it twenty times in a row, before I read any other book of the Bible. Let's see what results that will produce.

Friday, January 2, 2015

I Cant Help, But I Know Someone Who Can

Just a little job thought for us all.  I came across this pro bono thing where I really wanted to help this guy out.  He had a worthy cause and it really bothered me that I really just didnt have anything to offer this guy to help out his cause.  I feel like I have a pretty decent IT skillset, none of which he needed.
I did, however, know a guy who could help him out.  TRUTH: Its ok to not be able to help someone with your skillset.  Sometimes in life, it is enough just to know someone else who can help.