Thursday, May 31, 2012

Cisco Nexus 5000/2000 Install (Part 1): How To Configure Redundant 5000s With Redundant 2000 FEXs

Cisco Nexus 5000/2000 Install: How To Configure Redundant 5000s With Redundant 2000 FEXs

Im adding these links in as an afterthought.  I have a four part series on this blog about this 5000/2000 install, in case you are interested in seeing them all.  This article you are on is the first of the four: 
For the notes on "vPC", click here.
For the notes on the "FEXs", click here.
For some config examples without explanations, click here.

My components: (2) 5548UP Nexus Switches, (2) 2232PP Fabric Extenders (FEX), (4) glc-sx-mm= gbics, and (8) fet-10g gbics

Here is the topology of the Cisco Nexus setup that Im looking to do, with one exception.  I only had one fiber run to each FEX instead of two like the diagram shows:
This was an interesting install.  I ran into a few roadblocks along the way, but Ill highlight what I did and what I learned in this posting. 

First, I learned from TAC that when doing an install like this, you must start configuration of the 5000s first, then move to the vPC config between the two 5000s.  This will save you some headache when it comes to getting the 5000s to see the 2000s. 
Second, if TAC tells you that the 5000 will only see one 2232PP device instead of being able to see two, its not true.  I had two TAC guys tell me this.  Here is my proof that it will see more than one.  Actually, I think it supports up to 12:

N5K-1# sh fex
  FEX         FEX           FEX                       FEX              
Number    Description      State            Model            Serial    
100        FEX0100                Online    N2K-C2232PP-10GE   SSI155XXXX1
101        FEX0101                Online    N2K-C2232PP-10GE   SSI155XXXX2

Third, the Nexus config does have some changes from the Catalyst.  Its mostly similar, but one thing I really like about it is that its like an ASA in that you can run most commands no matter where you are in the command structure.  I dont like having to exit back one just to run a command.  I can even do a 'copy run start' from config mode.  I like that.
Fourth, my understanding is that you want to use your management interfaces for the vPC keep-alives. TAC tells me that you will have spratic issues come up if you dont use the management interfaces.  They did not elaborate on what issues I might face, but they made sure to mention it to me.
Fifth, you can not do a 'write mem' on the Nexus 5000.  You can only do a 'copy run start'.
One last thing.  When you are configuring these 5000s in a redundant setup, like in the topology above, you really should work on both 5000s at the same time.  Now this is my own opinion, but I found that this is helpful when getting the two ready for the 2000s.

So, here is what my experience was like:
I first started out changing the password.  The first time you log in, it asks you to change it, so I did.  You will have a login of admin and whatever your new password will be.
I then went on to configure the management interface.  I think the best approach for me personally is to configure both 5548UPs at the same time.  Looking back, I think it will be easier for me to know where I am in the config steps.  Plus, it seems that some things just need to be done on both switches for things to go smoothly (in a topology like Im doing).  So, lets configure the management interface:

interface mgmt0
  ip address
vrf context management
ip route

Now you have your management interface configured.  Do the other 5000 as well.
Configure the hostname.  I chose N5K-1 and N5K-2.  'hostname N5K-1'. 
You need to enable to services you will need on the Nexus.  Here is what I enabled on mine:

feature telnet
feature vpc
feature lacp
feature fex

I configured my trunk ports to my core 6500 now.  You got to hook into the network somehow, you may as well do it now.  I did trunk ports so I could carry multiple vlans.  I also tied a separate vlan on my 6500 for my management interfaces.  Dont ask why, but Im having to bond (5) Cat5 cable together.  My 6500 didn't have enough fiber ports.
Here is the trunk config I did on the 5000:

interface port-channel10
  description To_Core_Switch
  switchport mode trunk
switchport trunk allowed vlan all
speed 1000

interface Ethernet1/1-5
  description To_Core_Switch
switchport mode trunk
speed 1000
  channel-group 10

I actually ended up doing two of these (5) port port-channel trunks, to two different core switches for redundancy.
Now its time to configure the vPC domain.  'vpc domain 1'  I used 1, but you can use whatever number you want (within a certain limit, which I cant remember now).
Peer keep-alive config., remember to use your management interfaces:

 peer-keepalive destination ( is the management interface IP on my other 5000).

Now its time to configure the vPC peer 'link'.  This is the link between the two 5000s.  I did this as a trunk port, per Cisco's recommendation. 

 int ethernet 1/15-16
  switchport mode trunk
  speed 1000     (notice I have to say 1000 for 1G connection. 10G is the default)
  channel-group 1516 mode active  (keyword 'active' tells it to use LACP)

interface port-channel1516
  switchport mode trunk
  spanning-tree port type network
  speed 1000
  vpc peer-link

NOTE*** In the above config, for the channel-group command, I used keyword 'on', and it caused problems.  It wasn't until I used LACP (keyword 'active') that things were steady and reliable.
That should take care of the vPC link between the two 5000s.  **Note: I used the following GBICS for this connectivity: glc-sx-mm= (1G fiber GBICS)

Now, you should see something like this below for successful configuration:
N5K-1(config-if-range)# sh vpc brief
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 1
Peer status                     : peer adjacency formed ok     
vPC keep-alive status           : peer is alive                
Configuration consistency status: success
Per-vlan consistency status     : success                      
Type-2 consistency status       : success
vPC role                        : secondary                    
Number of vPCs configured       : 0  
Peer Gateway                    : Disabled
Dual-active excluded VLANs      : -
Graceful Consistency Check      : Enabled

Now its time to configure the FEXs (the 2232PPs). 
Now keep in mind that I did not configure the following three commands.  Even though they are needed, these commands were defaulted in when I added the fex command into the port-channel.  Again, these three commands below, I did not put in myself.  The switch did it on its own.

fex 100
  pinning max-links 1
  description "FEX0100"

They are needed, but again, when I did this port-channel config below, the above commands were put in automatically:

interface port-channel6
  switchport mode fex-fabric
  fex associate 100
  vpc 6

Now the interface (again, only one interface since I only had one fiber going to each FEX from each 5548):

interface Ethernet1/6
  description *** To FEX 100 ****** To FEX 100 ***
  switchport mode fex-fabric
  fex associate 100
  channel-group 6

I also added in for the second FEX (2232PP):
interface port-channel7
  switchport mode fex-fabric
  fex associate 101
  vpc 7

interface Ethernet1/7
  description *** To FEX 101 ****** To FEX 101 ***
  switchport mode fex-fabric
  fex associate 101
  channel-group 7

And again, when I added the commands above for the second FEX, it added these commands below on its own:

fex 101
  pinning max-links 1
  description "FEX0101"

Now its probably important to note that I used the following for connectivity from the 5548s to the 2232s.  Here is what I used:
fet-10g (10G fiber GBICS)
LC-LC/10GIG/AQ/1M (10G fiber patch cables)

Now some notes when you first connect to the FEXs:
The 2232PP has to download the same image that the 5548UP has.  See below:
N5K-1(config-if)# sh fex
  FEX         FEX           FEX                       FEX
Number    Description      State            Model            Serial
100        FEX0100        Image Download    N2K-C2232PP-10GE   SSIXXXXXXX1

Once it is done downloading (which takes a few minutes), it will show this below:
N5K-1(config-if)# sh fex
  FEX         FEX           FEX                       FEX
Number    Description      State            Model            Serial
100        FEX0100                Online    N2K-C2232PP-10GE   SSIXXXXXXX1

Another interesting command is the 'show interface fex':
N5K-1(config-if)# sh inter fex
     Fabric      Fabric       Fex                FEX
Fex  Port      Port State    Uplink    Model         Serial
100    Eth1/6        Active     8    N2K-C2232PP-10GE  SSIXXXXXXX1

Probably another useful insight at this point is that all the interfaces are 10G by default.  It comes that way on the 5000.  IF you want the links to be any other speed except 10G, you have to tell it.  Here is what you might see:
N5K-1(config-if)# sh int brief
Ethernet      VLAN    Type Mode   Status  Reason                   Speed     Port
Interface                                                                    Ch#
Eth1/1        1       eth  trunk  down    Link not connected         1000(D) 10
Eth1/2        1       eth  trunk  down    Link not connected         1000(D) 10
Eth1/3        1       eth  trunk  down    Link not connected         1000(D) 10
Eth1/4        1       eth  trunk  down    SFP Validation Failed      1000(D) 10

If you see that "SFP Validation Failed", then probably if you go on the port-channel (or interface if you are not doing port-channel) and set the speed correctly, it will resolve this issue.  It did for me.

Now it will be time to configure any vlan that needs to be on the switch, plus any vlan configuration on a port that you need.  They will be the same commands as on a Catalyst, except you dont have to go into a vlan database.
So a small recap: config is done on and between the 5000s first, then add the 2000s.