Im adding these links in as an afterthought. I have a four part series on this blog about this 5000/2000 install, in case you are interested in seeing them all. This article you are on is the first of the four:
For the notes on "vPC", click here.
For the notes on the "FEXs", click here.
For some config examples without explanations, click here.
My components: (2) 5548UP Nexus Switches, (2) 2232PP Fabric Extenders (FEX), (4) glc-sx-mm= gbics, and (8) fet-10g gbics
Here is the topology of the Cisco Nexus setup that Im looking to do, with one exception. I only had one fiber run to each FEX instead of two like the diagram shows:
First, I learned from TAC that when doing an install like this, you must start configuration of the 5000s first, then move to the vPC config between the two 5000s. This will save you some headache when it comes to getting the 5000s to see the 2000s.
Second, if TAC tells you that the 5000 will only see one 2232PP device instead of being able to see two, its not true. I had two TAC guys tell me this. Here is my proof that it will see more than one. Actually, I think it supports up to 12:
N5K-1# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------
100 FEX0100 Online N2K-C2232PP-10GE SSI155XXXX1
101 FEX0101 Online N2K-C2232PP-10GE SSI155XXXX2
Third, the Nexus config does have some changes from the Catalyst. Its mostly similar, but one thing I really like about it is that its like an ASA in that you can run most commands no matter where you are in the command structure. I dont like having to exit back one just to run a command. I can even do a 'copy run start' from config mode. I like that.
Fourth, my understanding is that you want to use your management interfaces for the vPC keep-alives. TAC tells me that you will have spratic issues come up if you dont use the management interfaces. They did not elaborate on what issues I might face, but they made sure to mention it to me.
Fifth, you can not do a 'write mem' on the Nexus 5000. You can only do a 'copy run start'.
One last thing. When you are configuring these 5000s in a redundant setup, like in the topology above, you really should work on both 5000s at the same time. Now this is my own opinion, but I found that this is helpful when getting the two ready for the 2000s.
So, here is what my experience was like:
I first started out changing the password. The first time you log in, it asks you to change it, so I did. You will have a login of admin and whatever your new password will be.
I then went on to configure the management interface. I think the best approach for me personally is to configure both 5548UPs at the same time. Looking back, I think it will be easier for me to know where I am in the config steps. Plus, it seems that some things just need to be done on both switches for things to go smoothly (in a topology like Im doing). So, lets configure the management interface:
interface mgmt0
ip address 10.10.10.4/24
exit
vrf context management
ip route 0.0.0.0/0 10.10.10.1
Now you have your management interface configured. Do the other 5000 as well.
Configure the hostname. I chose N5K-1 and N5K-2. 'hostname N5K-1'.
You need to enable to services you will need on the Nexus. Here is what I enabled on mine:
feature telnet
feature vpc
feature lacp
feature fex
I configured my trunk ports to my core 6500 now. You got to hook into the network somehow, you may as well do it now. I did trunk ports so I could carry multiple vlans. I also tied a separate vlan on my 6500 for my management interfaces. Dont ask why, but Im having to bond (5) Cat5 cable together. My 6500 didn't have enough fiber ports.
Here is the trunk config I did on the 5000:
interface port-channel10
description To_Core_Switch
switchport mode trunk
switchport trunk allowed vlan all
speed 1000
interface Ethernet1/1-5
description To_Core_Switch
switchport mode trunk
speed 1000
channel-group 10
I actually ended up doing two of these (5) port port-channel trunks, to two different core switches for redundancy.
Now its time to configure the vPC domain. 'vpc domain 1' I used 1, but you can use whatever number you want (within a certain limit, which I cant remember now).
Peer keep-alive config., remember to use your management interfaces:
peer-keepalive destination 10.10.10.5 (10.10.10.5 is the management interface IP on my other 5000).
Now its time to configure the vPC peer 'link'. This is the link between the two 5000s. I did this as a trunk port, per Cisco's recommendation.
int ethernet 1/15-16
switchport mode trunk
speed 1000 (notice I have to say 1000 for 1G connection. 10G is the default)
channel-group 1516 mode active (keyword 'active' tells it to use LACP)
interface port-channel1516
switchport mode trunk
spanning-tree port type network
speed 1000
vpc peer-link
NOTE*** In the above config, for the channel-group command, I used keyword 'on', and it caused problems. It wasn't until I used LACP (keyword 'active') that things were steady and reliable.
That should take care of the vPC link between the two 5000s. **Note: I used the following GBICS for this connectivity: glc-sx-mm= (1G fiber GBICS)
Now, you should see something like this below for successful configuration:
N5K-1(config-if-range)# sh vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary
Number of vPCs configured : 0
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Now its time to configure the FEXs (the 2232PPs).
Now keep in mind that I did not configure the following three commands. Even though they are needed, these commands were defaulted in when I added the fex command into the port-channel. Again, these three commands below, I did not put in myself. The switch did it on its own.
fex 100
pinning max-links 1
description "FEX0100"
They are needed, but again, when I did this port-channel config below, the above commands were put in automatically:
interface port-channel6
switchport mode fex-fabric
fex associate 100
vpc 6
Now the interface (again, only one interface since I only had one fiber going to each FEX from each 5548):
interface Ethernet1/6
description *** To FEX 100 ****** To FEX 100 ***
switchport mode fex-fabric
fex associate 100
channel-group 6
I also added in for the second FEX (2232PP):
interface port-channel7
switchport mode fex-fabric
fex associate 101
vpc 7
interface Ethernet1/7
description *** To FEX 101 ****** To FEX 101 ***
switchport mode fex-fabric
fex associate 101
channel-group 7
And again, when I added the commands above for the second FEX, it added these commands below on its own:
fex 101
pinning max-links 1
description "FEX0101"
Now its probably important to note that I used the following for connectivity from the 5548s to the 2232s. Here is what I used:
fet-10g (10G fiber GBICS)
LC-LC/10GIG/AQ/1M (10G fiber patch cables)
Now some notes when you first connect to the FEXs:
The 2232PP has to download the same image that the 5548UP has. See below:
N5K-1(config-if)# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------
100 FEX0100 Image Download N2K-C2232PP-10GE SSIXXXXXXX1
Once it is done downloading (which takes a few minutes), it will show this below:
N5K-1(config-if)# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------
100 FEX0100 Online N2K-C2232PP-10GE SSIXXXXXXX1
Another interesting command is the 'show interface fex':
N5K-1(config-if)# sh inter fex
Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
---------------------------------------------------------------
100 Eth1/6 Active 8 N2K-C2232PP-10GE SSIXXXXXXX1
Probably another useful insight at this point is that all the interfaces are 10G by default. It comes that way on the 5000. IF you want the links to be any other speed except 10G, you have to tell it. Here is what you might see:
N5K-1(config-if)# sh int brief
--------------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch#
--------------------------------------------------------------------------------
Eth1/1 1 eth trunk down Link not connected 1000(D) 10
Eth1/2 1 eth trunk down Link not connected 1000(D) 10
Eth1/3 1 eth trunk down Link not connected 1000(D) 10
Eth1/4 1 eth trunk down SFP Validation Failed 1000(D) 10
If you see that "SFP Validation Failed", then probably if you go on the port-channel (or interface if you are not doing port-channel) and set the speed correctly, it will resolve this issue. It did for me.
Now it will be time to configure any vlan that needs to be on the switch, plus any vlan configuration on a port that you need. They will be the same commands as on a Catalyst, except you dont have to go into a vlan database.
So a small recap: config is done on and between the 5000s first, then add the 2000s.
