Search This Blog

Friday, September 12, 2014

Cisco Nexus 5K with 2248PQ connectivity problems



The Cisco Nexus 2248PQ is a 48 1/10 Gigabit Ethernet and FCoE host interfaces (SFP+) and 4 QSFP+ Gigabit Ethernet and FCoE fabric interfaces (QSFP+).




Like all Nexus 2200 switches he uses one or more of his uplinks ports to connect to the parent switch, for the 2248PQ there are 4x 40G QSFP+.

I had 3 options at my hand:
QSFP-H40G-CU3M – which is twinax passive cable

QSFP-4SFP10G-CU3M – which is copper breakout cable 40G to 4x 10G

And FET-40G transceiver with straight MPO-12 cable with MTP on both sides

MPO-12 straight
 
MTP to MTP connectors

Cisco FET-40G brown label


First I used the QSFP-4SFP10G-CU3M to connect 2248PQ port 1 to ETH 1/1-4 ports at the parent switch, this was simply straight forward, and the switch discovers the 2248PQ very quickly.

This is the ports configuration:

interface port-channel103
  description FEX-103
  switchport mode fex-fabric
  fex associate 103
!
interface Ethernet1/1-4
  switchport mode fex-fabric
  fex associate 103
  channel-group 103
  no shutdown

Then I tried to use the QSFP-H40G-CU3M and the FET-40G, along with the MPO-12 cable, but the link didn’t come up:


N5K-1(config-if)# show interface eth 3/25 brief

--------------------------------------------------------------------------------
Ethernet      VLAN    Type Mode   Status  Reason                   Speed     Por
t
Interface                                                                    Ch
#
--------------------------------------------------------------------------------
Eth3/25       1       eth  fabric down    Link not connected          40G(D) --

After a little research I found the following document:

Although the 2248PQ has 4x 40G QSFP+ physical ports we need to treat those ports as 10G when connecting to the parent home, hence the 2248PQ have 16x 10G uplink ports which represented by 4x 40G ports!

So this is the reason why the QSFP-4SFP10G-CU3M works and the other 2 doesn’t. 

But how do we connect 40G interface to the parent switch and make it believe its 4 ports 10G?

According to the module, where the 40G port reside on your parent switch, in my case it was on expansion module 3:

N5K-1(config)#interface breakout slot 3 port 25-26 map 10g-4x
!
N5K-1(config)#poweroff module 3
N5K-1(config)#no poweroff module 3

Few notes:
      - We must configure breakout interface in groups – in my case I had to configure both 40G ports which reside on module 3
      - We need to reboot the module after changing the settings to take effect and this may take some time (about the time it’s take to the parent switch to reboot)
      - This cause interruption to the network activity (or more specifically to the hosts connected to the module which we reboot)

After that we get 4 more virtual ports under the single physical port - eth3/25/1-4, which we need to configure the same way as the QSFP-4SFP10G-CU3M.
B.T.W - The FET which stands for Fabric Extender Transceiver is a normal transceiver which Cisco marketing has decided to sell cheap but you won't be able to use them for any other purpose beside connecting FEX's.

B.T.W 2 - i had to use the command 'service unsupported-transceiver' in order for the FET-40G to work although it's genuine Cisco QSFP!!! I'm still waiting for some answers from Cisco TAC.


5 comments:

  1. What model 5K do you have and what version? I have a 5548UP running 6.0(2)N1(2a). I'm attempting to connect a 2248PQ with the breakout cable. All I get with the interface breakout command is an ERROR: Invalid Port Range specified no matter what range I use. Show interface gives me Transceiver validation failed. Any thoughts on what I may be missing?

    ReplyDelete
  2. Use service unsupported-transceiver, i know it's hidden command and not supported by Cisco TAC but in the last 2 month I'm busy with them on this issue and so far no solution seems in the end....

    ReplyDelete
  3. hi,
    thanks for your post.
    we have the same case N5672UP with 2 N2K-C2248PQ
    we have the FET-40G adapter and a straight MPO
    did you conect both nexus directly with the MPO cable?
    can you share your configuration?

    ReplyDelete
  4. yes, i have connected the Nexus 56128 directly to the Nexus 2248PQ using the MPO cable, between the two Nexus 56128 i have used native 40G cable.
    the port configuration is the only change i have done on the configuration related to this issue, the rest of the configuration is pretty regular.

    ReplyDelete
  5. Hi,

    Do you have the CISCO Part Number for this?

    MPO-12 straight
    FET-40G transceiver with straight MPO-12 cable with MTP on both sides

    ReplyDelete