MXL -- Can't ping management IP when MXL is installed in Fabric A

Thanks to all of you who responded to this problem I presented on Wednesday.  I’m not sure if anyone provided a solution that is consistent with the resolution we used, but here’s a brief summary that I shared with the customer.  You could very well encounter this problem in the future and you could spend hours working on the MXL when in its actually a problem with the CMC.  After spending several hours trouble-shooting with two different TAC engineers, they escalated to a Master Engineer who was quite confident he knew what the fix would be and sure enough it worked.  Note that we were trying to ping the management IP and the customer was using only a LOM for Fabric A.  No mezz cards were installed.

The problem is a known issue and the Master Engineer said they have not been able to debug the root cause, so the what was provided is really a preventative work around.  BTW, we also did a factory reset on the MXL and configured it from scratch while inserted in Fabric A but this didn’t work.  The only solution that worked was to use the rack rest command on the CMC.  Before executing the rack reset command, TAC collected several logs in an attempt to determine the root cause.

Summary for the Customer:

-------------------------------------

Re: Dell TAC Case 910245438 – Cannot access management IP of MXL when installed in Fabric Slots A1 or B1

The problem as reported to us yesterday has been resolved on the M1000e chassis in question, but I look forward to the customer confirming this at your earliest convenience by moving the MXLs back into Fabric slots A1 and A2.  Please also confirm that the CMC is configured as expected since we did an upgrade and a re-configuration.  I left the MXLs installed in the B1/B2 fabric slots and the B22s installed in the A1/A2 slots since this is how I found them when we started trouble shooting this morning (Thursday, April 23) and wasn’t sure if I would impact any ongoing traffic testing traversing Fabric A1/A2.  Before leaving this evening, I moved the MXLs from Slots C1 /C2 to B1/B2 to A1/A2 and was able to successfully ping the management IP addresses (10.26.17.240/241) with each move.  If there is any problem please contact me immediately.

Resolution:

The problem was resolved by running a rack reset command and then reconfiguring the CMC.  Our Dell support staff advises that this is a one-time event on a M1000e chassis and it can easily be prevented for any subsequent deployments of the M1000e chassis.

Additional notes:

The MXLs were upgraded from firmware Release 9.4 to 9.6.

The CMC was upgraded to 5.01.

These upgrades should have no effect on the capabilities of the CMC or the MXLs in context of the testing being performed by Robert and Tommy, but I recommend moving the MXLs to 9.7 in the not too distant future since OpenFlow 1.3 is supported on 9.7 while OpenFlow 1.0 is supported on 9.6.  Although 9.7 was released earlier this year, we would like to see a few more weeks of field exposure before recommending DirecTV move to this release.

Bill Tozer

Network Systems Engineer

Office: 805-498-2959

Mobile: 805-490-7409

Dell | Enterprise Solutions, Networking

Bill_Tozer@Dell.com

From: Tozer, Bill
Sent: Wednesday, April 22, 2015 4:59 PM
To: Cereijo, Manny; WW Networking Domain; Arrata, William
Subject: RE: MXL -- Can't ping management IP when MXL is installed in Fabric A

Thanks Manny,

I’ll try that when I’m on site tomorrow morning.

Bill

From: Cereijo, Manny
Sent: Wednesday, April 22, 2015 4:57 PM
To: Tozer, Bill; WW Networking Domain; Arrata, William
Subject: RE: MXL -- Can't ping management IP when MXL is installed in Fabric A

Dell - Internal Use - Confidential

Bill,

Is the MXL connecting to the same management network when in Fabric A, B and C?

Can they connect to the MXL via the CMC? Try to SSH or telnet to the CMC, then connect to the MXL with the connect switch-a1 command.

Manny

From: Tozer, Bill
Sent: Wednesday, April 22, 2015 7:48 PM
To: WW Networking Domain; Arrata, William
Subject: MXL -- Can't ping management IP when MXL is installed in Fabric A

Has anyone seen any issues with not being able to ping the management IP (or access via SSH) of an MXL when installed in Fabric A?  My customer has reported that everything works fine when the MXL is installed in Fabric B or C, but when the MXL is moved to Fabric A, they can no longer connect to it.

Midplane Version of the M1000e is 1.1

Release of the MXL is 9.4, but we will be upgrading it to Release 9.7 ASAP and opening a support case.

Bill Tozer

Network Systems Engineer

Office: 805-498-2959

Mobile: 805-490-7409

Dell | Enterprise Solutions, Networking

Bill_Tozer@Dell.com