DS8000 Service Documentation Version 6.3.3

MAP7000 Entry point for storage facility private network problems

This MAP provides guidance on repairing all network problems with the storage facility private network.

Related tasks:
How to start the service information center

MAP7000 Section-1

Procedure

  1. Private network problems can be caused by management console (HMC) or CEC enclosure problems that directly or indirectly affect their Ethernet ports.

    Display open serviceable events and repair any that could be related before going to the next step.

  2. If the storage complex has two active HMCs (primary MC 1 and secondary MC 2), ensure you have displayed open serviceable events on both HMCs.
    Serviceable events with the following SRCs are displayed only on the HMC that created them; they are not replicated on the other HMC:
    • BE17xxxx
    • BEB0001x
    • BEB10012
    • BEB20010
    • BEB20020
    • BEB20021
    • BEFxxxxx
    • Exxxxxxx
  3. Use Table 1 to find your purpose for using this MAP.
    Table 1. Entry for storage facility private network problems
    Purpose Go to:
    You were sent here to use the Network Topology Tool to verify the connectivity of both storage facility private networks (gray and black) MAP7001 Using the network topology tool
    You were sent here from a serviceable event with an SRC (system reference code) MAP7000 Section-2 Using an SRC
    You were sent here from another MAP to verify network connectivity MAP7000 Section-3 Visual Checks
    You have a visual symptom MAP7000 Section-3 Visual Checks
    The customer cannot remotely access the management console MAP6730 Customer access to management console problem

MAP7000 Section-2 Using an SRC

Procedure

  1. Determine the SRC in the serviceable event that sent you to this MAP.
  2. At the rear of each CEC enclosure managed by this HMC, check the input power LEDs of both CEC enclosure power supplies.

    Is at least one input power LED lit for each CEC?

  3. At the rear of each CEC enclosure managed by this HMC, ensure that the private network Ethernet cables are connected.
    • Models 921/931, 922/932, 9A2/9B2, see step 3.a.
    • Models 941, 951, see step 3.b.
    1. For models 921/931, 922/932, and 9A2/9B2 see Figure 1.
      • CEC enclosure service processor (FSP), black cable to P1-C8-T1 and gray cable to P1-C8-T2.
      • CEC enclosure integrated Ethernet ports on the I/O planar for base LPAR, black cable to P1-T6, and gray cable to P1-T7.
      • CEC enclosure Ethernet ports on PCI Ethernet card (only on models 9A2/9B2, with two LPARs per CEC), black cable to P1-C4-T1 and gray cable to P1-C4-T2.
      Figure 1. CEC Ethernet ports (Models 921/931, 922/932, 9A2/9B2)
      CEC Ethernet ports
    2. For Model 941and Model 951 see Figure 2.
      • CEC enclosure service processor (FSP), black cable to P1-C11-T3 and gray cable to P1-C11-T4.
      • CEC enclosure integrated Ethernet ports on the CEC enclosure VPD pass-through card, black cable to P1-C10-T1, and gray cable to P1-C10-T2.
      Figure 2. CEC Ethernet ports (right rear of CEC - Model 941)- Models 941 and 951
      CEC Ethernet ports (right rear of CEC)
  4. At the rear of each Rack-1 managed by this HMC, ensure that the private network Ethernet cables are connected to the Ethernet switches. See Figure 3, Figure 4, or Figure 5.
    Figure 3. 16-port Ethernet switch port designations (SW1, SW2 - Tx) - Models 921/931, 922/932, 9A2/9B2
    Ethernet switch port designations (SW1, SW2 - Tx)
    Figure 4. 8-port Ethernet switch port designations (SW1, SW2-Tx) Models 921/931, 922/932, 9A2/9B2
    8-port Ethernet switch port designations (SW1, SW2-Tx)
    Figure 5. 8-port Ethernet switch port designations (SW1, SW2-Tx) Model 941 and 951
    8-port Ethernet switch port designations (SW1, SW2-Tx)
  5. Not including the serviceable event that sent you here, are there any open serviceable events for the CEC, HMC, or Ethernet switches that are related to the private network?
    • Yes, exit this MAP and repair them now. After you have made the repairs, return to MAP7000 Section-1 to determine if the problem that sent you here has already been corrected.
    • No, continue to the next step.
  6. Is the SRC = BE193001?
    • No, it is most likely safe to delay the repair of any open serviceable event that is not related to the private network. Go to MAP7000 Section-3 Visual Checks.
    • Yes, continue at step 7.
  7. Close all open BE193001 Service events.
  8. Reboot this HMC.
  9. Wait 20 minutes for the HMC to initialize; then log in, and return to this step.
  10. Are there any new open BE193001 Service events?

MAP7000 Section-3 Visual Checks

About this task

Perform the visual checks that are specified in Table 2. If you find problems, take the action that is indicated.

Procedure

Use Table 2 to check each storage facility that is managed by this HMC.
Table 2. Visual checklist
Visual Check Action
If one or both internal Ethernet switches at the rear of Rack-1 in this storage complex are not powered on, see the Action column. Go to MAP2350 Ethernet switch power problem.
If the CEC enclosure power supplies at the rear of each CEC enclosure do not have the green 'Input power' LEDs lit, see the Action column. If a single CEC enclosure is failing:
  1. Check that the CEC enclosure power supply black input cables are connected.
  2. Refer to MAP4970 SRCs that require next level of support or special repair actions, Table 1, for SRC BE1E2551, even if no serviceable events exist with SRC BE1E2551.
  3. If there are no problems, contact your next level of support.
If both CEC enclosures are failing:
  1. Check that the CEC enclosure power supply black input cables are connected.
  2. Check that the UEPO switch is enabled.
  3. Check that customer AC input is available. The primary power supplies should indicate AC Input Good.
  4. If there are no problems, contact your next level of support.
At the front of the rack, observe each CEC enclosure control panel. The normal indication is:

01             N     V=F
HMC=1             P

If an SRC or other status is displayed, see the Action column.		 If the service processor panel is blank, go to MAP4100 Repairing a service processor (FSP) problem.

If an SRC is displayed, go to System reference codes overview.
At the front of the rack, observe the CEC enclosure processor regulator cards which are to the left of the boot drives. If the green LEDs are not lit, see the Action column. Use the management console to power on the storage facility. See MAP1211 Controlling the storage facility power from the management console.
Check the Ethernet switch ports that have cables connected that exit the rack and go to an external HMC or another storage facility. If the port LEDs are not lit properly, see the Action column. Go to MAP7000 Section-4 Problem with an external connection.
Check the Ethernet switch ports that have cables that are connected from the switch to the internal HMC. For both the 16-Port Ethernet switch and 8-Port Ethernet switch, check the T1 ports. If the port LEDs are not lit properly, see the Action column.
  • If Rack-1 has 16-port Ethernet switches, see Figure 3.
  • If Rack-1 has 8-port Ethernet switches, see Figure 4 or Figure 5.
  1. Inspect the Ethernet cables and connectors. Reseat any connection where the Ethernet port LED is not lit properly.
  2. If there is a USB-to-Ethernet adapter on the Black network, and the Ethernet port LED is not lit properly, reseat the USB connector on the internal HMC.
  3. If there is a USB-to-Ethernet adapter on the Gray network, and the Ethernet port LED is not lit properly, reseat the USB connector on the internal HMC.
  4. If the problem is not resolved, contact your next level of support.
None of the visual checks that are listed in this table apply. Go to MAP7001 Using the network topology tool.
Figure 6. Connectivity between storage facilities for HMCs using both 16-port Ethernet switches
Connectivity between storage facilities for HMCs using both 16-port Ethernet switches
Figure 7. 8-port Ethernet switch ports used for cables exiting the storage facility
8-port Ethernet switch ports used for cables exiting the storage facility
Figure 8. Connectivity between storage facilities for HMC and both 8-port Ethernet switches
Connectivity between storage facilities for HMC and both 8-port Ethernet switches

MAP7000 Section-4 Problem with an external connection

Procedure

  1. You are here because you inspected the Ethernet switches and found that one or more link LEDs were not lit (link LEDs correspond to an externally routed Ethernet cable). Use Table 3 to find the condition of the LED(s) and take the appropriate action.
    Table 3. Results of a visual check of the LEDs
    Condition of the LED(s) Action
    LEDs for both links to an external management console are not lit. Go to MAP7010 Private network isolation procedure 10.
    LED for a single link to an external management console is not lit.

    Go to SW1: MAP7011 Private network isolation procedure 11.

    Go to SW2: MAP7012 Private network isolation procedure 12.
    LEDs for both links to another storage facility are not lit. MAP7020 Private network isolation procedure 20
    LEDs for a single link to another storage facility is not lit.

    Go to SW1: MAP7030 Private network isolation procedure 30.

    Go to SW2: MAP7040 Private network isolation procedure 40.
  2. Go to MAP1500 Ending a service action to close the serviceable event and, save the network topology, and ensure good subsystem status.

MAP7000 Section-5 SRC=B3xxxxxx

Procedure

Most storage facility private network problems are reported with SRCs of BEB1xxxx. There are a few exceptions where the problems are reported with SRCs of B3xxxxxx which are from the eServer™ products. When this occurs, use Table 4 to determine the equivalent storage facility BEB1xxxx SRC and/or the appropriate action.
Table 4. Actions for B3xxxxxx SRCs
SRC in serviceable event and definition Equivalent storage facility SRC and definition Action
B3010002 - HMC or partition connection monitoring fault None Go to MAP7000 Section-3 Visual Checks.
B3030001 - The communication link between the service processor and the management console (HMC) has failed
  • BEB10021 (black) - Network Surveillance LINK_FSP_HMC: Single HMC lost link to single FSP over 172.16-BLACK network
  • BEB10022 (gray) Network Surveillance LINK_FSP_HMC: Single HMC lost link to single FSP over 172.17-GRAY network
  • BEB10023 - Not sure which network lost link.
The B3030001 SRC does not specify which private network (black or gray) has failed.
  • Go to MAP7000 Section-3 Visual Checks.
  • If MAP7000 Section-3 Visual Checks does not list a visual symptom, you are instructed to use the Network Topology Tool to determine which network fails.
  • If the black network fails, substitute SRC BEB10021 in place of B3030001
  • If the gray network fails, substitute SRC BEB10022 in place of B3030001
B3030002 - A single partition HMC link has failed.
  • BEB10041 (black) Network Surveillance LINK_PART_HMC_REDUND: Single HMC lost link to single partition on a system, the path through the 172.16-BLACK network is not available, the other network is ok
  • BEB10042 (gray) - Network Surveillance LINK_PART_HMC_REDUND: Single HMC lost link to single partition on a system, the path through the 172.17-GRAY network is not available, the other network is ok
  • BEB10043 - Not sure which network lost link.
The B3030002 SRC does not specify which private network (black or gray) has failed.
  • Go to MAP7000 Section-3 Visual Checks.
  • If MAP7000 Section-3 Visual Checks does not list a visual symptom, you are instructed to use the Network Topology Tool to determine which network fails.
  • If the black network fails, substitute SRC BEB10041 in place of B3030002.
  • If the gray network fails, substitute SRC BEB10042 in place of B3030002.
B3030003 - Multiple partition HMC links have failed. BEB10050 - Network Surveillance LINK_M_PART_HMC: Single HMC lost links to multiple partitions on single system; both paths are not available; FSP to HMC link is still working Go to MAP7000 Section-3 Visual Checks and substitute SRC BEB10050 in place of B3030003.
B3030004 - All partition links for a single system to HMC have failed. BEB10060 - Network Surveillance LINK_A_PART_HMC: Single HMC lost links to all partitions on single system; both paths are not available; FSP to HMC link is still working Go to MAP7000 Section-3 Visual Checks and substitute SRC BEB10060 in place of B3030004.
B3030005 - All system links from one system to a single HMC have failed. BEB10070 - Network Surveillance LINK_ALL_HMC: Single HMC lost links to single system (partition and FSP); both paths are not available Go to MAP7000 Section-3 Visual Checks and substitute SRC BEB10070 in place of B3030005.
B3030006 - The links between a single HMC and a single managed system have failed. BEB10080 - Network Surveillance LINK_HMC_SYS: Single HMC lost links to all systems (FSP and partition); both paths are not available Go to MAP7000 Section-3 Visual Checks and substitute SRC BEB10080 in place of B3030006.
B3030007 - A single HMC link to more than one managed system occurred. BEB10090 - Network Surveillance LINK_HMC_M_SYS: Single HMC lost links to multiple systems (FSP and partition); both paths are not available Go to MAP7000 Section-3 Visual Checks and substitute SRC BEB10090 in place of B3030007.
B3030008 - One HMC link to more than one HMC occurred. BEB10100 - Network Surveillance LINK_HMC_HMC: Lost HMC to HMC links; both paths are not available Go to MAP7000 Section-3 Visual Checks and substitute SRC BEB10100 in place of B3030008.
B3030009 - One HMC link to more than one HMC occurred. BEB10120 - Network Surveillance LINK_HMC_M_HMC: Lost single HMC to HMC link; only one path is not available Go to MAP7000 Section-3 Visual Checks and substitute SRC BEB10120 in place of SRC B3030009.
B303000A - The HMC host links to all managed systems. BEB10130 - Network Surveillance LINK_HMC_ALL: Lost HMC link to multiple HMCs; both paths are not available. This does not apply to the storage facility management console (HMC). Go to MAP7000 Section-3 Visual Checks and substitute SRC BEB10130 in place of B303000A.
B303000B - Multiple HMCs lost links to a single managed system. None N/A
B303000C - Multiple HMCs lost links to multiple managed systems. BEB10150 - Network Surveillance M_LINK_HMC_M_SYS: Multiple HMCs lost all links to multiple systems (FSP and partition) Go to MAP7000 Section-3 Visual Checks substituting SRC BEB10150 in place of B303000C.
B303000D - Multiple HMCs lost links to multiple managed systems. BEB10160 - Network Surveillance M_LINK_HMC_ALL: Multiple HMCs lost all links to all systems (FSP and partition) Go to MAP7000 Section-3 Visual Checks substituting SRC BEB10060 in place of B303000D.
B303000E - The network interface card (NIC) has failed.
  • BEB10011 (black) Network Surveillance NIC_FAILURE: Single HMC physical link unavailable on ethernet port - eth0 (172.16-BLACK network)
  • BEB10012 (gray) Network Surveillance NIC_FAILURE: Single HMC physical link unavailable on ethernet port - eth3 (172.17-GRAY network)
  • BEB10013 - Network Surveillance: Ethernet cables missing or swapped
 The B303000E SRC does not specify which private network (black or gray) has failed.
  • Go to MAP7000 Section-3 Visual Checks.
  • If MAP7000 Section-3 Visual Checks does not list a visual symptom, you are instructed to use the Network Topology Tool to determine which network fails.
  • If the black network fails, substitute SRC BEB10011 in place of B303000E
  • If the gray network fails, substitute SRC BEB10012 in place of B303000E
B303000F - A single partition HMC link failure has occurred on a redundant path.
  • BEB10041 (black) Network Surveillance LINK_PART_HMC_REDUND: Single HMC lost link to single partition on a system, the path through the 172.16-BLACK network is not available, the other network is ok
  • BEB10042 (gray) Network Surveillance LINK_PART_HMC_REDUND: Single HMC lost link to single partition on a system, the path through the 172.17-GRAY network is not available, the other network is ok
  • BEB10043 Not sure which network has lost link.
The B303000F SRC does not specify which private network (black or gray) failed.
  • Go to MAP7000 Section-3 Visual Checks.
  • If MAP7000 Section-3 Visual Checks does not list a visual symptom, you are instructed to use the Network Topology Tool to determine which network fails.
  • If the black network fails, substitute SRC BEB10041 in place of B303000F
  • If the gray network fails, substitute SRC BEB10042 in place of B303000F
B3030010 - A single HMC link to one HMC failure has occurred on a redundant path. BEB10110 - Network Surveillance LINK_HMC_HMC_REDUND Go to MAP7000 Section-3 Visual Checks and substitute the BEB10110 SRC in place of the B3030010 SRC.
B3030011 - The server firmware connection monitoring to the flexible service processor has failed. None Go to MAP7000 Section-3 Visual Checks.
B3030012 - The flexible service processor connection monitoring of server firmware has failed. None Go to MAP7000 Section-3 Visual Checks.
B3030013 - The communication path (mailbox) between the server firmware and the flexible service processor has failed. None Go to MAP7000 Section-3 Visual Checks.
B3100500 - Device Driver Message: mmm dd hh:mm:ss DR-RC02-OPENSYS kernel: e1000: ethN: e1000_watchdog_task: NIC Link is Down
  • BEB10011 (black) Network Surveillance NIC_FAILURE: Single HMC physical link unavailable on ethernet port - eth0 (172.16-BLACK network)
  • BEB10012 (gray) Network Surveillance NIC_FAILURE: Single HMC physical link unavailable on ethernet port - eth3 (172.17-GRAY network)
 The B3100500 SRC does not specify which private network (black or gray) has failed.
  • Go to MAP7000 Section-3 Visual Checks
  • If MAP7000 Section-3 Visual Checks does not list a visual symptom, you are instructed to use the Network Topology Tool to determine which network fails.
  • If the black network fails, substitute SRC BEB10011 in place of B3100500
  • If the gray network fails, substitute SRC BEB10012 in place of B3100500
B3100501 - Device Driver Message: mmm dd hh:mm:ss DR-RC02-OPENSYS kernel: e1000: ethN: unregister 'asix' usb-0000:00:xx.x-x, ASIX AX88772 USB 2.0 Ethernet
  • BEB10014 (black) Network Surveillance NIC_FAILURE: Single HMC physical link unavailable on ethernet port - eth0 on the USB adapter (172.16-BLACK network)
  • BEB10015 (gray) Network Surveillance NIC_FAILURE: Single HMC physical link unavailable on ethernet port - eth3 on the USB adapter (172.17-GRAY network)
The B3100501 SRC does not specify which private network (black or gray) has failed.
  • Go to MAP7000 Section-3 Visual Checks
  • If MAP7000 Section-3 Visual Checks does not list a visual symptom, you are instructed to use the Network Topology Tool to determine which network fails.
  • If the black network fails, substitute SRC BEB10014 in place of B3100501
  • If the gray network fails, substitute SRC BEB10015 in place of B3100501