Troubleshooting firmware upgrade timeout issues onHA810 G2 and HA820 G2 servers

Use the following guidelines and suggested steps to resolve firmware upgrade timeout issues on HA810 G2 and HA820 G2 servers.

Condition:

On HA810 G2 and HA820 G2 servers, firmware upgrade times out and does not complete, resulting in the following error:

Unable to upgrade the firmware on server 172.25.92.106 because of upgrade_status (workFlowId: 102) is still NOT in ERROR/COMPLETED state after long waiting time. Check configurations.

What it Means:

The firmware upgrade on the server does not complete because the wait time limit has been reached.

Corrective Action:

Increase the wait time on the UCP Advisor Master Node VM.

1. Log on to the Master Node VM as ucpadmin.
2. Open the infrastructure-operator-config file:

   kubectl edit cm infrastructure-operator-config -n ucp

3. Under serverfirmware.properties, locate the retryNumberToWaitCompleteStateAfterUpgradeIOCard parameter.
   The default values are shown below:

   serverfirmware.properties: |
       retryIntervalToCheckPowerStateAfterSendPowerOffReq=60 # seconds
       retryNumbeToCheckPowerStateAfterSendPowerOffReq=2 # times
       waitTimeAfterSendPowerOnReq=6 # minutes
       retryIntervalToCheckPowerStateAfterSendPowerOnReq=30 # seconds
       retryNumbeToCheckPowerStateAfterSendPowerOnReq=20 # times
       retryIntervalToWaitCompleteStateAfterUpgradeIOCard=3 # minutes
       retryNumberToWaitCompleteStateAfterUpgradeIOCard=20 # times
       waitTimeAfterUpgradeIOCard=5 # minutes

4. Modify the retryNumberToWaitCompleteStateAfterUpgradeIOCard value from 20 times to 25 times. This increases the wait time from 60 minutes to 75 minutes.
5. Save the infrastructure-operator-config file.
6. Retry the firmware upgrade operation.
   Note: No pod restart is required.