Recovering storage nodes (and preserving storage volumes)

To recover storage nodes when preserving storage volumes:

From the HCP Configuration Menu, enter s to display the HCP Service Menu.

In response to the confirmation prompt, enter y or yes to confirm your entry or n or no to try again.

When you enter y or yes, the HCP Service Menu appears.

HCP Service Menu
=============================================

[1] Recover an HCP Node
[2] Display the Current System Status
[3] Shut down the HCP System
[4] Free Space on Nodes
[5] Install a Hotfix
[6] Finalize Migration
[7] Change Region Count
[8] Configure Routine Database Maintenance
[9] Configure Time Settings
[10] Migrate Metadata Regions
[11] Perform Node Swap
[s] Perform SSD Maintenance
[m] Manage Database Volumes

[q] Return to Configuration Menu

Enter your choice.
[Default: 1]: m

From the HCP Service Menu, enter 1 for recovery.

In response to the confirmation prompt, enter y or yes to confirm your entry or n or no to try again.

When you enter y or yes, HCP Setup displays the Node Recovery Menu.

HCP Setup: Node Recovery Menu
=============================================

[1] Recover storage nodes
[2] Recover all storage nodes
[3] Reinitialize internal database

[b] Go Back to the Previous Menu
[q] Return to the Configuration Menu

Enter your choice.
[Default: 1]:

From the Node Recovery Menu, take one of these actions:
- To recover selected storage nodes, enter 1. Then follow the on-screen instructions to identify the nodes you want to recover. Make sure to use the back-end IP address to identify each node.
  Note: If you choose to use a range of IP addresses to identify the nodes, make sure the range you specify includes only the nodes you want to recover.
- If you identify fewer than half of the nodes in the HCP system, HCP Setup asks whether you want to delete and try to rebuild the database on those nodes.
```
 Do you want to delete the database and have HCP try to rebuild it?
 Enter yes only if you know that the database is unrecoverable. If you
are unsure, enter no.
 [Default: no]:
```
  In response:
  1. Enter y or yes to delete the database while recovering the OS or n or no to recover the OS without deleting the database.
  2. In response to the confirmation prompt, enter y or yes to confirm your entry or n or no to try again.
  3. Optionally, if you are performing the OS recovery on an HCP G10 Node with attached storage or an HCP system with dedicated database volumes, HCP Setup displays the following prompt.
```
Do you want HCP to format the internal database drive before rebuilding it?
If you enter yes, only the internal database drive is formatted.
Enter yes only if you know that the internal database drive needs formatting.
If you are unsure, enter no.
[Default: no]:
```
    If you receive this prompt, enter y or yes to format only the internal database drive, or enter n or no to keep the internal database drive in its original state.
  4. HCP Setup displays a unique key and prompts you to enter it back.
    If you identify half or more of the nodes in the HCP system, HCP Setup displays a unique key and prompts you to enter it back.
    
    To recover all storage nodes, enter 2.
Enter the unique key exactly as it is shown.
HCP Setup performs a series of prechecks and, if they are successful, recovers the OS on the selected nodes or all nodes, as applicable.

If any of the prechecks fail, HCP Setup exits. In this case, fix the problem and then start the OS recovery procedure again.

When the node recovery is complete, HCP restarts all the nodes that it recovered and displays this message:
```
>>> HCP Service Procedure successful Press ENTER to continue:
```
If the node that you are logged in to is one of the recovered nodes, the SSH or console session ends automatically when HCP restarts the node.
If this is not the case, in response to the prompt to continue, press Enter.

The HCP Service Menu appears.

Important: If HCP Setup exits at any time before the OS recovery processing is complete, contact your HCP support center. Do not try to recover the OS again.
From the HCP Service Menu, enter q to return to the HCP Configuration Menu.
In response to the confirmation prompt, enter y or yes to confirm your entry or n or no to try again.
From the HCP Configuration Menu, enter q to log out of the install shell.
In response to the confirmation prompt, enter y or yes to confirm your entry or n or no to try again.