Restoring

Virtual Storage Platform One SDS Block Bare Metal Storage Cluster Configuration Restore
Version
1.18.x
Audience
anonymous
Part Number
MK-24VSP1SDS026-01
Note:

  • Even if a failure might occur during the restore operation and you might be asked to obtain a dump, the normal API operations might not be accepted. In that case, obtain a dump file by following the procedure described in Creating a dump file (to be performed only when requested to obtain dump files).

  • Even if the teaming configuration of the control port (for the storage node for which restore is performed) differs before and after restore, there is no problem.

  1. [On the controller node] Verify the storage node on which restore operations are to be performed.

    Verify the SystemConfigurationFile.csv VSSB configuration file, and then select a storage node with the clustermaster role. Then, verify the IP address indicated in ControlNWIPv4.

  2. [On the controller node] Change the password of the built-in user.

    CLI: user_password_set

    If you receive no response after waiting approximately 1 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

    The parameters and values required to change the password are as follows:

    • --ignore_certificate_errors

    • --host <IP-address-obtained-in-step-1>

    • --user_id admin

    (Execution example)

    hsds --ignore_certificate_errors --host <IP-address-obtained-in-step-1> \ 
user_password_set --user_id admin

    After running the command, you are prompted to enter the current password and a new password by using the standard input.

    Enter hsds-admin for "current_password:." Enter any new password for "new_password:."

    Note:

    The password for the admin user which is set in this procedure is only temporarily used during restore. You can specify any password.

    The number and types of characters that can be used are as follows:

    Number of characters: 8 to 256

    Usable character types: Numbers (0 to 9), upper-case alphabet (A to Z), lower-case alphabet (a to z), symbols (! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~)

    During the restore procedure, this password is reset back to the one in the backup data.

  3. [On the controller node] Obtain a list of event logs by running the CLI (event_log_list), and then verify that event log KARS08100-I is output indicating that the storage cluster startup completed.

    If you receive no response after waiting approximately 1 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

    The required parameters are as follows:

    • --ignore_certificate_errors

    • --host <IP-address-obtained-in-step-1>

    • --user admin

    • --severity Info

    (Execution example)

    hsds --ignore_certificate_errors --host <IP-address-obtained-in-step-1> \
--user admin event_log_list --severity Info

    After running the command, you are prompted to enter a password. Enter the new password you set in step 2.

    Note:

    The KARS08100-I event log is output each time the storage cluster is started. Check the output time to verify that the event log is output by performing this procedure.

  4. [On the controller node] Obtain the status of the storage cluster by running the following command: CLI: storage_show. Verify that the following conditions are met.

    • The ID is "00000000-0000-0000-0000-000000000001."

    • The status is "Ready."

    If you receive no response after waiting approximately 1 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

    If the status of the storage cluster does not change to "Ready", contact the Hitachi design team.

    If the status is "Ready" and the ID is different from the preceding one, installation for performing configuration restore is unsuccessful.

    Reperform step 9 of Preparing reinstallation of a storage cluster in Reinstalling the storage cluster. The installation options might not be set properly, and therefore review the operation in step 9.

    The required parameters are as follows:

    • --ignore_certificate_errors

    • --host <IP-address-obtained-in-step-1>

    • --user admin

    (Execution example)

    hsds --ignore_certificate_errors --host <IP-address-obtained-in-step-1> \
--user admin storage_show

    After running the command, you are prompted to enter a password. Enter the new password you set in step 2.

  5. [On the controller node] Perform this step only when replacing an FCHBA.

    Obtain compute port information by using the CLI to verify that the status summary (statusSummary) and the status (status) are "Normal" and the port speed (portSpeed) is other than 0.

    CLI: port_list

    If you receive no response after waiting approximately 1 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact Hitachi's design team.

    The required parameters are as follows:

    • --ignore_certificate_errors

    • --host <IP-address-obtained-in-step-1>

    • --user admin

    • --password <new-password-you-set-in-step 2>

    (Execution example)

    hsds --ignore_certificate_errors --host <IP-address-obtained-in-step-1> --user admin --password <new-password-you-set-in-step-2> port_list
    CAUTION:

    If the specified values are not displayed, the compute port might not be in Link Up status and FCHBAs might not have been replaced properly. Verify the SKU (for more information, see VSP One SDS Block Bare Metal Hardware Compatibility Reference) of the newly installed FCHBA, its installation location, and whether there is any failure.

    Then, destroy the storage cluster once. If there is still a problem with FCHBA replacement, replace the FCHBA again. After that, retry the operation from Installing the storage software (for Hitachi Advanced Server HA800 series, HA800 G2 series, and HA800 G3 series) or Installing the storage software (for physical servers other than Hitachi Advanced Server HA800 series, HA800 G2 series, and HA800 G3 series).

  6. [On the controller node] Restart the storage cluster.

    CLI: storage_shutdown

    If you receive no response after waiting approximately 1 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

    The required parameters are as follows:

    • --ignore_certificate_errors

    • --host <IP-address-obtained-in-step-1>

    • --user admin

    • --reboot true

    • --config_parameter_setting_mode true

    (Execution example)

    hsds --ignore_certificate_errors --host <IP-address-obtained-in-step-1> \
--user admin storage_shutdown --reboot true \
--config_parameter_setting_mode true

    After running the command, you are prompted to enter a password. Enter the new password you set in step 2.

    Verify that both of the following are met to confirm that all storage nodes have been restarted, and then perform the next step.

    • A message indicating the restart (for example, Server reset.) is displayed in the BMC log after you run the command shown in this step.

      For how to view the BMC log, see the document provided by the vendor of the physical server.

    • The Keyboard layout configuration screen is displayed in the console interface window.

    CAUTION:

    Performing this procedure restarts all storage nodes. After the storage nodes are restarted, the VSP One SDS Block Administrator, REST API, and CLI are temporarily unavailable (except for the operations described in this document) until step 12 is complete.

    Note:

    If the config_parameter_setting_mode option is specified, it takes about 20 minutes (a maximum of 90 minutes) for startup of all storage nodes to complete.

  7. [On the controller node] Set the storage cluster to the status in which configuration restore is possible.

    CLI: set_stationary_point

    If you receive no response after waiting approximately 20 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

    The required parameters are as follows:

    • --ignore_certificate_errors

    • (If the controller node is Linux) :

      --auth_ticket `less ticket_for_initialization`

      (If the controller node is Windows) :

      --auth_ticket <Character-string-in-the-ticket-file-(ticket_for_initialization)>

    • --user <Username-for-ticket-authentication>

    • --host <IP-address-obtained-in-step-1>

    • --point UpSOAPI

    • --system_configuration_file <Path-to-the-VSSB-configuration-file-(SystemConfigurationFile.csv)>

    (Execution example for Linux)

    hsds --ignore_certificate_errors --auth_ticket \
`less ticket_for_initialization` --user <username-for-ticket-authentication> \
--host <IP-address-obtained-in-step-1> set_stationary_point \
--point UpSOAPI --system_configuration_file \
<path-to-VSSB-configuration-file-(SystemConfigurationFile.csv)>

    (Execution example for Windows)

    hsds --ignore_certificate_errors --auth_ticket ^
<Character-string-in-the-ticket-file-(ticket_for_initialization)> ^
--user <username-for-ticket-authentication> ^ 
--host <IP-address-obtained-in-step-1> set_stationary_point ^ 
--point UpSOAPI --system_configuration_file ^ 
<path-to-VSSB-configuration-file-(SystemConfigurationFile.csv)>

    You will be prompted to enter the password after running the command. Enter the password for ticket authentication.

    When "Stationary point setting end successfully." is displayed, execution of the CLI is complete. Wait for approximately five minutes until the storage cluster is ready for configuration restore, and then perform the next step.

    CAUTION:

    • While you are using the CLI, if the message "There exists a storage node where the configuration parameter setting mode is disabled" is output, it is probable that step 6 was not performed appropriately or restart was performed without specifying the --config_parameter_setting_mode true parameter in step 6. In either case, reperform the procedure from step 6.

      If the message “There exists a storage node where the configuration parameter setting mode is disabled.” reappears after you reperform the procedure from step 6, the storage cluster might have not been restarted normally. In this case, contact the Hitachi design team.

    • If command execution from the CLI is interrupted due to an operation on the storage node power or a network failure, or by maintenance staff, the storage node might not be able to enter the status in which configuration restoration can be performed. In this case, contact the Hitachi design team.

  8. [On the controller node] Run the restore command.

    CLI: storage_restore_from_configuration_backup_file

    If you receive no response after waiting approximately 15 minutes, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

    The required parameters are as follows:

    • --ignore_certificate_errors

    • (If the controller node is Linux) :

      --auth_ticket `less ticket_for_initialization`

      (If the controller node is Windows) :

      --auth_ticket <Character-string-in-the-ticket-file-(ticket_for_initialization)>

    • --user <Username-for-ticket-authentication>

    • --host <IP-address-obtained-in-step-1>

    • --configuration_backup_file <Path-to-the-configuration-backup-file-in-tar-format>

    (Execution example for Linux)

    hsds --ignore_certificate_errors --auth_ticket \
`less ticket_for_initialization` --user <username-for-ticket-authentication> \
--host <IP-address-obtained-in-step-1> \
storage_restore_from_configuration_backup_file \
--configuration_backup_file <path-to-configuration-backup-file-in-tar-format>

    (Execution example for Windows)

    hsds --ignore_certificate_errors --auth_ticket ^
<Character-string-in-the-ticket-file-(ticket_for_initialization)> ^
--user <username-for-ticket-authentication> ^
--host <IP-address-obtained-in-step-1> ^
storage_restore_from_configuration_backup_file ^
--configuration_backup_file <path-to-configuration-backup-file-in-tar-format>

    You will be prompted to enter the password after running the command. Enter the password for ticket authentication.

  9. [On the controller node] Run the restore status show command to verify whether the storage cluster has been restored.

    CLI: storage_restoration_status_show

    If you receive no response after waiting approximately 1 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

    The required parameters are as follows:

    • --ignore_certificate_errors

    • (If the controller node is Linux) :

      --auth_ticket `less ticket_for_initialization`

      (If the controller node is Windows) :

      --auth_ticket <Character-string-in-the-ticket-file-(ticket_for_initialization)>

    • --user <Username-for-ticket-authentication>

    • --host <IP-address-obtained-in-step-1>

    (Execution example for Linux)

    hsds --ignore_certificate_errors --auth_ticket \
`less ticket_for_initialization` --user <username-for-ticket-authentication> \
--host <IP-address-obtained-in-step-1> storage_restoration_status_show

    (Execution example for Windows)

    hsds --ignore_certificate_errors --auth_ticket ^
<Character-string-in-the-ticket-file-(ticket_for_initialization)> ^
--user <username-for-ticket-authentication> ^
--host <IP-address-obtained-in-step-1> storage_restoration_status_show

    When a message such as shown below is displayed, restore operations are completed.

    (Started Time and Completed Time vary depending on the execution environment.)

    Started Time: 2021-01-01T00:00:00Z
Completed Time: 2021-01-1T23:59:59Z
Status: Completed
Error: None

    You will be prompted to enter the password after running the command. Enter the password for ticket authentication.

  10. [On the controller node] Make an SSH connection to the storage node selected in step 1 of Restoring.

    Perform the subsequent operations of [On the storage node] for the target one node.

    $ ssh -i <passphrase-of-private-key> -o UserKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no \
hitachi_rescue_person@<IP-address-of-storage-node>
    Note:

    The target private key files are the ones that you obtained in step 4 of Collecting information required for restore and that do not include ".pub" in their file names. When you created the key files as shown in the execution example of step 4, key files sample_ed25519 and sample_ed25519.pub exist, and sample_ed25519 serves as a private key file.

  11. [On the storage node] Initialize the internal information. 
    $ sudo /usr/local/bin/hsds-configuration-reset-internal

    After the internal information is initialized successfully, the storage cluster starts automatically.

    When the message “succeeded” appears, log out from the storage node.

    $ exit

    Verify that both of the following are met to confirm that all storage nodes have been restarted, and then perform the next step.

    • A message indicating the restart (for example, Server reset.) is displayed in the BMC log after you run the command shown in this step.

      For how to view the BMC log, see the document provided by the vendor of the physical server.

    • The Keyboard layout configuration screen is displayed in the console interface window.

    CAUTION:

    If error information is output and a restart is not performed, the internal information cannot be initialized. In this case, contact the Hitachi design team.

    It takes about 20 minutes (a maximum of 90 minutes) for startup of all storage nodes to complete.

    Note:

    If you do not log out from the storage node for some time after you run the command for initializing the internal information, the storage cluster restarts automatically and the SSH connection is disconnected. Note that the initialization is successful in this case. Confirm that the storage cluster is restarted, and then perform the next step.

  12. [On the controller node] Change the storage cluster to the maintenance mode and then mapping information clear mode.
    CAUTION:

    The ticket to be specified by --auth_ticket changes from the ticket for reinstallation to the ticket for restore.

    CLI: set_stationary_point

    If you receive no response after waiting approximately 20 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

    The required parameters are as follows:

    • --ignore_certificate_errors

    • (If the controller node is Linux) :

      --auth_ticket `less ticket_for_restore`

      (If the controller node is Windows) :

      --auth_ticket <Character-string-in-the-ticket-file-(ticket_for_restore)>

    • --user <Username-for-ticket-authentication>

    • --host <IP-address-obtained-in-step-1>

    • --point NonstopBoot

    • --maintenance true

    • --initialize_storage_controller true

    • --system_configuration_file <Path-to-the-VSSB-configuration-file-(SystemConfigurationFile.csv)>

    (Execution example for Linux)

    hsds --ignore_certificate_errors --auth_ticket \
`less ticket_for_restore` --user <username-for-ticket-authentication> \
--host <IP-address-obtained-in-step-1> set_stationary_point \
--point NonstopBoot --maintenance true \
--initialize_storage_controller true --system_configuration_file \
<path-to-VSSB-configuration-file-(SystemConfigurationFile.csv)>

    (Execution example for Windows)

    hsds --ignore_certificate_errors --auth_ticket ^
<Character-string-in-the-ticket-file-(ticket_for_restore)> ^
--user <username-for-ticket-authentication> ^
--host <IP-address-obtained-in-step-1> set_stationary_point ^
--point NonstopBoot --maintenance true ^
--initialize_storage_controller true --system_configuration_file ^
<path-to-VSSB-configuration-file-(SystemConfigurationFile.csv)>

    You will be prompted to enter the password after running the command. Enter the password for ticket authentication.

    When "Stationary point setting end successfully." is displayed, execution of the CLI is complete.

    CAUTION:

    If the storage cluster has not restarted normally, the message "There exists a storage node where the configuration parameter setting mode is disabled." might appear after running the CLI.

    If command execution from the CLI is interrupted due to an operation on the storage node power or a network failure, or by maintenance staff, the storage node might not be able to enter the status in which configuration restoration can be performed.

    In either case, contact the Hitachi design team.

  13. [On the controller node] Use the following procedure to verify that the storage cluster is started.
    CAUTION:

    • The password for the admin user is restored to as it was before the backup by the restore operation. Therefore, ask the system administrator for the user to enter the CLI authentication password in the subsequent steps.

    • "Alerting" might be shown for statusSummary. This is temporary. Wait for approximately two minutes, and then run the command again.

    1. Obtain a list of event logs by running the following CLI command: event_log_list. Verify that the following event logs are output.

      If you receive no response after waiting approximately 1 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

      The required parameters are as follows:

      • --ignore_certificate_errors

      • --user <user-with-Security-or-Storage-or-Monitor-or-Service-role>

      • --host <IP-address-obtained-in-step-1>

      • --severity Info

      You will be prompted to enter the password after running the command. Ask the system administrator to enter the password.

      • KARS08109-I: Indicates that the storage cluster was started in mapping information clear mode.

      • KARS08105-I: Indicates that the storage cluster was started in maintenance mode.

      • KARS08100-I: Indicates that startup of the storage cluster was completed.

        CAUTION:

        If event log KARS08100-I (startup of the cluster was completed) is not output after 175 minutes have passed since the storage cluster startup operation, contact the Hitachi design team.

        Note:

        The KARS08100-I event log is output each time the storage cluster is started. Check the output time to verify that the event log is output by performing this procedure.

    2. Obtain the status of the storage cluster by running the following CLI command: storage_show. Verify that the status is "Ready."

      If the status of the storage cluster does not change to "Ready", contact the Hitachi design team.

      If you receive no response after waiting approximately 1 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

      The required parameters are as follows:

      • --ignore_certificate_errors

      • --user <user-with-Security-or-Storage-or-Monitor-or-Service-role>

      • --host <IP-address-obtained-in-step-1>

      You will be prompted to enter the password after running the command. Ask the system administrator to enter the password.

    3. Obtain a list of event logs by running the following CLI command: event_log_list.

      If event logs other than the following logs are output, contact the Hitachi design team.

      • KARS03166-E

      • KARS03167-W

      • KARS05010-E

      • KARS06132-C

      • KARS06201-E

      • KARS06220-C

      • KARS06221-C

      • KARS06222-C

      • KARS07290-C

      • KARS13300-W

      • KARS16023-W

      • KARS16024-W

      • KARS16025-E

      • KARS16119-E

      • KARS16143-W

      • KARS25001-E

      If you receive no response after waiting approximately 1 minute, stop the CLI command forcibly with Ctrl+C, and then retry the operation. If the problem persists, contact the Hitachi design team.

      The required parameters are as follows:

      • --ignore_certificate_errors

      • --user <user-with-Security-or-Storage-or-Monitor-or-Service-role>

      • --host <IP-address-obtained-in-step-1>

      • --severity_ge Warning

      (Execution example)

      hsds --ignore_certificate_errors --user \
<user-with-Security-or-Storage-or-Monitor-or-Service-role> \
--host <IP-address-obtained-in-step-1-in-Restoring> \
event_log_list --severity_ge Warning

      You will be prompted to enter the password after running the command. Ask the system administrator to enter the password.

      If the applicable event logs are output, record all the message contents indicated in the event logs.

      Note:

      Take the following actions according to the output event log:

      • When the event log KARS06201-E, KARS06220-C, KARS06221-C, or KARS06222-C is output:

        Record all the message contents shown in the corresponding event log.

        Perform restore according to Restore workflow, and then verify that the chunk format operation is complete following the procedure of step 3 in Verifying that restore is completed. After that, perform the procedure of Formatting volumes with data reduction enabled (only when event log KARS06201-E, KARS06220-C, KARS06221-C, or KARS06222-C is output).

      • When the event log KARS06132-C is output:

        Record all the message contents shown in the corresponding event log.

        Perform restore according to Restore workflow, and then verify that the chunk format operation is complete following the procedure of step 3 in Verifying that restore is completed. When the event log KARS06201-E, KARS06220-C, KARS06221-C, or KARS06222-C is output, perform the procedure of Formatting volumes with data reduction enabled (only when event log KARS06201-E, KARS06220-C, KARS06221-C, or KARS06222-C is output).

        After that, perform restore according to Restore workflow, and then troubleshoot it as described in Requesting the system administrator to delete snapshot volumes and volume paths (required only when event log KARS06132-C is output).

      • When the event log KARS03166-E, KARS03167-W, KARS05010-E, KARS07290-C, KARS16023-W, KARS16024-W, KARS16025-E, KARS16119-E, KARS16143-W, KARS25001-E is output:

        No action is required.

      • When event log KARS13300-W is output:

        After completing the restore operation according to Restore workflow, request the system administrator to delete the Universal Replicator pair according to the VSP One SDS Block and SDS Cloud Universal Replicator, and then to recreate a Universal Replicator pair as necessary.

      CAUTION:

      If VSP One SDS Block Administrator, REST API, or CLI is not accessible after 30 minutes have passed since the storage cluster was started, the storage cluster is deemed not to have become "Ready."