This section describes how to replace an initiator node that is blocked for some reason during data migration.
-
Required role: Service and Storage
Note on the procedure
-
Run Terraform in an environment that can support the execution.
- Log in to the controller node.
-
Obtain information about the storage cluster to verify the version of the
storage software.
REST API: GET /v1/objects/storage
CLI: storage_show
-
Verify the version of the storage cluster resource.
Run the terraform output command in the terraform working directory. Verify the value of the "TemplateVersion" key.
terraform output
If the storage cluster resource version is the same as the storage software version, go to the next step.
If the versions are different, update the version of the storage cluster resource according to Tasks required after the software is updated in the VSP One SDS Block and SDS Cloud System Administration, and then go to the next step.
-
Verify the ID and status of the initiator node to be replaced.
REST API: GET/v1/objects/storage-nodes
CLI: storage_node_list
Verify that the status of the initiator node to be replaced is "MaintenanceBlockage", "TemporaryBlockage", "PersistentBlockage", "InstallationFailed", "RemovalFailedAndTemporaryBlockage", "RemovalFailedAndMaintenanceBlockage", or "RemovalFailedAndPersistentBlockage", and then go to the next step.
-
See Exporting configuration files (Cloud for Google
Cloud) in the VSP One SDS Block and SDS Cloud System Administration to obtain configuration files
from VSP One SDS Block for deleting initiator nodes during initiator node
replacement.
When exporting configuration files, specify "ReplaceStorageNode" for the exportFileType parameter, and specify the machineImageId parameter and the nodeId parameter.
Obtain the configuration files for replacing initiator nodes to ensure that you use the latest configuration files.
CAUTION:Specify the name of the VM image that matches the storage cluster version for the machineImageId parameter. For the VM image name to specify, refer to VM image name (Cloud for Google Cloud) in the VSP One SDS Block and SDS Cloud System Administration.
-
See Deploying VM configuration files in the Terraform working
directory (Cloud for Google Cloud) in the VSP One SDS Block and SDS Cloud System Administration to unzip the configuration file,
and then deploy the obtained files.
CAUTION:
Make sure you do not edit the unzipped configuration files unless instructed to do so by the manual. If you edit these files, a problem such as storage cluster blockage might occur.
-
Verify the impact of deploying configuration files for deleting the initiator
node.
Run the following command.
terraform plan
When you run the command, the message Terraform will perform the following actions appears, followed by the contents of resource changes.
Verify that the following resource changes are performed.
Resource
Resource manipulation
Resource name
Confirmation details
google_compute_instance.initiator_instance[ ]
delete
<clusterName>-initiatornode
One resource is deleted.
google_compute_instance.instance[ ]
update in-place
<clusterName>-snXX
If a change is made in the following item, there is no problem.
- metadata
- deletion_protection
-
If there is no problem as a result of confirmation by using the terraform plan
command, delete the initiator node to be replaced.
Run the following command.
terraform apply
When [Enter a value:] is displayed after running the command, enter [yes].
If [Apply complete!] is output after running the command, there is no problem.
-
See Exporting Configuration Files (Cloud for Google Cloud) in the
VSP One SDS Block and SDS Cloud System Administration to obtain configuration files
from VSP One SDS Block
for recovering the deleted initiator node during storage node replacement.
When exporting configuration files, specify "ReplaceStorageNode" for the exportFileType parameter.
Also specify nodeId, machineImageId, and recoverSingleNode parameters. Specify the values you specified in step 5 for the nodeId and machineImageId parameters.
The recoverSingleNode parameter should be set to "true."
You need to obtain the configuration files for replacing initiator nodes to ensure that you use the latest configuration files.
-
See Deploying VM configuration files in the Terraform working
directory (Cloud for Google Cloud) in the VSP One SDS Block and SDS Cloud System Administration to unzip the configuration file,
and then deploy the obtained files.
CAUTION:
Make sure you do not edit the unzipped configuration files unless instructed to do so by the manual. If you edit these files, a problem such as storage cluster blockage might occur.
-
Verify the impact of deploying configuration files for adding initiator
nodes.
Run the following command.
terraform plan
When you run the command, the message Terraform will perform the following actions appears, followed by the contents of resource changes.
Verify that the following resource changes are performed.
Resource
Resource manipulation
Resource name
Confirmation details
google_compute_instance.initiator_instance[ ]
create
<clusterName>-initiatornode
One resource is created.
google_compute_instance.instance[ ]
update in-place
<clusterName>-snXX
If a change is made in the following item, there is no problem.
- metadata
-
If there is no problem as a result of confirmation by using the terraform plan
command, recreate the initiator node to be replaced.
Run the following command.
terraform apply
When [Enter a value:] is displayed after running the command, enter [yes].
If [Apply complete!] is output after running the command, there is no problem.
-
Replace the initiator node.
You can perform this only for the cluster master node (primary) or a load balancer.
REST API: POST /v1/objects/storage-nodes/<id>/actions/replace-with-configuration-file/invoke
CLI: storage_node_replace_with_configuration_file
Verify the job ID that is displayed after the command is run.
Note:Additional information in the event log may be uploaded to Cloud Storage when a job completes. If the action in the event log indicates a reference to an "Error message file", retrieve the additional information from Cloud Storage based on the path to the Error Message File in the message.
The Cloud Storage directory to which the information is uploaded is the directory specified for gsUri in the terraform.tfvars file during storage cluster configuration.
For information on gsUri, Verify the key for gsUri in terraform output command.
terraform output
-
Verify the state of the job.
Run either of the following commands with the job ID specified.
REST API: GET /v1/objects/jobs/<jobid>
CLI: job_show
After running the command, if you receive a response indicating "Succeeded" as the state, the job is completed.
Until replacement is completed, it might not be possible to perform other operations. If any other operation is unsuccessful, take action according to the error message or event log for that operation.
Note:- When you perform this procedure, event log KARS16018-W or KARS16143-W might be output. Also, when replacing an initiator node with another one whose drives differ from those used at the time of blocking, event logs KARS05010-E, KARS07005-E, and KARS06596-E might be output for the drives before replacement. Generation of these event logs at this timing is not a problem because the subsequent processing addresses them, and you can continue with this procedure.
- If an error occurs in the storage node replacement job for any reason, the error handling process will power off the initiator node VM. However, the power-off process might not be executed properly due to network failure or other reasons. If the storage node replacement job is terminated because of an error, take action according to the event logs or VSP One SDS Block and SDS Cloud Troubleshooting Reference.