If an instance suffers an unrecoverable failure, you need to replace that instance with a new one.
- In the Admin App, view the Instances page to determine whether the failed instance was a master instance.
- Select a new server or virtual machine to add as a new instance to the system.
- Remove the failed instance from the system.
CAUTION:If the failed instance was a master, after you remove the instance, you have only two master instances remaining. If any other instance fails while you are in this state, the system becomes completely unavailable until you add a third master back to the system by completing this procedure.
- Add the replacement instance to the system.
Important: If the instance you are replacing was a master instance, when you run setup on the replacement instance, the list of masters that you specify for the -m option needs to include:
- The IP addresses of the two remaining healthy master instances.
- The IP address of the new instance that you're adding.
For example, in a system with master instance IPs ranging from 192.0.2.1 to 192.0.2.3 and you are replacing instance 192.0.2.3 with 192.0.2.5, run setup with these options:
sudo bin/setup -i 192.0.2.5 -m 192.0.2.1,192.0.2.2,192.0.2.5
This does not apply when you're replacing a worker instance. In that case, specify the IP addresses of the three existing masters.