Cloud storage nodes, Amazon S3, HCP and HCP cloud scale are used to transfer application data to cloud storage. All use the S3 protocol for communication.
Cloud storage nodes represent the storage in Amazon S3 cloud or on the HCP. When creating the node, you will define the necessary credentials to access the S3 service. These include:
- Access Key ID
- Secret Access Key
- Bucket Name
- Region (Amazon S3 only)
Additionally, you will define a Proxy node that will be the intermediary between the application and the S3 service. The proxy node can be any Windows or Linux OS nodes.
The role of the proxy node is to:
- Provide a secure system that can access the S3 service. Although the proxy node can be the same OS host as an application, it is recommended for security reason that this is an independent system to that of the application. The benefit of this is that this system can be given limited access ensuring that the application servers do not have access to the S3 credentials. This S3 node can then be used by multiple applications without the danger of compromising security or data leakage. It also means that the application servers do not need direct access to the internet.
- Provide a cache of the recovery points stored on the S3 server. This cache is only a copy of the index information and metadata, it is not the data itself and can be rebuilt from the S3 content. Having the cache however improves responsiveness of the system and decreases some of the S3 access, thereby saving cost.
- Provide an UBI compliant application interface. This allows you to use S3 storage with any of the next generation applications in the same fashion that you can use Generation 2 Repository. For a list of UBI compliant applications see XYZ. As UBI compliant storage, the S3 node can also receive data from or send data to other UBI compliant storage, such Generation 2 Repository or Generation 2 Hitachi Content Platform. This means it can be used to form a layered data retention solution.
Cloud storage node key features include:
- Search friendly native format
- A key feature of S3 support is to keep data stored on S3 in a native format as much as practicable. Any files less then 4GB in size, has its data stored in a single object. This means that 3rd party search, indexing and analytics tools can process the data in S3 directly. By default, files bigger than 4GB are split into 4GB segments that can easily be stitched together if required.
- Delta block differencing
- To reduce bandwidth and improve transfer speeds, by leveraging S3 partial transfers only the block changes within a file are sent to S3.
- Virtual snapshots
- A system whereby data is retained and indexed in S3 in such a way that recovery points use objects that are whole and not need rehydration but will share data that is common between other recovery points, saving storage space and providing optimal recovery speed.
- Multi-factor parallelism
- To further optimise data transfer speed in addition to delta block differencing, multiple objects are sent to S3 at the same time and multiple data blocks within those objects are sent separately to S3 in parallel.
- Self-consistent store
- Although some metadata is cached on the proxy, the data stored in S3 is self-consistent. Should a site experience complete data loss, a new instance of Protector can be spun and an S3 node can be configured to import the data.