Nodegrid Backup to S3 Buckets
S3 buckets or Objects stores are a common way to store and retrieve files or other objects.
Originally S3 buckets was only available through cloud providers like AWS S3. This is now available through commercial Backup and Storage solutions like NetApp StorageGRID.
This guide explains how to use the Nodegrids Ansible examples to regularly backup files to an S3 bucket.
Overview
The following steps are required to enable stprage of Nodegrid backups in an S3 bucket.
- Installation of Ansible Examples on a Nodegrid appliance with access to systems that backups are sourced.
- Installation of rclone on each system.
This is not documented in the guide: Files are copied to the Ansible Controller and uploaded from there to the S3 bucket - which assumes backup files are uploaded from each Nodegrid appliance to the storage.
- Run the first backup
This guide requires the following:
| Item | Details | Comments |
|---|---|---|
| Nodegrid | Nodegrid Version 5.8.3 or higher | Older versions have not been tested. |
| rclone | from link | |
| Access Key | S3 Access Key ID | |
| Access Secret | S3 Access Secet | |
| EndPoint | Only required for private deployments with custom endpoints. |
Installation
1. Install Ansible Examples
For a detailed description to install the Ansible examples, review the article Installation of Nodegrid Ansible libraries on Nodegrid OS
2. Install rclone
- Download the Linux version of rclone (rclone-current-linux-amd64.zip) from the official web page
or directly from this link - Upload the zip file to '/var/local/file_manager/admin_group'. The file can be uploaded using scp or on the WebUI through the File Manager.
- Extract the zip file from the shell (ansible user role) . Open a consle connection (admin role). Switch to ansible user role to unzip the file
shell sudo su - ansible
unzip /var/local/file_manager/admin_group/rclone-current-linux-amd64.zip
- (ansible user role) To configure S3 connection, use the following command to start the Wizard.
~/rclone-v1.61.1-linux-amd64/rclone config
- Configuration Option Examples:
5.1 AWS S3
| Setting | Option |
Comment |
|---|---|---|
| No remotes found, make a new one? | n | Create a new connection |
| Enter name for new remote. | s3 | Use s3 as the local name (to identify only the connection, not the bucket ) |
| Type of storage to configure. | 5 | Select Amazon S3 Compliant Storage Providers |
| Choose your S3 provider | 1 | Select Amazon Web Services (AWS) S3 |
| Get AWS credentials from runtime | 1 | Provide AWS details in the next step |
| AWS Access Key ID | {{Access Key}} | Enter the AWS S3 Access Key |
| AWS Secret Access Key (password) | {{ Secret }} | Enter the AWS S3 secret |
| Region to connect to | 1 | This defines the Region which defines the location of the S3 bucket. |
| Endpoint for S3 API | N/A | Enter a custom Endpoint; this is not required |
| Location constraint - must be set to match the Region | 1 | This defines a region constraint and must match the location of the S3 bucket |
| Permissions (ACL) | 1 | In most cases, the default values a good (1). Provides the creator full access but sets the ACLs to private |
| Option server_side_encryption | 1 | Define a Server site Encryption as required |
| Option sse_kms_key_id | 1 | KMS ID must be provided when KMS used for server-side encryption |
| Option storage_class | 1 | Define a default Storage Class, leave at the default value (1) |
| Edit advanced config? | N | Select No |
| Configuration complete. | y | Select Yes to store the configuration |
| Quite | q | Select Quite to exit the wizard |
5.2 NetAPP StorageGRID
| Setting | Option | Comment |
|---|---|---|
| No remotes found, make a new one? | n | Create a new connection |
| Enter name for new remote. | s3 | Use s3 as the name; this is a local name to identify the connection only and not the bucket |
| Type of storage to configure. | 5 | Select Amazon S3 Compliant Storage Providers |
| Choose your S3 provider | 25 | Select Any other S3 compatible provider |
| Get AWS credentials from runtime | 1 | Provide AWS details in the next step |
| AWS Access Key ID | {{Access Key}} | Enter the AWS S3 Access Key |
| AWS Secret Access Key (password) | {{ Secret }} | Enter the AWS S3 secret |
| Region to connect to | 1 | This defines the Region which defines the location of the S3 bucket |
| Endpoint for S3 API | {{ Endpoint }} | Enter the NetApp StorageGRID custom Endpoint |
| Location constraint - must be set to match the Region | N/A | |
| Permissions (ACL) | 1 | In most cases, the default value is good (1). Provides the creator full access but sets the ACLs to private |
| Edit advanced config? | N | Select No |
| Configuration complete. | y | Select Yes to store the configuration |
| Quit | q | Select Quit to exit the wizard |
3. Run the first backup
- (ansible user role) Test the connection to the s3 bucket with nodegrid-backup to represent the S3 bucket name.
~/rclone-v1.61.1-linux-amd64/rclone ls s3:nodegrid-backup
(This command does not list all existing files at the defined S3 bucket.)
If the S3 storage uses a Self Signed Certificate, the CA must be locally installed. Or, the certificate checks are disabled with the flag --no-check-certificate
For example
~/rclone-v1.61.1-linux-amd64/rclone ls s3:nodegrid-backup --no-check-certificate
- Run the first backup with the existing ansible example
ansible-playbook /etc/ansible/playbooks/nodegrid/ng_backup_to_s3.yml -l localhost
This alternative uses --no-check-certificate
ansible-playbook /etc/ansible/playbooks/nodegrid/ng_backup_to_s3.yml -l localhost -e cert_check=--no-check-certificate
- Confirm the backup file was created.
~/rclone-v1.61.1-linux-amd64/rclone ls s3:nodegrid-backup
4.: Optional: The playbook can be edited to be adjusted and expanded.
4. Schedule Backup to S3 bucket
Existing playbooks can be efficiently executed and scheduled from the UI through the Central Management feature. To run the backup regularly, schedule the playbook execution through the WebUI.
- To enable the Central Management feature, see the Nodegrid User Guide.
- Navigate to System :: Central Management
- Select localhost and click Run.
- On the page, provide the following details:
| Setting | Value | Comment |
|---|---|---|
| Playbook | ng_backup_to_s3.yml | |
| Variables | cert_check=--no-check-certificate | leave blank if not needed |
| Type | Schedule | |
| Task Name | Backup_to_S3 | |
| Minute | 0 | |
| Hour | 1 | |
| Day of Month | * | |
| Month | * | |
| Day of Week | * |
- Click Save.