Nodegrid Backup to S3 Buckets
  • 13 Apr 2023
  • 4 Minutes to read
  • Dark
    Light
  • PDF

Nodegrid Backup to S3 Buckets

  • Dark
    Light
  • PDF

Article summary

Nodegrid Backup to S3 Buckets

S3 buckets or Objects stores are a common way to store and retrieve files or other objects.
Originally S3 buckets was only available through cloud providers like AWS S3. This is now available through commercial Backup and Storage solutions like NetApp StorageGRID.

This guide explains how to use the Nodegrids Ansible examples to regularly backup files to an S3 bucket.

Overview

The following steps are required to enable stprage of Nodegrid backups in an S3 bucket.

  1. Installation of Ansible Examples on a Nodegrid appliance with access to systems that backups are sourced.
  2. Installation of rclone on each system.
Alternative

This is not documented in the guide: Files are copied to the Ansible Controller and uploaded from there to the S3 bucket - which assumes backup files are uploaded from each Nodegrid appliance to the storage.

  1. Run the first backup

This guide requires the following:

ItemDetailsComments
NodegridNodegrid Version 5.8.3 or higherOlder versions have not been tested.
rclonefrom link
Access KeyS3 Access Key ID
Access SecretS3 Access Secet
EndPointOnly required for private deployments with custom endpoints.

Installation

1. Install Ansible Examples

For a detailed description to install the Ansible examples, review the article Installation of Nodegrid Ansible libraries on Nodegrid OS

2. Install rclone

  1. Download the Linux version of rclone (rclone-current-linux-amd64.zip) from the official web page
    or directly from this link
  2. Upload the zip file to '/var/local/file_manager/admin_group'. The file can be uploaded using scp or on the WebUI through the File Manager.
  3. Extract the zip file from the shell (ansible user role) . Open a consle connection (admin role). Switch to ansible user role to unzip the file
shell sudo su - ansible
unzip /var/local/file_manager/admin_group/rclone-current-linux-amd64.zip
  1. (ansible user role) To configure S3 connection, use the following command to start the Wizard.
~/rclone-v1.61.1-linux-amd64/rclone config
  1. Configuration Option Examples:
    5.1 AWS S3
Setting
Option
Comment
No remotes found, make a new one?nCreate a new connection
Enter name for new remote.s3Use s3 as the local name (to identify only the connection, not the bucket )
Type of storage to configure.5Select Amazon S3 Compliant Storage Providers
Choose your S3 provider1Select Amazon Web Services (AWS) S3
Get AWS credentials from runtime1Provide AWS details in the next step
AWS Access Key ID{{Access Key}}Enter the AWS S3 Access Key
AWS Secret Access Key (password){{ Secret }}Enter the AWS S3 secret
Region to connect to1This defines the Region which defines the location of the S3 bucket.
Endpoint for S3 APIN/AEnter a custom Endpoint; this is not required
Location constraint - must be set to match the Region1This defines a region constraint and must match the location of the S3 bucket
Permissions (ACL)1In most cases, the default values a good (1). Provides the creator full access but sets the ACLs to private
Option server_side_encryption1Define a Server site Encryption as required
Option sse_kms_key_id1KMS ID must be provided when KMS used for server-side encryption
Option storage_class1Define a default Storage Class, leave at the default value (1)
Edit advanced config?NSelect No
Configuration complete.ySelect Yes to store the configuration
QuiteqSelect Quite to exit the wizard

5.2 NetAPP StorageGRID

SettingOptionComment
No remotes found, make a new one?nCreate a new connection
Enter name for new remote.s3Use s3 as the name; this is a local name to identify the connection only and not the bucket
Type of storage to configure.5Select Amazon S3 Compliant Storage Providers
Choose your S3 provider25Select Any other S3 compatible provider
Get AWS credentials from runtime1Provide AWS details in the next step
AWS Access Key ID{{Access Key}}Enter the AWS S3 Access Key
AWS Secret Access Key (password){{ Secret }}Enter the AWS S3 secret
Region to connect to1This defines the Region which defines the location of the S3 bucket
Endpoint for S3 API{{ Endpoint }}Enter the NetApp StorageGRID custom Endpoint
Location constraint - must be set to match the RegionN/A
Permissions (ACL)1In most cases, the default value is good (1). Provides the creator full access but sets the ACLs to private
Edit advanced config?NSelect No
Configuration complete.ySelect Yes to store the configuration
QuitqSelect Quit to exit the wizard

3. Run the first backup

  1. (ansible user role) Test the connection to the s3 bucket with nodegrid-backup to represent the S3 bucket name.
~/rclone-v1.61.1-linux-amd64/rclone ls s3:nodegrid-backup

(This command does not list all existing files at the defined S3 bucket.)

SSL Certificate check error

If the S3 storage uses a Self Signed Certificate, the CA must be locally installed. Or, the certificate checks are disabled with the flag --no-check-certificate
For example
~/rclone-v1.61.1-linux-amd64/rclone ls s3:nodegrid-backup --no-check-certificate

  1. Run the first backup with the existing ansible example
ansible-playbook /etc/ansible/playbooks/nodegrid/ng_backup_to_s3.yml -l localhost

This alternative uses --no-check-certificate

ansible-playbook /etc/ansible/playbooks/nodegrid/ng_backup_to_s3.yml -l localhost -e cert_check=--no-check-certificate
  1. Confirm the backup file was created.
~/rclone-v1.61.1-linux-amd64/rclone ls s3:nodegrid-backup

4.: Optional: The playbook can be edited to be adjusted and expanded.

4. Schedule Backup to S3 bucket

Existing playbooks can be efficiently executed and scheduled from the UI through the Central Management feature. To run the backup regularly, schedule the playbook execution through the WebUI.

  1. To enable the Central Management feature, see the Nodegrid User Guide.
  2. Navigate to System :: Central Management
    image.png
  3. Select localhost and click Run.
  4. On the page, provide the following details:
    image.png
SettingValueComment
Playbookng_backup_to_s3.yml
Variablescert_check=--no-check-certificateleave blank if not needed
TypeSchedule
Task NameBackup_to_S3
Minute0
Hour1
Day of Month*
Month*
Day of Week*
  1. Click Save.

Was this article helpful?

ESC

Eddy AI, facilitating knowledge discovery through conversational intelligence