Attunity CloudBeam for Amazon S3 - Glacier Archiving
Challenge: Extract, Load, and Transform Data -- Amazon Redshift
While Attunity CloudBeam does not provide a means of directly loading files to Amazon Glacier, customers have the following option:
- Use 'Attunity CloudBeam for Amazon S3' to load files to Amazon S3.
- Within Amazon, apply an 'Object Lifecycle Configuration', archiving files from Amazon S3 to Amazon Glacier.
- Use 'Attunity CloudBeam for Amazon S3' to manage on-going replication of 'only newly added' source files to Amazon S3 (which can later be archived to Amazon Glacier).
Object Lifecycle Configurations
For customers which maintain Amazon S3 Buckets, Amazon Web Services provides an option to archive files to Amazon Glacier.
The procedure is known as 'Object Lifecycle Configurations', outlined here: http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html
For this 'Best Practice' example, let's say that a customer applies a policy that "any file introduced to the Amazon S3 Bucket will be moved to Amazon Glacier after 30 days".
Attunity CloudBeam for Amazon S3 Configure Corresponding Replication Job Properties
'Attunity CloudBeam for Amazon S3', featuring 'Attunity RepliWeb' software is designed to 'mirror' a good-known source to the Amazon S3 target Bucket.
So if a customer has an 'archiving policy' of "30 days" (example above), then 'Attunity RepliWeb', by default, would replicate any files that exist on the source, but no longer on the target (as they may have been archived). This could create a vicious cycle of unncessary, duplication of files.
How does one overcome the combination of 'replication' with a target that has an 'archiving policy'?
1. Within an upload job's properties, select the 'General' tab > Synchronization Options.
2. Select 'Transfer only files modified during the last' option and select a time value aligned with the 'archiving policy' within Amazon.
3. By way of on-going replication, 'Attunity CloudBeam for Amazon S3' will transfer only files that are new to the Amazon S3 Bucket and not yet archived.