CloudBeam for Amazon Redshift - Performance Tuning
Attunity CloudBeam's Replicate-for-Redshift -- Performance Tuning
To optimize data transfer from an on-premises source to Amazon Redshift, one may consider a handful of 'tuning' options.
As documented, Attunity Replicate uses Attunity's proprietary transfer protocol called the 'Large File Accelerator' (LFA). Among other scenarios, the LFA is used when transferring data between source environments and the Amazon S3 bucket (managed by the customer), whereby data is staged and copied into Amazon Redshift.
The LFA protocol allows for a variety of 'tuning' options, including:
- Compression: by default, all of the data is compressed on the source and then decompressed when written to the S3 Bucket. Compressing data can increase CPU consumption, however, it drastically improves transfer speeds.
- Multi-Part Transfers: LFA leverages Multi-Part Upload technology for transferring files to Amazon S3, as described here: http://aws.amazon.com/about-aws/whats-new/2010/11/10/Amazon-S3-Introducing-Multipart-Upload
In addition to the built-in optimization, LFA performance can be further enhanced using the following tuning parameters (which are especially effective when loading very large data sets):
- Edit the Amazon Redshift target in the Attunity Replicate Console and click the Advanced tab.
- Select the Override connection string parameters check box.
- After the last semi colon, add the following string: CIFTAC_CID_FIXED_PART_SIZE=10;CIFTAC_CID_UPLOAD_STREAMS=10
The two parameters are used for the following settings:
- First Parameter: CIFTAC_CID_FIXED_PART_SIZE=10, sets the size of the parts being uploaded to 10 MB.
- Second Parameter: CIFTAC_CID_UPLOAD_STREAMS=10, sets the number of concurrent TCP sessions or sockets to 10.