New Year Reflections: Pipelines, Platforms and Clouds


As one year winds down and the next looms, it’s always useful to take stock of what’s happening in the market and amongst our customers. In that spirit, here are some observations about the state of the data integration business. It has been a busy 12 months, loaded with increasingly ambitious and strategic data and analytics initiatives. Attunity is privileged to help some of the world’s largest enterprises reinvent how they identify, measure and capitalize on opportunities.

Here is my take on three top-of-mind priorities as we enter the New Year.

  • Rising focus on data pipelines. “Success” has shifted from loading the data lake to actually preparing its data for analytics. Many enterprises have used Attunity solutions for years to capture real-time production transactional data for streaming ingestion into their data lake platform of choice. Now they also use our software to merge those multi-sourced change streams into a comprehensive change history, then provision data subsets from that such as point-in-time snapshot views or a real-time operational data store. Many customers are siphoning these data subsets into data warehouses for structured operational reporting, using our automation software to speed and streamline the process.
  • In short, data architects are building and managing multi-zone pipelines to enable the real work of analytics. They are focused on the end game: insights for the line of business.
  • Many platforms. One data integration process. We recently signed an agreement with a major energy firm that is applying Attunity replication and transformation solutions to the following end points:
  • Sources: Oracle; SAP HANA; SQL Server
  • Targets: Actian Vector; AWS Aurora MySQL; AWS Aurora PostgreSQL; AWS EMR; AWS Kinesis; AWS RDS MariaDB; AWS RDS MySQL; AWS RDS Oracle; AWS RDS PostgreSQL; AWS RDS SQL Server; AWS Redshift; AWS S3; Azure ADLS; Azure Event Hubs; Azure HDInsight; Azure SQL DW; Cloudera; CSV File; DB2 LUW (RDBMS); DB2 zOS; Google Cloud SQL; Hortonworks; HP Vertica; HWX NiFi; IBM Netezza; Informix; Kafka; MapR; MapR Streams; Microsoft PDW; MySQL; ODBC; Oracle; Pivotal (Greenplum); PostgreSQL; SAP HANA; SAP Sybase IQ; Snowflake; SQL Server; Sybase ASE; Teradata.
  • When you need to balance data across an environment with this level of complexity, you cannot afford to have your replication process vary among them.
  • This firm selected Attunity in part because they can manage data flows across all these platforms with a single 100% automated process and a single graphical console. More and more companies are taking this approach, seeking to optimize analytics workloads on multiple specialized platforms. As they do so, they need to keep the data flow process simple.
  • Cloud adoption is so 2017. Today you may want more than one cloud. As you might have noticed in the prior example, our energy firm customer is copying its production data to both AWS and Azure targets. They are in good company. I see many enterprises considering all their public-cloud options as they seek to match the right platform with the right analytics job. These hybrid, multi-cloud environments include all three major players (and Attunity partners) – AWS, Azure and Google – and often involve data flows between clouds.

I am looking forward to an exciting 2019, full of new developments and strategic initiatives. If you are interested in learning more about our data lake pipeline offerings, check out this whitepaper: Increase Data Lake ROI with Streaming Data Pipelines.

Dev Tool:

Request: blog/new-year-reflections-pipelines-platforms-and-clouds
Matched Rewrite Rule: blog/([^/]+)(?:/([0-9]+))?/?$
Matched Rewrite Query: name=new-year-reflections-pipelines-platforms-and-clouds&page=
Loaded Template: single.php