Many UK organisations that I talk to in both the public and private sectors have disparate data systems spread across the company and feel that they’ve lost control over which department has what data, and where the data resides. This should change for everyone, across sectors. The challenge that organisations are facing is how to best manage their data on limited resources and diminishing budgets.
If we look back at why so many versions of data have sprung up within organisations, we find it is typically because line of business users haven’t been able to get the data they want when they want it in a format they need. When data is requested, the IT team – often overworked and under-resourced – haven’t been able to meet the requested due date. So, what happens? The end users find a back door. For example, an individual might need a set of reports for legislative reasons and to meet the SLA, they figure out a way to produce them without the IT department. This common scenario is an example of why “Shadow IT” sprang up.
Some software vendors have taken advantage of end user behaviour and instead of selling to the IT department, they go straight to the business units where they have sold a few licenses here and there. It seemed a good idea at the time, but many IT and data teams complain that it’s their worst nightmare because they can’t control who has access to what data across the disparate data silos.
I also see organisations using the good old spreadsheet as their business intelligence (BI) tool. Why? Because it is accessible and easy to use, and staff can simply cut and paste, import or manually copy data over time. The problem is that spreadsheets using ungoverned data exist everywhere and they are a challenge for those responsible for corporate governance. Each of the approaches highlight the challenges when data is no longer controlled, and multiple versions of the “truth” are created.
Looming Legislation Means Organisations Can No Longer Duck Out of Data Management
There is no one single solution, but with the GDPR looming over the horizon – with only 9 months to go – departments and authorities are being forced to get their ducks in a row. Until recently, GDPR probably hasn’t had the attention it should have received. In fact, it wasn’t initially apparent how far-reaching it would be. Of course, it must be added to the list of regulation changes that are continually happening, not least planning for Brexit, which of course has an impact on almost every UK department and authority.
One approach is to consolidate all data repositories The concept of a master data repository is not particularly new but has always been difficult to achieve. With new technologies, this is getting easier. Software solutions like Attunity Replicate, available through G-Cloud 9, fits the bill for many organisations and can perform data replication from most databases such as SQL and Oracle as well as legacy platforms that have been around for decades. Attunity Replicate is unlike most of the newer solutions in that it does not require expert ETL or programming knowledge and can facilitate agile change. This is a key pain in the public sector. Coupled with a cloud-based target such as AWS, UKCloud or Azure, data can be quickly consolidated and organisations can better manage their data while achieving compliance.
In addition, organisations then need to find a way of delivering data in a timely way that is compliant and doesn’t fall foul of any regulations. If that isn’t remedied, then the same situation will happen again. Data can’t just be dumped in one place and not managed; it needs to be prepared and, in a controlled manner, delivered back to users in a timely fashion. There should be an agile approach to deliver the changes back to the business users on demand. Data warehouse automation solutions such as Attunity Compose can help organisations discover a data model from the data sources, build out the data warehouse and provide the facility to quickly deliver data marts to the business. One of Attunity’s customers stated that by using Attunity Compose they had reduced their ETL development from 45 days down to 2 days and were able to deliver their first data warehouse release in just 12 weeks.
Will the Public Sector get their Ducks in a Row in time?
Is the above just a pipe dream? No. Attunity is working with several organisations that are implementing end-to-end solutions quickly and easily. One client has many hundreds of databases, most of which are a slightly different version or format than the main source data. They are using Attunity Replicate to consolidate all the databases into the enterprise data warehouse (EDW) which is managed by Attunity Compose, meaning that they will have just a handful of data sources in the future. Any data requests will be fulfilled from the master EDW source. Their timelines are very aggressive with the entire project, including the consolidation, which they aim to complete within 18 months. If this is happening in the private sector, then there is no reason for UK Public Sector bodies not to replicate this approach.
The benefits of this approach including Attunity Replicate:
- Solutions available through G-Cloud9 have a much easier procurement process
- Cloud offers a lower cost and flexible infrastructure
- Organizations have more control over data and who is accessing it
- IT has less systems to maintain – offering a reduction of operating costs
- IT can respond to business demands much more quickly
- There is less reliance on specialist staff such as ETL developers – who can focus on higher value tasks for the business
Do you have all your data integration and management ducks in a row to meet compliance requirements and SLAs? Contact Attunity. We can help.