How the UK public sector can benefit from robotic data migration

In the second edition of Opinion, NDL CEO Declan Grogan explores the complexity and roadblocks faced throughout mass data migration projects in the public sector. From avoiding costly exit migration charges, to the most efficient and accurate forms of extraction, Declan explains Robotic Process Automation’s (RPA) valuable role in data transferal below.

The great data lock-in

The ability to control personal data has become a matter of principle, as well as a legal right. Currently, Article 20 of the GDPR Act requires certain vendors to provide their users with the autonomy to migrate and transfer their data between suppliers – such as a built-in method of transferring a playlist between modern music streaming providers.

Unfortunately, those same rights don’t seem to be offered to public organisations in respect of the data they keep on the public’s behalf - which they need to function. Current contractual arrangements with many Document Management or CRM vendors do not seem to cover mass data migration, nor do they seem to have data migration methods or utilities built into their systems - and we are not aware of any legislation to cover this gap.

So, what should an organisation do if they want to move their data to a newer, more suitable system?

Providing an API (or equivalent tool), which allows customers to warehouse or migrate all their data to a new platform, often isn’t in the vendors’ financial best interest. Little surprise, then, that this functionality has been historically unavailable, with some vendors offering it as a last gasp, ‘cash-cow’ of a service - charging six and even seven figure sums to access data that is, after all, already the property of the customer.

Now we’re all wiser, modern software or cloud services often include migration and exit functions as part of standard contracts. However, many systems currently in use were licenced decades ago, and contain no such provisions. Due to this, many public sector organisations can find themselves locked into systems that no longer serve their best interests – nor the general population’s.

In our view, this quandary stifles innovation - preventing the adoption of newer, more suitable systems for use by council, housing, policing and healthcare workers. For publicly funded organisations, the costs involved with vendor-delivered or manually executed data migration are unattainable. Due to this, many organisations simply cannot afford to provide their teams with the technology best suited to their duties.

How can RPA assist data migration?

Given the above scenario, what choices do we have? Do nothing due to migration costs, foregoing the benefits of a new system? Do we ‘pay-up’ for the managed service, or enlist an API that hopefully exists to perform the migration and go it alone?

An alternative solution might be to attempt direct, independent extraction from the vendor supplied tables in the database. This is often possible, however does carry a significant risk due to the complex way data relates to its structure. Unless the person carrying out this operation is highly experienced and has specialised knowledge, then it’s quite possible for data elements to become orphaned - losing their context and therefore unable to be imported into the new system.

There remains, of course, the option to attempt to manually extract the data. In some cases, assuming 100% efficiency over a 37.5-hour working week, this could take hundreds of years for a single person to complete. But what if we used a bot, or rather a whole workforce of bots, to perform this error-prone, highly repetitive task?

Speed, resources, and accuracy

If we examine the task to be performed, what we have is a well-structured, highly repetitive, rules-based task - the very thing software robotics excels at. We can simply teach the bot to use the system, give it rules to follow and set it away. Unlike a person, the bot can work as fast as the target system is capable of - realistically much, much faster than a person. Bots are also 100% accurate, so data quality is not affected by fatigue or distraction - the bot can run without a break for days, weeks or months without stopping.

And, while we can appreciate the efficiency, quality, and operational effectiveness gained by using a bot over a human worker for data migration - that’s only a one-to-one comparator. Where software robotics comes into its own in this scenario, is scale. By simply copying and pasting the now trained bot, we instantly double capacity. We can build a team, and clone that. Within minutes, we can create a whole team of digital workers, extracting and moving data 24/7, at a pace that cannot be matched.

Fostering better data practices

In any legacy dataset, there are bound to be inconsistencies that require identification and resolution. When migrating between systems, the metadata you wish to accompany the main data payload can be just as important.

It may also be that there is a need to transform the data type or to perform ‘in-flight’ operations. Indeed, it may simply be that you wish to leave some data behind, so as not to clutter up the new system with irrelevant information - or in the instance of cloud solutions, to minimise ongoing charges. Assuming these operations conform to well-understood and structured rules, then the bots can apply them as part of the migration process too.

Focussing on the tasks that matter

Over the last few years, the NDL Community has used our RPA platform to migrate hundreds of millions of records. These migrations have been delivered by customers themselves, using their existing corporate RPA platform licence, or by enlisting and collaborating with the NDL Delivery Team to do the work for them.  In doing this, their teams have achieved much more with the time that they have, while fostering better processes for themselves going forward – not to mention the costs and overheads saved as a result.

For the NDL team this is, by now, a well-trodden path. Celebrating 40 years in the industry, their understanding of scalability, management control and auditing - along with their practical understanding of public sector systems and how best to drive them - is second to none. To read more about robotic data migration in action, don’t forget to take a look at our latest whitepaper release, Transforming Data Migration with RPA – as well as our library of customer Success Stories. For more information around any of our services, don’t hesitate to get in touch.