Data Migration as a Service
Adoption of Cloud Computing continues to rapidly increase as companies continue to recognise its importance as a driver for digital transformation. It is clear companies that are successful in adopting the cloud bring new capabilities to market more quickly, scale more efficiently and demonstrate greater business agility. However, the journey to the cloud is not an easy one and many companies have stalled along the way.
Big data is what drives most modern businesses, and big data never sleeps. That means data integration and data migration need to be well-established, seamless processes – whether data is migrating from inputs to a data lake, from one repository to another, from a data warehouse to a data mart, or in or through the cloud, must be established. Without a competent data migration strategy, data migration programmes can run over budget, end up with overwhelming data processes, or find that their data operations are functioning below expectations, this leads to the big question, what is Data Migration?
Data migration is the process of moving data from one system to another. While this might seem straightforward, it is not. Data migration can involve complex storage, database or application changes that must be effectively managed to avoid problems such as data loss, data corruptions and data duplication.
In the context of the extract/transform/load (ETL) process, any data migration will involve at least the transform and load steps. This means that extracted data needs to go through a series of functions before it can be loaded in to the target location.
Organisations undertake data migrations for several reasons. They might need to overhaul an entire system, upgrade databases, establish a new data warehouse, or merge new data from an acquisition or other source. Data migration is also necessary when deploying another system that sits alongside existing applications.
Like any other structured programme of work, to be successful, a data migration project requires a strong strategy to be in place to ensure the objectives and benefits of the programme beyond the transition period are delivered.
Project goals must be well defined, stakeholders engaged and key resources with the necessary knowledge and experience to guide the project to a successful conclusion secured.
The benefits and impact of the migration on the business should be clearly communicated and understood so that if requirements change, the value of implementing these changes is thoroughly assessed and understood.
Often at the outset of a data migration project during the Discovery Phase issues regarding data integrity, quality and the real value of the data to the business are uncovered. Decision will need to be made as to the extent to which such issues must be addressed prior to any migration occurring. For example, before the data migration process begins, it is important to understand what records need to move across to the new target system (remembering that your current system could include duplicates) or if there are any data events that could impact data quality. Transferring records from one system to another comes with risks such as data loss, compatibility issues or a security breach, all of which could be incredibly bad for business.
Data migration is a big undertaking, so it is vital to ensure that you have a knowledgeable and experienced technical team in place from the outset who can provide input into creating a solid data migration strategy.
Data migration projects typically include the following key phases:
- Strategy
- Planning
- Data Discover
- Programme Design
- Transformation
- Testing
- Transition
- Post Transition activities
Research shows that data migration projects are challenging and difficult to achieve successfully. However, by following a structured methodology, using a rigorous approach to project management and taking account of the many pitfalls that beset data migration projects, organisations will achieve higher success rates and a better level of satisfaction from all stakeholders.
Why NWT?
Data migration is a complex process, requiring a robust methodology. NWT provides a proven data migration methodology that includes pre-migration scoping followed by project assessments and a core migration process.
Our flexible and responsive Data Migration as a Service (DMaaS) provides the expertise you need on demand, when you need it, to deliver a seamless, effortless and secure data migration from an existing system to a new or updated system with minimal, if any, downtime in between.
NWT’s reputation for providing the highest quality senior Data Migration Consultants is unmatched. Our data migration specialists have considerable expertise in business and system data reconciliation strategies and execution within both the private and public sector. Following industry specific best practices and adhering to legal requirements and data privacy standards, our specialist use thorough, risk-adverse migration techniques for both manual and automated data extraction, cleansing, loading, verification, testing and implementation.
Our specialists manage the entire migration process on your behalf or work as part of your own internal data migration team, immediately adding value through knowledge transfer and sharing.
As with any project, engaging with the business is paramount to ensuring that the motives are understood and that the objectives of the migration are clear. During the Strategy Phase, NWT works in partnership with the client to help prove the viability of the data migration, if this has not already been established. NWT has considerable real-world experience in data migration and understands that best strategy for each data migration project or programme will depend upon how much data you need to move, the types of workloads involved and how quickly the migration must be completed, together with an understanding of the security requirements that must be adhered to during migration. We do this by:
During the Data Discovery Phase, NWT can help to confirm and define data quality and the basis for mappings and transformations. We will conduct a thorough data assessment to identify areas of potential problems and, if necessary, address those before the migration occurs. Ideally the process of understanding the data should be driven from the business requirements, not from the data supporting them – this will help reduce the reliance on the technical aspects of the existing data which can potentially limit the delivery of the business information. At this stage, inhibitors to the migration can be identified through data profiling, such as inconsistent referential integrity or non-compliant data formats.
During the Design Phase, NWT will help to define the overall design of the migration, identifying the components, architecture, processes and interactions that will be necessary for the migration. During this phase, the mappings and transformations will be specified and, crucially, this will be the first point at which the business will sign off the initial mappings and assumptions. Coding of the transformations will then start, with testing being an integral part of the development process – preferably automated. The tools being used should provide business-focused translation of the technical transformations so that business users can understand the data flow and dependencies between data elements. At key stages the transformations and the output can then be viewed in the context of the business rules and this process will be repeated throughout to provide a level of confidence in the transformation processes during rehearsals and the transition itself.
During the Transformation Phase, the actual migration activity begins including all the conforming and cleansing that has previously been identified and accommodated within the code, along with other data flow process such as flow control and error handling. This phase will inevitably include some reporting to verify the volumetrics and context of the transformations and provide the basis on which the migration is proven to be accurate.
Both the Testing and Resilience and failover/recover Phases includes work at multiple levels including Testing and Resilience strategy, creation of a Recovery Plan to ensure that the business can continue to function in the event of a partial or complete failure.
During Transition Phase, NWT can help to define a Work Plan that covers in detail the mechanics of the transition, estimated timings, responsibilities, contingencies, entry and exit criteria at all levels, escalation points and paths. The Work Plan is an absolute, detailed definition of what will happen and when it is planned to happen including defining multiple dependencies and parallel tasks.
This plan is the reference point for all those involved in the migration and the transition and will have been widely circulated so that there is no ambiguity as to the content. The teams will have provided detailed input to the plan so that they are comfortable with the level of detail and criteria defined for the handover points within the plan. The plan includes all the pre-requisites which may be required or start weeks before the actual transition weekend. Typically, the plan documents:
- Pre-cutover activities and pre-requisites
- Approval
- Resources
- System and environment readiness
- Escalation paths
- References to other processes and documents required for successful transition
- Impact and dependencies on other systems – such as operational tasks, backups etc.
- Timings for all tasks and contingency available at different granularities
- Tasks identified that can run in parallel and where dependencies exist between tasks
NWT’s key objective during the Transition Phase is to carry out the data migration and all associated activities successfully including ensuring that:
- The stated objectives of the migration have been agreed, understood and signed off
- Testing has been satisfactorily completed and a good level of confidence achieved
- The transition has been rehearsed several times with a predictable and consistent outcome, notwithstanding extraneous factors that may be present on the day of the actual transition
- Entry and exit criteria – including reporting and reconciliation – have been well defined and agreed such that key people are happy to make decisions based upon such information
- There’s an unambiguous set of tests that help key individuals determine at any stage whether the migration is a “go” or “no go”.
During every phase, quality real time information must be available at a sufficiently granular level to be able to make quick, well-founded and substantiated decisions. This will provide an adaptable environment within which dynamic elements of the transition can be managed and accommodated. The level of detail must be more than an event summary for example, it would be insufficient to report that a 15 hour process has had problems without understanding at which point in those 15 hours a problem occurred and, if possible, the level of detail should give metrics that would help identify problems at the earliest opportunity
NWT will help you to understand and combat the potential risks of data migration be it a data warehouse or a back office settlement system. Our data migration specialists will help you design and implement a successful migration and ensure that the checks and controls have been placed properly to deliver a seamless, effortless and secure migration. Our consultants will use a proven set of repeatable and predictable data migration processes and data migration tools to accelerate your migration.
Our fully-managed DMaaS service provides the expertise, tools and bandwidth you need, when you need it to help you get your migration project completed on time whilst minimising your project risks. If required, we can also provide ongoing support to maintain high availability and resolve uptime issues quickly.
For more information, please feel free to contact NWT at info@newworldtech.io
