NetApp BlueXP OnPrem to Cloud ONTAP Data Migration

Cloud Volume ONTAP Data Migration

Overview

Before we get into the data migration topic, we need to understand the difference between the process of Migration Vs Conversion Vs Integration.

Data Migration: Data migration is the process of moving data between location , formats or system. (Re-host - Lift and Shift, Redeploy data and application on IaaS without making any changes.)

Data Conversion: Is the process of transforming data from one format to another. Data will be extracted from the source, altered and loaded into the new target. (Revise - Re architect, Modify or extend the existing application to fit into the new cloud environment.)

Data Integration: Is the process of combining data at different source to provide users with a unified view of all data.

In this blog, we would be mostly talking about the data migration process. Also, I would be writing seperate blog for NetApp Blue XCP Networking requirments.

Data migration is the process of moving data from one place to another. Mostly for all the window's volume shares, robocopy command is widely used. However, there are few challenges when copying or migrating data from any to Cloud Net App considering the data gravity.

How data affects other data as it grows
How data is integrated into a business
How data becomes customized over time

***************************************************************************

Different type of data migration

Storage Migration : Process of moving data off existing arrays into more modern one that would enable other system to access it. Data management features such as cloning, snapshots, backup and disaster recovery.

Cloud Migration : Process of moving data, application or other business data from either on-premises data center to cloud or from one cloud to another.

Application Migration : Process of moving an application program from one environment to another. Moving the entire application from on-premises IT Center to cloud or moving between two clouds or moving the application's data to a new form of application hosted by the provider.

Migration Plan
Moving important or sensitive data and decommissioning legacy systems can be disaster for data retrieval in worst case scenarios. So having a solid plan is must.

Please refer Migration plan and checklist
https://www.datamigrationpro.com/
https://www.datamigrationpro.com/data-migration-checklist-planner/

Data Migration involves 3 steps

1. Extract data
2. Transform data
3. Load data

All these steps are not needed for every migration, depends on every situation it will be unique and the approaches from stakeholders would be different as well.

***************************************************************************

Pre-Migration Check List

Ensure no Duplicate data is considered during data migration, because that duplicate data may not be necessary to move to the cloud. Migrating this data will increase the total amount of time and cost for the migration. Also, storing duplicate data in the cloud will also increase the cost.

Action Items: We need to ensure the discovery should be perfomred to identify the duplicate data and Analyze is there a reason to migrate more than a single copy of this data. And make maximum efforts to remove any unnecessary duplicate data.

Cloud Data Sense can pointout the duplicate data and show it's usage patterns so we can remove it before the migration takes place.

Non-Business Data A large portion of data set will not be related to the operations or business. And migrations of non business data will increase the length of the migration process.

Analyze the data set to identify mission-critical data from non-business data. Remove the non-business data from the target data to migrate. Determine if this data needs to be migrated to the cloud. Identify any non-business data that contains sensitive personal information that should not be migrated.

Data Sense can analyze all the data, identifying the non-critical data that doesn’t need to be in the cloud

Sensitive data requires a higher level of attention and care than other data. Becayse when it comes to GDPR, CCPA, and other data regulations have restrictions on where data can be stored. Certain industries have requirements to keep certain data stored on-prem.

Locate all of the sensitive data in data migration. Analyze the value of migrating the sensitive and potentially sensitive data to the cloud. Tag all the potentially sensitive data that don’t want to migrate.

Data Sense is able to map, identify, and flag potentially sensitive data that have in the storage systems. The report will also show the data with open permissions that will require immediate attention.

Stale data is more than just not using frequently in the organization, it just not relevant anymore. Moving those data to the cloud will waste time and increase cloud costs. Stale data can also contain sensitive information that should not be migrated.

Locate and tag all the stale data during migration.Determine the business value of this data.Decide whether to keep this data for archival purposes or to delete it.

Data Sense scans can identify the stale data in your systems before you make the mistake of moving it to the cloud.

Custom Data might have specific data that organization wants to keep out of the cloud. This data should be located and separated from the migration. There may be multiple custom types of data that fall into this category.

Consider whether there are any specific data types that should not migrate.Identify where this data exists. Remove the custom data from the migration data set.

Data Sense allows to add custom data types that can be identified so the data can be sectioned off before the migration takes place.

***************************************************************************

Pre-migration planning : In this step, gather all the data being moved from on-prem to Cloud.

Project Initiation: Identify the key stakeholder will be part of the migration. Like (Storage team, Active Directory team, Network team, Application team and Migration team including the project management)

Analysis: Perform the landscape analysis and establish a data quality rule management process on the goals of project

Solution Design: Identify which data to move and the data quality of before and after move.

Build & Test: Test the migration with a mirror of production environment.

Execute & Validate: After testing the use case with non production environment, determine whether the data move is working as expected

Decommission and Monitor: After making sure all the data migration is completed and application and system functionality are working as needed, then plan to decommission the old / legacy system and monitor the new environment.

Migration Prerequisites: To initiate the data transfer from source to destination, we would need to install and configure the XCP tool. This tool is a client software enables fast and reliable any to NetApp and Net App to Net App data migration.

This tool is free to use, it was designed by NetApp to handle volumes with millions of files, it scans and discovers the statistics of files and directory, it can copy fast and parallelized file transfer, it can sync and to get the incremental updates to the target, it would also verify the target state, when there was any interruption the copy operation can be resumed. The main importance of this migration tool is that we can perform live migration from live data source or read only data source.

***************************************************************************

Migration Tool Setup

XCP is free tool, installing , configuring and obtaining license is a matter of minutes. NetApp offers a free one year license. After one year we can renew the lincense. XCP is available for both Linux and Windows OS.

Latest NetApp XCP is now available 1.9.2. https://xcp.netapp.com/

Step 1: Register NetApp NSS Account. https://mysupport.netapp.com/site/user/registration

Step 2: Download XCP NFS or XCP CIFS to a host. https://mysupport.netapp.com/site/products/all/details/netapp-xcp/downloads-tab

Step 3: Login and download a license.
https://xcp.netapp.com/license/xcp.xwic

Step 4: Deploy the license file to XCP host.

XCP host must have internet connectivity so that it can access NetApp site for activation and license control.

Migration Workflow

Ports used by XCP

Service Ports

CIFS 445 TCP/UDP

HTTP 80

HTTPs 443

NFS 111 TCP/UDP and 2049 TCP/UDP

XCP 5030

Porstgresql 5432

HDFS 7222

***************************************************************************

Installing XCP tool on Linux machine : NFS as root user

Step 1: Log in to the Linux machine as the root user and download and install the license:

[root@orgspr19872023 ~]# ls -l
total 36188
-rw-r--r--  1 root root 37043983 Oct  5 09:36 NETAPP_XCP_<version>.tgz
-rw-------. 1 root root     1994 Sep  4  2019 license

Step 2: To extract the tool, untar XCP

[root@orgspr19872023 ~]# tar -xvf NETAPP_XCP_<version>.tgz
[root@orgspr19872023 ~]# ls
NETAPP_XCP_<version>.tgz license xcp
[root@orgspr19872023 ~]# cd xcp/linux/
[root@orgspr19872023 ~]# ls
xcp

Step 3: Verify that the /opt/NetApp/xFiles/xcp path is available on the system from a previous version of XCP.

If /opt/NetApp/xFiles/xcp is available, then activate the license by using the xcp activate command and proceed with data migration.

If /opt/NetApp/xFiles/xcp is not available, when you run the xcp activate command is run for the first time, the system creates the XCP host configuration directory in /opt/NetApp/xFiles/xcp.

The xcp activate command fails because the license is not installed:

[root@orgspr19872023 ~]#  ./xcp activate
(c) yyyy NetApp, Inc.
xcp: Host config file not found. Creating sample at '/opt/NetApp/xFiles/xcp/xcp.ini'

xcp: ERROR: License file /opt/NetApp/xFiles/xcp/license not found.
Register for a license at https://xcp.netapp.com

Step 4: Copy the license to /opt/NetApp/xFiles/xcp/:

[root@orgspr19872023 ~]# cp ~/license /opt/NetApp/xFiles/xcp/

Step 5: Verify that the license file was copied to /opt/NetApp/xFiles/xcp/:

[root@orgspr19872023 ~]#  ls -altr /opt/NetApp/xFiles/xcp/
total 44
drwxr-xr-x 3 root root    17 Oct  1 06:07 ..
-rw-r--r-- 1 root root   304 Oct  1 06:07 license
drwxr-xr-x 2 root root     6 Oct  1 10:16 xcpfalogs
drwxr-xr-x 2 root root    21 Oct  1 10:16 xcplogs
-rw-r--r-- 1 root root   110 Oct  5 00:48 xcp.ini
drwxr-xr-x 4 root root    83 Oct  5 00:48 .
[root@orgspr19872023 ~]#

Step 6: Activate XCP

[root@orgspr19872023 ~]#  ./xcp activate
XCP <version>; (c) yyyy NetApp, Inc.;
XCP activated

***************************************************************************

Installing XCP on Windows VM : SMB Setup Procedure

Step 1: Download the license and the XCP SMB binary NETAPP_XCP_<version>.tgz on a Windows client.

Step 2: Extract the NETAPP_XCP_<version>.tgz file.

Step 3: Copy the xcp.exe file to your Windows C: drive, this binary is available inside NETAPP_XCP_1.8\NETAPP_XCP_1.8\xcp\windows after you extract the tgz file.

Step 4: Verify that the C:\NetApp\XCP path is available on the system from a previous version of XCP.

If C:\NetApp\XCP is available, activate XCP by using the xcp.exe activate command and proceed with data migration.

If C:\NetApp\XCP is not available, the system creates the XCP host configuration directory and files it at C:\NetApp\XCP when you run the xcp.exe activate command for the first time. The xcp.exe activate command fails and creates an error message asking for a new license.

C:\>xcp.exe activate
(c) yyyy NetApp, Inc.

License file C:\NetApp\XCP\license not found.
Register for a license at https://xcp.netapp.com

Step 5: Copy the license to the newly created folder C:\NetApp\XCP:

C:\>copy license c:\NetApp\XCP
        1 file(s) copied.

Step 6: Activate XCP:

C:\>xcp.exe activate
XCP SMB; (c) yyyy NetApp, Inc.;

XCP activated

***************************************************************************

Migration Commands

SMB Data Migration Plan

Show

The show command shows all SMB shares available on the server with the permissions and space available.

Example:

xcp show \\<SMB file server IP/FQDN>
xcp show smb_server01.netapp.com

Scan

The scan command recursively scans the entire SMB share and lists all the files at the end of the scan.

Example:

xcp scan \\SMB server\share1
xcp scan smb_server01.netapp.com:/share1

After planning the migration with show and scan commands we can initiate the migration.

Copy

The copy command scans and copies the entire source directory structure to a destination SMB share. The copy command requires having source and destination paths as variables. The scanned and copied files, throughput/speed, and elapsed time details are printed to the console once every five seconds.

Syntax 
C:\xcp>xcp copy -acl -root -fallback-user \\<source SMB share> \\<destination SMB share>
Example 
C:\xcp>xcp copy -acl -root -fallback-user "domain\senthil.adm" -fallback-group BUILTIN\Users \\10.14.1.17\IND-Finance-App$ \\10.28.1.17\IND-Finance-App$  > C:\xcpcopy\IND-Finance-App.log

***************************************************************************

Post Migration Comparison and Sync
Sync

The sync command scans for changes and modifications in the source and target shares in parallel, and applies the appropriate actions (remove, modify, rename, and so on) to the target to make sure that the target is identical to the source. The sync command compares data content, time stamps, file attributes, ownership, and security information.

Syntax
C:\xcp>xcp sync \\<source SMB share> \\<destination SMB share>
Example
C:\xcp>xcp sync -v -acl -root -fallback-user "domain\senthil.adm"
 -fallback-group BUILTIN\Users \\10.14.1.17\IND-Finance-App$ \\10.28.1.17\IND-Finance-App$

Verify

The verify command reads both source and target shares and compares them, providing information about what is different. You can use the command on any source and destination, regardless of the tool used to perform the copy or sync.

Syntax
C:\xcp>xcp verify \\<source SMB share> \\<destination SMB share>
Example
C:\xcp>xcp verify \\10.14.1.17\IND-Finance-App$ \\10.28.1.17\IND-Finance-App$

***************************************************************************

NFS Data Migration

***************************************************************************

Migration Challenges

Data governance

Lack of Expertise

Lack of planning

Erros during migration

xcp copy interrupted

If your environment is having security tools like CyberArc , then it would chnage the domain admin password every 24 hours and that would interrupt the migration , Added exclusion of domain admin account for password change frequency to make the uninrerrupted migration.

Application scripts accessing the shares during the migration, which was causing migration failure. We have created the action plan for such shares to have less downtime , and do the data migration and do the cut over during weekend by making the source share as read only permission.

Migration prerequisites checks need to be perfomred, flow and access are should be validated. For the on-prem CVO shares which is used for migration it will not have root level permission initially. Need to grant the root level permission.

xcp scan Error: The network name cannot be found,

This is mostly when you type the incorrect share name.

xcp copy Error: ERROR failed to obtain fallback security principal

The fallback user/group does not exist at the target system (destination box) or active directory. Rerun the command with correct fallback user/group options

xcp copy with acl migration

Error message logged in xcp.log file:
pywintypes.error: (1314, 'GetNamedSecurityInfo', 'A required privilege is not held by the client.')

A user is facing an issue related to security descriptors because with the privileges that the migrations user owns, XCP can only get owner, group, and DACL, it cannot get SACL. Add your migration user to "Manage Audit and Security Log” policy in your Active Directory.

***************************************************************************

Scenario

Copy works, with Symbol symbols or emoji in the file name:

Source

xcp 1.9.1

c:\>xcp.exe copy \\10.x.x.x\source \\10.y.y.y\dest

xcp.exe copy \\10.x.x.x\source \\10.x.x.x\dest

4 scanned, 0 matched, 0 skipped, 3 copied, 274 (214/s), 0 errors

Total Time : 1s

STATUS : PASSED

Sending statistics...

Target

Both source and destination volume in this case have utf8mb4 volume language:

cluster1::*> vol show -fields language

vserver volume language

----------- ------ --------

cluster1-01 vol0 -

svm1 svm1_root

C.UTF-8

svm1 vol1 utf8mb4 <<<<<<<<<<<< source

svm1 vol2 utf8mb4 <<<<<<<<<<<< destination

svm1 vol3 C.UTF-8

5 entries were displayed.

But when I modify the destination to c.utf-8 volume language, the copy fails:

c:\>xcp.exe copy \\10.x.x.x\source \\10.x.x.x\dest2

--- Logging error ---

Traceback (most recent call last):

File "logging\__init__.py", line 1086, in emit

File "encodings\cp1252.py", line 19, in encode

UnicodeEncodeError: 'charmap' codec can't encode characters in position 91-92: character maps to <undefined>

Call stack:

File "threading.py", line 930, in _bootstrap

File "threading.py", line 973, in _bootstrap_inner

File "threading.py", line 910, in run

File "xcp\xcpLogging.py", line 249, in receive

File "logging\handlers.py", line 75, in emit

File "logging\__init__.py", line 1187, in emit

File "logging\__init__.py", line 1091, in emit

Message: 'failed to copy "Filename🏖️.pdf.txt": (123, \'CreateFile\', \'The filename, directory name, or volume label syntax is incorrect.\')'

Arguments: None

failed to copy "Filename🏖️.pdf.txt": (123, 'CreateFile', 'The filename, directory name, or volume label syntax is incorrect.')

--- Logging error ---

Traceback (most recent call last):

File "logging\__init__.py", line 1086, in emit

failed to set attributes for "🎑bla": (123, 'CreateDirectory', 'The filename, directory name, or volume label syntax is incorrect.')

File "encodings\cp1252.py", line 19, in encode

UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f391' in position 97: character maps to <undefined>

Call stack:

File "threading.py", line 930, in _bootstrap

File "threading.py", line 973, in _bootstrap_inner

File "threading.py", line 910, in run

File "xcp\xcpLogging.py", line 249, in receive

File "logging\handlers.py", line 75, in emit

File "logging\__init__.py", line 1187, in emit

File "logging\__init__.py", line 1091, in emit

Message: 'failed to set attributes for "🎑bla": (123, \'CreateDirectory\', \'The filename, directory name, or volume label syntax is incorrect.\')'

Arguments: None

xcp.exe copy \\10.x.x.x\source \\10.x.x.x\dest2

4 scanned, 0 matched, 0 skipped, 2 copied, 274 (185/s), 2 errors

Total Time : 1s

STATUS : FAILED

Sending statistics...

If you get similar error then check the volume language on destination volume.

***************************************************************************

Supported Features.

Unsupported Features.

When migrating data from Onprem or any source to NetApp cloud using NetApp XCP tool then it does not scan or copy or sync the file or folders which contains special symbols. As of now NetApp confirms to rename the source side NetApp Folders/Files and redo the migration.

IPv6 - Does not support IP version 6 (IPv6)

Does not support third-party to NetApp NFSv4 ACLs

Linux XCP is no longer supported on earlier distributions of Linux that were supported by XCP 1.6.3.

XCP does not support NFS to S3 migration.

XCP does not support combining baseline or incremental Snapshot copy operations with live source migrations.

***************************************************************************

Next Blog Link
In Progress - Blog for NetApp Blue XCP Networking requirments.

Search This Blog

Sensiffy Cloud Creators

NetApp BlueXP OnPrem to Cloud ONTAP Data Migration

Comments

Post a Comment