FAQ - How to copy DSEFS files from one DSE cluster to another

Overview

This article provides instructions on how to copy DSEFS files across DataStax Enterprise clusters.

Prerequisites

• DSE source and target clusters must have nodes running in Analytics mode.
• DSE Analytics nodes have DSEFS enabled and configured.
• DSE clusters need to be running the same major DSE version for compatibility.

Limitations

• Some features may not be compatible when source and target clusters are on different DSE versions.
• Execute DSEFS commands on the cluster running with the newer DSE version though it is not guaranteed to work with mixed versions.
• If authentication is enabled, both clusters must share the same credentials since it not possible to supply multiple credentials on the command line.

Background

There are cases where a user or application requires the ability to copy files across clusters. Examples are data migration, moving data between production and staging, populating data on a development environment.

In order to provide working examples, the following 2 clusters were created for this article:

CLUSTER 1 - DSEFS source

Datacenter: Analytics
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns    Host ID                               Rack
UN  10.100.1.10   126.79 KiB 128          ?       38d9fe7a-a848-4b77-b440-803550d28db3  rack1

CLUSTER 2 - DSEFS destination

Datacenter: Analytics
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  10.200.2.20    126.79 KiB 128          ?       96a457be-d47b-43f8-ace7-76dabcea49d5  rack1

Preparation

On the source cluster, a test directory was created in DSEFS as follows:

10.100.1.10 $ dse fs "mkdir portfolio"
10.100.1.10 $ dse fs "ls"
portfolio

Then one of the files from a local demo directory was copied to DSEFS:

10.100.1.10 $ dse fs "put demos/portfolio_manager/README.md portfolio/README.md"
10.100.1.10 $ dse fs "ls -R portfolio"
/portfolio:
README.md

On the destination cluster, the destination directory was created in DSEFS as follows:

10.200.2.20 $ dse fs "mkdir assets"
10.200.2.20 $ dse fs "ls"
assets

Procedure

The file was copied to the remote cluster as follows:

10.100.1.10 $ dse fs "put portfolio/* dsefs://10.200.2.20:5598/assets/"

Check that the file exists remotely:

10.100.1.10 $ dse fs "ls -R dsefs://10.200.2.20:5598/assets"
/assets:
README.md

Check locally on the destination cluster:

10.200.2.20 $ dse fs "ls -R assets/"
/assets:
README.md