FAQ - How to copy DSEFS files from one DSE cluster to another – DataStax Support

Overview

This article provides instructions on how to copy DSEFS files across DataStax Enterprise clusters.

Prerequisites

DSE source and target clusters must have nodes running in Analytics mode.
DSE Analytics nodes have DSEFS enabled and configured.
DSE clusters need to be running the same major DSE version for compatibility.

Limitations

Some features may not be compatible when source and target clusters are on different DSE versions.
Execute DSEFS commands on the cluster running with the newer DSE version though it is not guaranteed to work with mixed versions.
If authentication is enabled, both clusters must share the same credentials since it not possible to supply multiple credentials on the command line.

Background

There are cases where a user or application requires the ability to copy files across clusters. Examples are data migration, moving data between production and staging, populating data on a development environment.

In order to provide working examples, the following 2 clusters were created for this article:

CLUSTER 1 - DSEFS source

Datacenter: Analytics
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns    Host ID                               Rack
UN  10.100.1.10   126.79 KiB 128          ?       38d9fe7a-a848-4b77-b440-803550d28db3  rack1

CLUSTER 2 - DSEFS destination

Datacenter: Analytics
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  10.200.2.20    126.79 KiB 128          ?       96a457be-d47b-43f8-ace7-76dabcea49d5  rack1

Preparation

On the source cluster, a test directory was created in DSEFS as follows:

10.100.1.10 $ dse fs "mkdir portfolio"
10.100.1.10 $ dse fs "ls"
portfolio

Then one of the files from a local demo directory was copied to DSEFS:

10.100.1.10 $ dse fs "put demos/portfolio_manager/README.md portfolio/README.md"
10.100.1.10 $ dse fs "ls -R portfolio"
/portfolio:
README.md

On the destination cluster, the destination directory was created in DSEFS as follows:

10.200.2.20 $ dse fs "mkdir assets"
10.200.2.20 $ dse fs "ls"
assets

Procedure

The file was copied to the remote cluster as follows:

10.100.1.10 $ dse fs "put portfolio/* dsefs://10.200.2.20:5598/assets/"

Check that the file exists remotely:

10.100.1.10 $ dse fs "ls -R dsefs://10.200.2.20:5598/assets"
/assets:
README.md

Check locally on the destination cluster:

10.200.2.20 $ dse fs "ls -R assets/"
/assets:
README.md