Transferring Data

Warning

This page has been deprecated, and relevant information is now available on Data Storage and Transfers. Please update any bookmarks to use that page.

Using common terminal tools

In general, when transferring data into or out of the OLCF from the command line, it’s best to initiate the transfer from outside the OLCF. If moving many small files, it can be beneficial to compress them into a single archive file, then transfer just the one archive file.

scp and rsync are available for remote transfers.

  • scp - secure copy (remote file copy program)

    • Sending a file to OLCF
    scp yourfile $USER@dtn.ccs.ornl.gov:/path/
    • Retrieving a file from OLCF
    scp $USER@dtn.ccs.ornl.gov:/path/yourfile .
    • Sending a directory to OLCF
    scp -r yourdirectory $USER@dtn.ccs.ornl.gov:/path/

  • rsync - a fast, versatile, remote (and local) file-copying tool

    • Sync a directory named mydir from your local system to the OLCF
    rsync -avz mydir/ $USER@dtn.ccs.ornl.gov:/path/

    where:

    • a is for archive mode
    • v is for verbose mode
    • z is for compressed mode
    • Sync a directory from the OLCF to a local directory
    rsync -avz  $USER@dtn.ccs.ornl.gov:/path/dir/ mydir/
    • Transfer data and show progress while transferring
    rsync -avz --progress mydir/ $USER@dtn.ccs.ornl.gov:/path/
    • Include files or directories starting with T and exclude all others
    rsync -avz --progress --include 'T*' --exclude '*' mydir/ $USER@dtn.ccs.ornl.gov:/path/
    • If the file or directory exists at the target but not on the source, then delete it
    rsync -avz --delete $USER@dtn.ccs.ornl.gov:/path/ .
    • Transfer only the files that are smaller than 1MB
    rsync -avz --max-size='1m' mydir/ $USER@dtn.ccs.ornl.gov:/path/
    • If you want to verify the behavior is as intended, execute a dry-run
    rsync -avz --dry-run mydir/ $USER@dtn.ccs.ornl.gov:/path/

See the manual pages for more information:

$ man scp
$ man rsync
  • Differences:
    • scp cannot continue if it is interrupted. rsync can.
    • rsync is optimized for performance.
    • By default, rsync checks if the transfer of the data was successful.

Note

Standard file transfer protocol (FTP) and remote copy (RCP) should not be used to transfer files to the NCCS high-performance computing (HPC) systems due to security concerns.

Using Globus from your local machine

Globus is most frequently used to facilitate data transfer between two institutional filesystems. However, it can also be used to facilitate data transfer involving an individual workstation or laptop. The following instructions demonstrate creating a local Globus endpoint, and initiating a transfer from it to the OLCF’s Alpine GPFS filesystem.

  • Visit https://www.globus.org/globus-connect-personal and Install Globus Connect Personal, it is available for Windows, Mac, and Linux.
  • Make note of the endpoint name given during setup. In this example, the endpoint is laptop_gmarkom.
  • When the installation has finished, click on the Globus icon and select Web: Transfer Files as below
../_images/globus_personal1.png
  • Globus will ask you to login. If your institution does not have an organizational login, you may choose to either Sign in with Google or Sign in with ORCiD iD.
../_images/globus_google.png
  • In the main Globus web page, select the two-panel view, then set the source and destination endpoints. (Left/Right order does not matter)
../_images/globus_laptop_summit.png
  • Next, navigate to the appropriate source and destination paths to select the files you want to transfer. Click the “Start” button to begin the transfer.
../_images/globus_laptop_transfer.png
  • An activity report will appear, and you can click on it to see the status of the transfer.
../_images/globus_laptop_activity.png
  • Various information about the transfer is shown in the activity report. You will receive an email once the transfer is finished, including if it fails for any reason.
../_images/globus_laptop_activity_done.png