Using Globus to Transfer Files

Globus provides a mechanism for high-speed, reliable file transfers between two computers.  Currently it can be used to move files from individual computers to ITS Value Storage or Rivanna storage through a data transfer node (DTN).

Getting Started

Globus transfers data between endpoints, also now called collections.  You must log in to the Globus website and log on to initiate any transfers. Visit http://globus.org and click on Log In at the top of the page.  You can log in with Netbadge by selecting the dropdown box below "Use your existing organizational login."  Search for University of Virginia to start the login process. You must have an Eservices account for this login to succeed.  If you do not, you must contact the ITS Helpdesk to enable your Eservices identity. If this is your first login with your organizational credentials, it will offer you the option to associate a personal Globus account with this login.  If you do not have a personal account you can ignore that.  If you have a personal account and do not associate it at first, you can do so later through the Account tab on the left sidebar.  At your first login, you will also have to accept the Globus terms of service before you can finalize your account.

If you only need to transfer data from an external provider that supports Globus such as XSEDE or BlueWaters, skip to Transferring Files below.  If you wish to transfer files from a local system such as a workstation or your laptop, follow the instructions below to install Globus Connect Personal on your machine.
 

Creation of a Personal Endpoint for transfer of files to or from your local computer

If you wish to transfer files from your workstation, desktop, or laptop via Globus, you must create a personal endpoint on that machine by installing Globus Connect Personal. You do not need administrator privileges for this installation.

Follow these instructions to set up your computer as an endpoint:

  • Select the Endpoints tab on the left sidebar. A new page will open with a plus sign in a circle to Create new endpoint in the upper right.  You must select Globus Connect Personal.

  • A page will appear for you to describe your new endpoint. You may choose any name but it must be unique. You may wish to start the name with UVA or similar characters.  Do not check "this will be a high assurance endpoint."  If you need secure data transfers please contact us.  When you have named you endpoint, click the blue Generate Setup Key button.

  • When the setup key has been generated, click the clipboard icon to copy it to your clipboard.  Do not copy anything else to your clipboard until your Globus Connect software has been installed.

  • Windows and Mac Users:  Follow the instructions to install the software in the usual way for your operating system.  When the installation has finished, it will automatically start the program unless you uncheck the box.  Allow it to run the program.  A dialog window will appear, prompting you for the key.  Paste in the key that you copied earlier. 
  • Globus Connect Personal will run in the background until you terminate it or shut down your computer.  If you do not change the default settings, it will automatically start on Windows and Mac when you start or restart your computer.
  • Linux Users:  Download the tarfile in a directory of your choice.  Type the following instructions at the command line to unpack the file and run the software:
    tar xzf globusconnectpersonal_latest.tgz
    cd globusconnectpersonal-X.X.X
    ./globusconnect & 

    Note:  You may need to install tcllib on your system.
    Running the binary will bring up a textbox requesting a security code.  The security code is the key that you copied earlier.  Paste the key into the textbox.  Unlike WIndows and OSX, the Linux version will not start automatically when you start or restart your computer; it must be started in this manner each time you wish to make a connection.

    • Command line interface (CLI) for Linux. 
      • You can set up and run the Globus Connect Personal for Linux using only the command line.   This could be useful if you wish to install your own personal endpoint on Rivanna, for example.  To do this, download and unpack the globusconnectpersonal_latest.tgz as described above and cd into its directory as described above, but do not start the executable.  Run as below, substituting your own name choice for "mst3k-rivanna."
                    globus endpoint create --personal mst3k-rivanna
  • This will generate a key.  Copy the key to the clipboard, then run
  • ./globusconnectpersonal -setup 224532bb-8a4b-4d32-8995-e1fb442be98e
  • where the long string is the setup-key generated in the creation step.
  • You may now start your endpoint running with
  • ./globusconnectpersonal -start &

Adding Folders

By default, Globus sets a limited number of folders to which your personal endpoint has access, particularly on Windows.  To add folders, find the Globus icon. This process will depend on your operating system.  Please note that the folder must exist; Globus cannot create folders.

Windows: From the tray in the lower right hand corner of the taskbar, bring up the icons and right-click on the g logo.  Select Options.  Click the + button on the lower right.  It will bring up a file browser from which you can select folders.  

Mac: The g logo will appear in the taskbar.  Click and select Preferences.  Choose Access, click the + button, and navigate to the new folder you wish to include.  If you are seeking to add an externally-mounted drive, on Macs those are usually mounted under /Volumes.

Managing Endpoints

If you need to modify or remove an endpoint, log in to globus.org and select the Endpoints tab if you are not already there.  Select Endpoints Administered by You.  The box enclosing an arrow icon to the right lets you start a transfer from the selected endpoint.  The triple dots show you options for the endpoint when clicked.  These icons may appear grayed out but are functional.

To modify the properties of your personal endpoint, click the "Administered by You" tab if you are not already there.  From the right-pointing arrow corresponding to the endpoint, you can open a page showing the properties of the endpoint.  You should rarely need to modify these properties, but you can do so by clicking Edit Endpoint.  You can click Delete Endpoint to remove the endpoint.

Transferring Files

You are now ready to transfer files.  If you left the page to install a personal endpoint, you can go to the File Manager tab or you can start from the Endpoints tab.

Note: it will remember your credentials for a fixed interval of time; you do not need to log in every time.

From the File Manager, start typing the name of your endpoint into the Collection text box.  It should find it.  You can also start by beginning from Endpoints and selecting Recent (if appropriate) or Your Collections and clicking on the name of your endpoint.  Your Globus Connect Personal process must be running for it to find the endpoint; it will warn you if it is not.  If necessary, start the process, then click the Refresh button.  Once your endpoint is selected, click Transfer or Sync to in order to select the other endpoint.  Note that the gray blocks merely refers to the current view and it not a "graying out."  In the other text box type uva#main-DTN.  It should not be necessary to type it in full since Globus will locate it.  You will see a list of all folders that are accessible to you.  If you have a Rivanna account you will see home and tiny.  Value storage is nv and Project storage is project.  Click on the appropriate folder.  It should open to another folder that is the name of your space (your user ID for home and tiny, or the group name for nv and project).  Click again and your files and folders will appear.

Navigate to the appropriate folder on each endpoint.  To transfer a file (or folder), click on the file name.  One of the arrows between the endpoint names will turn blue, indicating the direction for the transfer.  Click on the blue arrow to initiate the transfer.   A text box will appear above the endpoint names that states:   Transfer request submitted successfully. Task id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.

Depending on the size of the file(s) you are transferring, this may take some time.  The file transfer happens in the background; you will receive an email when the transfer is completed.  You can safely log out through the Account tab once all your file transfer requests have been submitted and acknowledged. 

Globus may not handle a large number (several thousand or more) of files in a folder.  If you have a folder with such a large number of files you wish to transfer, you may need to tar or zip or otherwise package them.

Below the file-view panes are additional transfer settings.  By default, only file integrity will be verified.  To change the default options, click the Transfer and Sync dropdown.  By checking the appropriate boxes before initiating the transfer, you may also specify settings to sync the files (i.e. do not copy a file that exists and has not changed), to delete files in a folder that do not exist on the source, to preserve modification times, and to encrypt the data.

File Sharing

You can share your folders or an individual file with others.  It is not necessary that they have UVA credentials, but they must have a Globus account.  If their institution does not support automatic Globus accounts through CILogin or another means, your collaborator can sign up for an individual Globus account.  You must know the user's Globus ID in order to share files.

Files can only be shared from folders accessible from the uva#main-DTN.  If the folder you wish to share is not visible, please contact us. Set the UVA DTN as one of your endpoints.  Navigate so that the folder or file you wish to share is visible.  Highlight it, then click share in the upper right of the right sidebar.   When the dialog appears, pick a name for the "Share Display Name."  Globus will treat this as an endpoint.  Click Create Share.  If you have already selected the folder you want to share, the Path textbox can be left alone. 

Creating the share will bring up an Add Permissions page.  Click the People+ icon to set up the sharing.  If you want the user to be able to write to your shared directory, click the appropriate checkbox, otherwise leave it unchecked.  Keep in mind that if you grant write permission, the user will be able to delete files/folders as well as modify them. Enter the Globus user ID for the recipient; for UVA users this should be the UVA email address.  Once it finds the identity/email, the Add button will ungray and you can click it.  You must click on Add Permission even if you made no changes. 

The recipient will receive an email containing a link.  When he or she clicks the link, he or she will be required to log in to Globus.  The recipient will then see the folder or the files you wanted to share.  He or she will then need to select an endpoint to which the files will be written.  Once that is done, file transfer is carried out as described above.

Globus Documentation                                                                                                                      

For more information, see the excellent Globus documentation at https://docs.globus.org/how-to/