Skip to content

Bulk data tranfsfer#

To make massive data transfers, there are two transfer nodes that allow large amounts of data to be copied and downloaded to the user space by directly accessing the data backbone network.

We have two nodes enabled for this purpose that provide greater bandwidth between the nodes themselves and the shared storage system. These nodes are accessible on the addresses:

Name IP Address
tn-t01.hpc.iter.es 193.146.150.177
tn-t02.hpc.iter.es 193.146.150.178

The transfer of information through these nodes is through the use of secure protocols over SSH such as SFTP (SecureSHell File Transfer Protocol or Secure File Transfer Protocol). SFTP allows you to easily download and upload files, while providing us with confidentiality of the transmitted data. In contrast, on an FTP server we do not have any type of security, since user credentials and all data traffic are sent unencrypted. For this, it is necessary to have a client for this protocol on the local machine, such as sftp on linux or psftp on windows .

Remember to disconnect from the VPN when you want to transfer massive data. Otherwise, your computer will route all traffic through the VPN with a lower performance.

Rsync for Linux users#

Rsync is a very flexible synchronization tool that due to its universal presence on Linux and Unix-like systems, and its popularity as a system scripting tool, is included in most Linux distributions by default.

To sync the contents of a directory named source to another destination on the same system, type the following:

rsync -r source/ destination

The arguments source/ destination is different from source destination

rsync -r source/ destination copy source content in destination rsync -r source destination copy source directory inside destination

Sync with a remote system:

rsync -av ~/source username@remote_host:destination

Sync remote system with a local directory

rsync -av username@remote_host:source ~/destination

Opciones útiles para rsync#
-r: means recursive, (for directories)
-a means all similar to -r
-v means verbose, show list of copied files/directories
-n o --dry-run check argument byt no copy
-az compress tranfer
-P o --progress show progress
--delete delete files in destination if they are not source
--exclude 'f o d' exclude files or directory from source directory
--include 'f o d' limit the files trasnferred by rsync

Rsync for Windows Users#

The easiest and most recommended is to install Windows Subsystem for Linux (WSL) on your computer. It works as a Windows utility and allows you to run the Linux distribution in command line mode under Windows 10 and 11 operating system.

The WSL installation procedure is not complicated and it is well documented on the official Microsoft website https://learn.microsoft.com/es-es/windows/wsl/install. There you will find a detailed description of what you are doing. In summary it consists of:

  1. Open a Windows Powershell window
  2. Install WSL: wsl-install The default installed distribution is Ubuntu
  3. Open the default distribution: wsl
  4. Use the linux commands to go to the directory containing the files. cd C:\mydirectory\....
  5. Follow the same steps as indicated in the section rsync for linux users

wsl-install

sFTP for Linux users#

You can open the connection to the server through IP public:

C:\Users\Usuario>psftp myuser@subdomain.domain.com
C:\Users\Usuario>psftp myuser@IPserver

Most used commands#

Once the session is open, the help command will display the list of commands that can be used. The most used ones are described below and they are valid for Linux users such as Windows:

sftp> help      -- Show help

sftp> cd  dir   -- change to remote directory .
sftp> lcd dir   -- change local directory.

sftp> pwd       -- show remote directory where you are working.
sftp> lpwd      -- show local directory where you are working.

sftp> put file1.zip ...       -- Upload from local directory to remote.
sftp> get file1.zip ...       -- Download file from remote to local directory

sftp> put -r directory        -- Upload directory from local to remote
sftp> get -r directory        -- Download from remote to local

sFTP for Windows users#

For Windows users there are many applications both with the command line and with a graphical interface. Among them the best known are: psftp, Filezilla y WinSPC.

psftp#

PSFTP is the SFTP command line client that is installed with the PuTTY (a very popular SSH client for windows).

There are three ways to run PSFTP:

  1. Click on the Windows Start button and go to All Programs. From the list of program, Click on PuTTY and then PSFTP.
  2. To enter the following path in the address bar C:\Program Files (x86)\PuTTY and then double-click on psftp.exe.
  3. From Windows command-line interpreter Command Prompt or PowerShell.

Windows command line Windows command line

However, to open it from the command prompt, It is recommended to include the directory in which PSFTP has been installed in the user's PATH here

set PATH=C:\path\to\putty\directory;%PATH%

You can open the connection to the server through its public IP.

C:\Users\Usuario>psftp myuser@IPserver

Filezilla#

FileZilla is one of the most used programs to be used as an FTP / FTPS and FTPES client, but it also incorporates the possibility of connecting to an SFTP server. You just have to enter the following syntax Server: sftp: // IP in the address bar and the username and password that have been provied in its corresponding place.

Filezilla img

WinSCP#

WinSPC is a popular free download client available for Windows. It supports the SFTP, SCP, WebDAV and FTP protocols and is mainly focused on the transfer of files, the use of scripts and basic functionalities of a file manager.

WinSPC

Otros#

sFTP for MAC users#

The MAC user can use the terminal in the same way as Linux users or install an application with a graphical interface like the following: