Skip to main content

How to move files to Swestore using iRODS?

Printer-friendly version

Question

How to move files to Swestore using iRODS?

Answer

The main storage at UPPNEX is optimized for high-speed and parallell access, which makes it expensive and not the right place for longer time archiving. Thus, for files which you're not actively working on, you should consider moving to the national storage on SweStore. This is how to do it.

Introduction

For accessing the Swestore national storage, we use (since Sep 2012) iRODS, and more specifically the iRODS client called "icommands", which is a set of commands similar to the unix commands "ls", "cd", "imkdir" etc, but with an "i" in front, plus two FTP-like commands: "iput" and "iget" (plus some more, but those are the ones you need most).

Note that data put on Swestore via iRODS at Uppmax is not accessible through other means.

Activate the iRODS icommands

The iRODS icommands client is activated through the module system, as so many other things on UPPMAX.

1. Log in to milou
2. Execute:

module load irods/swestore

List files

After activating the iRODS icommands, you will be placed in one of your projects. if you run "ils", you will then see a listing of files and folders in that project, something like this:

[samuel@milou1 ~]$ ils
  /ssUppnexZone/proj/b2011221:
    C- /ssUppnexZone/proj/b2011221/firstRun

Navigate around

To enter one of the (or the only) folder(s), do:

icd [proj-id]

... in this case:

icd firstrun

To switch to another project, use

icd ..

to back up a level. After that, you can change to another project or folder using icd as previously

icd b2011222

Upload files

To upload the above mentioned folder, do (-r is needed to recurse into directories):

iput -r [a local folder]

For single files, the -r flag is not needed:

iput [a local file]

To list the newly uploaded file/directory:

ils

To create folders, do:

imkdir [folder-name]

If you want to verify the upload outside of iRODS, there's a utility called ssverify.sh that will let you do that.

$ ssverify.sh test /ssUppnexZone/proj/p2061001/test
Checking if test/a matches /ssUppnexZone/proj/p2061001/test/a... seems good!
Checking if test/b matches /ssUppnexZone/proj/p2061001/test/b... seems good!
Checking if test/c matches /ssUppnexZone/proj/p2061001/test/c... seems good!
Checking if test/du matches /ssUppnexZone/proj/p2061001/test/du... seems good!
Checking if test/foo matches /ssUppnexZone/proj/p2061001/test/foo... seems good!
It seems test matches /ssUppnexZone/proj/p2061001/test on Swestore.
$  echo femtiofyra > test/c
$ ssverify.sh test /ssUppnexZone/proj/p2061001/test
Checking if test/a matches /ssUppnexZone/proj/p2061001/test/a... seems good!
Checking if test/b matches /ssUppnexZone/proj/p2061001/test/b... seems good!
Checking if test/c matches /ssUppnexZone/proj/p2061001/test/c... NO! Something is wrong! 
Either the files don't match or something went wrong, please contact 
staff@uppmax.uu.se to investigate.

There is also a command

irsync

that behaves in a similar way to regular rsync, which is recommended for moving large/deep data structures.

 

Downloading files

To download a file again, do:

iget [a file in iRODS]

... or, for folders, do it recursively:

iget -r [a folder in iRODS]

File removal

To remove a file, you'll like need to use -f to irm in order to bypass the use of a trash area, which is currently not supported at Uppmax.

Accessing Swestore directly

If you for some reason want to be able to access your files in Swestore directly, without having to login to UPPMAX first, then you can contact us for a separate password and then install a client like e.g. iDrop on your personal computer. Note that this requires that you are connected to SUNET.

More hints

  • You can have iput show the progress with the -P flag
  • If you want to be placed in a particular directory when you load the irods module, you can create a file called
    $HOME/.irods/.irodsEnv
    containing
    irodsHome '/ssUppnexZone/proj/myproj'
    irodsCwd '/ssUppnexZone/proj/myproj'
    where
    myproj
    is replaced by your project id.
  • Read more info about the respective i-commands with "[command] -h"

Need help?

Mail support@uppmax.uu.se