How to move files to Swestore using iRODS?

How to move files to Swestore using iRODS?

The main storage at UPPNEX is optimized for high-speed and parallell access, which makes it expensive and not bing the right place for longer time archiving. Thus, for files which you're not actively working on, you should consider moving to the national storage on SweStore. This is how to do it.

Introduction

For accessing the Swestore national storage, we use (since Sep 2012) iRODS, and more specifically the iRODS client called "icommands", which is a set of commands similar to the unix commands "ls", "cd", "imkdir" etc, but with an "i" in front, plus two FTP-like commands: "iput" and "iget" (plus some more, but those are the ones you need most).

Note that data put on Swestore via iRODS at Uppmax is not accessible through other means.

Activate the iRODS icommands

The iRODS icommands client is activated through the module system, as so many other things on UPPMAX.

1. Log in to kalkyl
2. Execute:

module load irods/swestore

List files

After activating the iRODS icommands, you will be placed in one of your projects. if you run "ils", you will then see a listing of files and folders in that project, something like this:

[samuel@kalkyl4 ~]$ ils
  /ssUppnexZone/proj/b2011221:
    C- /ssUppnexZone/proj/b2011221/firstRun

Navigate around

To enter one of the (or the only) folder(s), do:

icd [proj-id]

... in this case:

icd firstrun

To switch to another project, use

icd ..

to back up a level. After that, you can change to another project or folder using icd as previously

icd b2011222

Upload files

To upload the above mentioned folder, do (-r is needed to recurse into directories):

iput -r [a local folder]

For single files, the -r flag is not needed:

iput [a local file]

To list the newly uploaded file/directory:

ils

To create folders, do:

imkdir [folder-name]

If you want to verify the upload outside of iRODS, there's a utility called ssverify.sh that will let you do that.

$ ssverify.sh test /ssUppnexZone/proj/p2061001/test
Checking if test/a matches /ssUppnexZone/proj/p2061001/test/a... seems good!
Checking if test/b matches /ssUppnexZone/proj/p2061001/test/b... seems good!
Checking if test/c matches /ssUppnexZone/proj/p2061001/test/c... seems good!
Checking if test/du matches /ssUppnexZone/proj/p2061001/test/du... seems good!
Checking if test/foo matches /ssUppnexZone/proj/p2061001/test/foo... seems good!
It seems test matches /ssUppnexZone/proj/p2061001/test on Swestore.
$  echo femtiofyra > test/c
$ ssverify.sh test /ssUppnexZone/proj/p2061001/test
Checking if test/a matches /ssUppnexZone/proj/p2061001/test/a... seems good!
Checking if test/b matches /ssUppnexZone/proj/p2061001/test/b... seems good!
Checking if test/c matches /ssUppnexZone/proj/p2061001/test/c... NO! Something is wrong! 
Either the files don't match or something went wrong, please contact 
staff@uppmax.uu.se to investigate.

Downloading files

To download a file again, do:

iget [a file in iRODS]

... or, for folders, do it recursively:

iget -r [a folder in iRODS]

More hints

• You can have iput show the progress with the -P flag
• If you want to be placed in a particular directory when you load the irods module, you can create a file called

  $HOME/.irods/.irodsEnv

  containing

  irodsHome '/ssUppnexZone/proj/myproj'
  irodsCwd '/ssUppnexZone/proj/myproj'

  where

  myproj

  is replaced by your project id.
• Read more info about the respective i-commands with "[command] -h"

Need help?

Mail support@uppmax.uu.se