How do I move files to and from Swestore using iRODS?

Please note; the Swestore over iRODS service will be sunsetting during 2016. We strongly discourage you from uploading more data.

The main storage at UPPNEX is optimized for high-speed and parallell access, which makes it expensive and not the right place for longer time archiving. Thus, for files which you're not actively working on, you can consider moving them to other storage, such as the Swestore/dCache backed iRODS service provided by UPPNEX.

Please note that there are no guarantees of continued service or even a grace period in the event of service termination. Because of thist the Swestore backed iRODS service is not suitable for long time storage of data. We strongly recommend you make sure to have other copies of any data you upload to Swestore through iRODS.

Introduction

For accessing the Swestore national storage, we use (since Sep 2012) iRODS, and more specifically the iRODS client called "icommands", which is a set of commands similar to the unix commands "ls", "cd", "imkdir" etc, but with an "i" in front, plus two FTP-like commands: "iput" and "iget" (plus some more, but those are the ones you need most).

Note that data put on Swestore via iRODS at Uppmax is not accessible through other means.

Activate the iRODS icommands

The iRODS icommands client is activated through the module system, as so many other things on UPPMAX.

1. Log in to milou
2. Execute:

module load irods/swestore

List files

After activating the iRODS icommands, you will be placed in one of your projects. if you run "ils", you will then see a listing of files and folders in that project, something like this:

[samuel@milou1 ~]$ ils
/ssUppnexZone/proj/b2011221:
C- /ssUppnexZone/proj/b2011221/firstRun

Navigate around

To enter one of the (or the only) folder(s), do:

icd [proj-id]

... in this case:

icd firstrun

To switch to another project, use

icd ..

to back up a level. After that, you can change to another project or folder using icd as previously

icd b2011222

Upload files

To upload the above mentioned folder, do (-r is needed to recurse into directories):

iput -r [a local folder]

For single files, the -r flag is not needed:

iput [a local file]

To list the newly uploaded file/directory:

ils

To create folders, do:

imkdir [folder-name]

If you want to verify the upload outside of iRODS, there's a utility called ssverify.sh that will let you do that.

$ ssverify.sh test /ssUppnexZone/proj/p2061001/test
Checking if test/a matches /ssUppnexZone/proj/p2061001/test/a... seems good!
Checking if test/b matches /ssUppnexZone/proj/p2061001/test/b... seems good!
Checking if test/c matches /ssUppnexZone/proj/p2061001/test/c... seems good!
Checking if test/du matches /ssUppnexZone/proj/p2061001/test/du... seems good!
Checking if test/foo matches /ssUppnexZone/proj/p2061001/test/foo... seems good!
It seems test matches /ssUppnexZone/proj/p2061001/test on Swestore.
$  echo femtiofyra > test/c
$ ssverify.sh test /ssUppnexZone/proj/p2061001/test
Checking if test/a matches /ssUppnexZone/proj/p2061001/test/a... seems good!
Checking if test/b matches /ssUppnexZone/proj/p2061001/test/b... seems good!
Checking if test/c matches /ssUppnexZone/proj/p2061001/test/c... NO! Something is wrong! 
Either the files don't match or something went wrong, please contact 
staff@uppmax.uu.se to investigate.

There is also a command

irsync

that behaves in a similar way to regular rsync, which is recommended for moving large/deep data structures.

Downloading files

To download a file again, do:

iget [a file in iRODS]

... or, for folders, do it recursively:

iget -r [a folder in iRODS]

File removal

To remove a file, you'll like need to use "irm -f". (If you try without the "-f" flag then it will not work at UPPMAX, because we do not currently support the use of a trash area.)

Accessing Swestore directly

If you for some reason want to be able to access your files in Swestore directly, without having to login to UPPMAX first, then you can contact us for a separate password and then install a client like e.g. iDrop on your personal computer. Note that this requires that you are connected to SUNET.

More hints

  • You can have iput show the progress with the -P flag
  • If you want to be placed in a particular directory when you load the irods module, you can create a file called 
    $HOME/.irods/.irodsEnv
    containing 
    irodsHome '/ssUppnexZone/proj/myproj'
irodsCwd '/ssUppnexZone/proj/myproj'
    where 
    myproj
    is replaced by your project id.
  • Read more info about the respective i-commands with "[command] -h"

What does the error UNIV_MSS_UNLINK_ERR mean?

This you usually get if you've tried to use "irm" without the flag "-f". If you do not use "-f" it will try to move the file to the trash area, which we do not support. So don't forget this flag!

If you get this error even when you're using the "-f" flag then please whether or not the file still exists after a while. It could be a temporary problem, and the file might actually be removed afterwards anyway.

What does the error UNIV_MSS_STAGETOCACHE_ERR mean?

This usually means that our local cache layer could not reach the backend (which is located elsewhere in Sweden). Our caching servers will try to process the request (upload/download) later when the connection between us and the backend is restored again.

If you tried to upload with e.g. iput then you should after a couple of hours-days verifity that our caching servers successfully uploaded the data. You do this with e.g. the script ssverify.sh.

If you tried to download with iget then our servers will try to download the data to our local cache, but the file(s) will not be copied to your project folders. So after a couple of hours-days you should rerun your download. If our servers had restored the connection then the download to the project folders will be much quicker this time, because the file will be copied internally within UPPMAX instead of being downloaded from elsewhere in Sweden.

Need help?

Mail support@uppmax.uu.se

Skriv ut