The University of Sheffield
Sheffield-WRGRID

  

File Storage on Iceberg

If you exceed your file storage allocation

As soon as the quota is exceeded your account becomes frozen. In order to avoid this situation it is strongly recommended that you -

 If your account becomes frozen  you can remove unwanted files by using the RM command (note upper case) and you will be automatically unfrozen.

Alternatively, if you have larger file storage requirements please request it by emailing  hpchub@sheffield.ac.uk .  

Efficiency considerations

For jobs requiring a lot of I/O, it may sometimes be necessary to store copies of the data on the actual compute node on which your job is running. For this, you can create temporary areas of storage under the directory /scratch . The next best I/O performance that requires minimum amount of work is achieved by keeping your data in /fastdata area and running your jobs on the new intel nodes ( by specifying -l arch=intel* ) .

These methods provide much faster access to data than the network attached storage on either /home or /data areas, but you must remember to copy important data back onto your /home area. 

If you decide to use the /scratch area we recommend that under /scratch you create a directory with the same name as your username and work under that directory to avoid possibility of clashing with other users.

The /scratch area is local to each worker node and is not visible to the other worker nodes or to the head-nodes. Therefore any data created by jobs should be transfered to your either /data or /home area before the job finishes. Here is an example script that will copy filex from your /home area into scratch and copy filey from scratch area to your home directory;

 

mkdir   /scratch/${USER}
cp   ${HOME}/filex    /scratch/${USER}
#  run your program now that uses filex as input and produces filey as its output
cp   /scratch/${USER}/filey    ${HOME}

 

 Anything under the /scratch is deleted periodically when the worker-node is idle, whereas files on the /fastdata area will be deleted only when they are 3 months old.