National Center for Computational Sciences

Access

Each user of an NCCS system is provided an account on the NCCS HPSS.

The HSI utility allows automatic authentication and provides a user-friendly command line and interactive interface to HPSS. HSI should be used to transfer data to and from the NCCS HPSS.

Storing and Retrieving Data

The hsi utility provides the ability to access the HPSS from the command line or through an interactive session with the HPSS.

From the HPSS

Establish an interactive session with the HPSS by typing hsi from the command line of a NCCS system. An interactive session is as close to logging into the HPSS as a user is allowed. From within an hsi interactive session users may perform many standard UNIX tasks. An interactive session provides users with the ability to create directories, remove files and directories, and transfer data.

Once connected, data can be transfered using:

• get

  Retrieve remote file and store it on the HPSS.

• mget

  Retrieve multiple remote file and store it on the HPSS.

• put

  Store a file on the HPSS onto the remote machine.

• mput

  Store multiple files on the HPSS onto the remote machine.

A full list of commands can be found by typing help from within an hsi interactive session.

From a NCCS system

HSI also allows users to pass commands into the hsi utility from the command line. This allows the ability to transfer data from the command line of a NCCS system. You can use

hsi [options] command(s)

to execute a set of HSI commands and then return.

For example the following command will place a.out in the calling user’s HPSS home directory:
> hsi put a.out

The following example will retrieve the file a.out from a HPSS project directory, /proj/projectid:
> hsi get /proj/projectid/a.out

From Offsite

Because HSI is a third-party package, clients may be available for your system. However, the NCCS currently supports access to the HPSS only through HSI clients on the NCCS HPC systems. To transfer data directly to or from the NCCS HPSS, you will need to use an NCCS resource as a staging system. For example, to transfer data from your directory on HPSS to a system outside the NCCS, you will need to copy the data in reasonable chunks to an NCCS system using the HSI utility. Once a portion of the data is on an NCCS system, you can use a utility such as BBCP or SFTP/SCP to move the data to the system outside the NCCS.

Transfer File Size

If you want to upload several small files (less than 1 GB per file), you should tar the files up before storing them in HPSS. Below is an example of storing and getting a bunch of files in a directory using tar and HSI. HSI can read from standard input and write to standard output.

  tar cvf - . | hsi put - : <filename.tar>

  hsi get - : <filename.tar> | tar xvf -

Ideal hsi and htar transfer limits:

File Size	Logins	Puts	Concurrent Sessions
2GB – 256GB	<500 a day	<500 a day	< 10

Setting Number of Copies

At the NCCS, we support up to two tape copies for files. By default users get 1 tape copy. For very critical files that have no backup elsewhere and cannot be easily re-created, you may want to store these files as two copy. You can specify two copies by issuing the “copies=2″ command before any put statements in hsi.

HSI non-interactive example

>hsi "copies=2; put test.file"
put  'test.file' : '/home/$USER/test.file' ( 56420696 bytes, 30566.2 KBS (cos=6003))

HSI interactive example

> copies=2
> put test.file
put  'test.file' : '/home/$USER/test.file' ( 56420696 bytes, 58627.9 KBS (cos=6003))

htar example

> htar -H copies=2 -cf test.tar .
HTAR: HTAR SUCCESSFUL

How do I verify my file is stored as 2 copy?

Home and Project Storage Areas

Each user has access to a home area and at least one project area on the HPSS.

Home Areas

Each user is provided a home directory on the HPSS. By default, files placed on the HPSS without providing a path will be stored in your home area.

For example:

hsi put a.out

place a.out in the users home directory, /home/userid/a.out.

hsi put a.out dir1/a.out

place a.out into the dir1 directory within the user’s home area, /home/userid/dir1/a.out.

hsi put a.out /proj/projectid/dir1/a.out

to access files outside of your home directory a path beginning with a leading / should be used.

Project Areas

Each project is provided a directory on the HPSS.

/proj/projectid

Each project area is available to all members of the project. It is requested that users place data pertaining to the project in the project directory. The directories are initially created with permissions set such that only project members may access, read, and write into the directory.

Accounting

Each file and directory is associated with an account. This associated account is used to determine storage of a user and project.

Available Accounts

Overhead Account

By default files and directories created in a user’s home area will be associated with the user’s overhead project. This project is named the same as your userid.

Project Accounts

By default files and directories created in a project area, /proj/projectid, will be associated with the project’s account. The project account is named the same as project’s projectid.

Legacy Account

Files stored on the system prior to March 15, 2008 are associated with a legacy project. The legacy project is used to record a file’s storage time period. Users are not able to associate new files to the legacy account.

Determine Available Accounts

The showproj utility can be used from any NCCS system to determine your available accounts. For example:

> showproj -s hpss

 userid is a member of the following project(s) on hpss:
  xxxyyy
  userid

>

Viewing File/Directory’s Associated Account

The hsi ls -UH can be used to view file and directory’s associated account.

For example:

:[/home/userid]: ls -UH
Mode     Links Owner     Group       COS       Acct   Where     Size    Date   Time  Entry
/home/userid:
drwxr-x---   3 userid    1099                 userid               512 Apr 01  2008 Backups
-rw-r-----   1 userid    1099       6001       legacy TAPE       4280320 Oct 24  2006 file.tar
-rw-r-----   1 userid    1099       6007       xxxyyy DISK    1956472248 Mar 20  2008 a.inp

In the above example,

the associated account for directory Backups is userid. By default all files created within the Backups directory will be associated with the userid account.

file.tar was created prior to March 15, 2008 and is therefore associated with the legacy account.

a.inp is associated with the xxxyyy project.

Modifying File/Directory’s Associated Account

A new file and directory will inherit the parent directory’s account association. This is the easiest and recommended method for managing data stored against an account.

However, users are able to manually change a file or directory’s associated account through the hsi newacct command.

The newacct syntax:
chacct [-R] newacct directory or file

For example:
chacct xxxyyy a.out
will set a.out’s associated account to xxxyyy.

chacct -R xxxyyy dir1
will set the associated account of dir1 and all files and directories within dir1 to xxxyyy.

Please Note:

The account identifier is case sensitive. You should use lower case when referring to the account.

When moving an existing file it will not inherit the parent directory’s account association.

Viewing Storage

The showusage utility may be used to view current storage associated with a user’s overhead project and other allocated projects for which the user is a member. The utility should be executed from a NCCS system; it may not be executed from within an hsi interactive session.

For example:

> showusage -s hpss

HPSS Storage in GB:
                                  Project Totals          userid
 Project                       Storage                    Storage
__________________________|__________________________|______________
 userid                   |       8.11               |       8.11
 legacy                   |                          |      25.67

>

Quotas

Space on the HPSS is for files that are not immediately needed. Users must not store files unrelated to their NCCS projects on HPSS. They must also periodically review their files and remove unneeded ones.