A Minimal Know How - HPC Computing Facility

Yes.

We have the full GNU tool-chain available on both the login nodes so normal compilation tools such as autoconf, automake, libtool, make, gcc, g++, gfortran, gdb, ddd, java, python, perl, etc are available to you. Please let us know if there are other tools or libraries you need that aren’t available.

Use gnu-tool chain.

Adding a package is similar to keeping the executation or your own program. If you keep any executable in your area, it will be accessible in all computenode.

The detail is given here.

HPC disk storage is limited and an user quota has been set for all user spaces (/home). All users are given 10GB of user space. There is no notification system to inform you if you exceed the quota limit. When you exced the quota limit you won’t be able to perform write operation i.e. can’t create file or run jobs if you pass the hard quota limit.

HPC has about 7TB of storage on to be shared among IISER HPCs users, and the instantaneous needs of those users varies tremendously. We do not use disk quotas on this storage to enforce user limits to allow substantial dynamic storage use. However, if you use hundreds of GB, the onus is on you to clean up your files and decrease that usage as soon as you’re done with it.

If your code accessing large I/O, you should use local scratch space. This is to prevent this network jam, there is a large /tmp directory on each node (writable by all users, but sticky - files written can only be deleted by the user who wrote them or by admin. However, if you use hundreds of GB, the onus is on you to clean up your files and decrease that usage as soon as you’re done with it. (automatic script as regular cleanup will be added if needed).

Use quota -s command to see your ussage.

This means:

Our cluster is configured with SGE Batch Job Submission tool. The Sun Grid Engine (SGE) queuing system is useful when you have a lot of tasks to execute and want to distribute the tasks over a cluster of machines. It has three basic features

If you would like to know more, you may prefer to check a plenty of documents available in the web. Let me start with what minimal set of information (mostly taken from web from several sources)

A job in SGE represents a task to be performed on a node in the cluster and contains the command line used to start the task. A job may have specific resource requirements but in general should be agnostic to which node in the cluster it runs on as long as its resource requirements are met.

Submitting jobs is done using the qsub command. Let’s try submitting a simple job that runs the hostname command on a given cluster node:

Notice that the qsub command, when successful, will print the job number to stdout. You can use the job number to monitor the job’s status and progress within the queue as we’ll see in the next section.

Now that our job has been submitted, let’s take a look at the job’s status in the queue using the command qstat:

From this output, we can see that the job is in the r state which is running in node compute-0-0. Once the job has finished, the job will be removed from the queue and will no longer appear in the output of qstat. You should see the job outputs

SGE creates stdout and stderr files in the job’s working directory for each job executed. If any additional files are created during a job’s execution, they will also be located in the job’s working directory unless explicitly saved elsewhere (I will discuss later).The job’s stdout and stderr files are named after the job with the extension ending in the job’s number. For the simple job submitted above, we have:

Notice that SGE automatically named the job hostname and created two output files: hostname.e4001 and hostname.o4001. The e stands for stderr and the o for stdout. The 4001 at the end of the files’ extension is the job number. So if the job had been named extraordinary_job and was job #47 submitted, the output files would look like: extraordinary_job.e47 extraordinary_job.o47

What if a job is stuck in the queue, is taking too long to run, or was simply started with incorrect parameters? You can delete a job from the queue using the qdel command in SGE. Below we launch a simple job mpi-ring.qsub, and we can kill it using qdel:

Check the Job status

qw means Job is in waiting stage. you can an kill it here, but we want job to run and then kill. Checking again after few second:

r means Job has started. Send a kill signal by qdel jobid and check the status.

After running qdel you’ll notice the job is gone from the queue: qstat returns nothing, i.e. qdel has killed the job.

SGE uses qstat to check the job status. I have submitted 10 jobs, which need 10 core each. I would type qstat and it will show me everything

You can also view the average load (load_avg) per node using the ‘-f’ option to qstat:

In the ‘Submitting a Job’ section we submitted a single command hostname. This is useful for simple jobs but for more complex jobs where we need to incorporate some logic we can use a so-called job script. A job script is essentially a bash script that contains some logic and executes any number of external programs/scripts. The shell script that you submit (for example job_name.sh) should be written in bash and should completely describe the job, including where the inputs and outputs are to be written (if not specified, the default is your home directory). The following is a simple shell script that defines bash as the job environment, calls date, waits 20s and then calls it again.

Note that your script has to include (usually at the end) at least one line that executes something - generally a compiled program but it could also be a Perl or Python script (which could also invoke a number of other programs). Otherwise your SGE job won’t do anything.

And to submit above script job_name.sh, you would do:

HPC depends on a network-shared /data filesystem. The actual disks are on a network file server node so users are local to the data when they log in. However, when you submit an SGE job, unless otherwise specified, the nodes have to read the data over the network and write it back across the network. This is fine when the total data involved is a few MB, such as is often the case with molecular dynamics runs - small data in, lots of computation, small data out. However, if your jobs involve 100s or 1000s of MB, the network traffic can grind the entire cluster to a halt.

To prevent this network jam, there is a large /tmp directory on each node (writable by all users, but sticky - files written can only be deleted by the user who wrote them or by admin. However, if you use hundreds of GB, the onus is on you to clean up your files and decrease that usage as soon as you’re done with it. (automatic script as regular cleanup will be added if needed).

following is an example script (self explanatory)

In this example all data output will be stored in $HOME/test/output and this is expected to be small output. If it is large output, you must redirect it to the NFS data-space.

Similarly, you can write scripts for parallel jobs.

StarCluster by default sets up a parallel environment, called orte, that has been configured for OpenMPI integration within SGE and has a number of slots equal to the total number of processors in the cluster. You can inspect the SGE parallel environment by running:

The general workflow for running MPI code is: Compile the code using mpi compilers like mpicc.. The produced executable can be used in parallel environment of SGE. It is important that the path to the executable is identical on all nodes for mpirun to correctly launch your parallel code. The easiest approach is to copy the executable somewhere under /home/user on the usernode since /home/user is NFS-shared across all nodes in the cluster.

where the hostfile looks something like:

However, when using an SGE parallel environment with OpenMPI you no longer have to specify the -np, -hostfile, -host, etc. options to mpirun. This is because SGE will automatically assign hosts and processors to be used by OpenMPI for your job. You also do not need to pass the -byslot and -bynode options to mpirun given that these mechanisms are now handled by the fill_up and round_robin modes specified in the SGE parallel environment.

Instead of using the above formulation create a simple job script that contains a very simplified mpirun call:

Then submit the job using the qsub command and the orte parallel environment automatically configured for you by StarCluster:

The -pe option species which parallel environment to use and how many slots to request. The above example requests 24 slots (or processors) using the orte parallel environment. The parallel environment automatically takes care of distributing the MPI job amongst the SGE nodes using the allocation_rule defined in the environment’s settings.