Helpdesk

Human Genetics provides computational services for bioinformatics. This comes in the form of Unix servers with bioinformatics software installed and configured in a cluster to handle the needs of our users. The number and specifications of the cluster varies (ie increases) over time, but as of this writing (2007-06-15) consists of 20 "nodes" of dual processor computers of 2.4 - 2.8 GHz each CPU, for a total of 96 effective GHz.

Please note that some nodes are available only through Grid Engine See below for details on how to use the Grid Engine.

NOTE! Due to problematic interactions between OpenMosix kernels and threading used by perl/python/java, you may experience strange and random segfaults when running programs written in these languages. Please use the -g option to bgrun for such programs to prevent them from running on OpenMosix nodes.

While there may be many and variable numbers of nodes available, users connect to only one of them, called the "head node". This head node is named calypso and has the full hostname calypso.genetics.ucla.edu. This is the server to which users log in and run software.

How the cluster works

Two clustering technologies are employed by the Human Genetics cluster. They each have their benefits and while they function well together, they are separate entities and the in and outs of each should be kept in mind.

openMosix

The easiest clustering technology to use is called openMosix. You don't have to do anything special to use it as it is completely transparent. Just run your programs or "jobs" as if no clustering were involved and openMosix will automatically transfer and run it on the best possible node. In fact, it may move your job again across nodes if another one becomes available and it might run faster there.

As openMosix is transparent to the user, your jobs can be monitored and managed via the usual Unix tools. For example, the process list command ps will list your processes as if they were running on calypso, even if they are actually running on another node. Likewise top will monitor running processes.

In addition to the regular Unix tools, openMosix includes a few extra tools that provide additional details and insight into the functioning of the openMosix cluster. mtop is a replacement for top that is openMosix-aware: in addition to the normal information, it also lists on which node a job is running, and how many times it has moved from node to node. mtop should be used on our cluster instead of top for this reason.

mosmon is another tool and it provides a pseudo-graphical monitoring display of cluster load, using a histogram. With it, you can see a visual representation of how many openMosix nodes are available and the relative load on each of them. Enter mosmon -h to see a detailed description of what mosmon can do as well as a list of keyboard commands available to alter the display while it is running.

One caveat with openMosix cluster is that not all jobs will migrate to different nodes and instead will "stick" on the head node. While this is unusual, it is still something to be mindful of as it would be ill-advised to run 10 jobs of this type since they will swamp the head node rather than distributing across the cluster. Whether this will happen depends on the program you're running and the specifics of a particular job, so it's hard to know in advance if this will be the case - please monitor using the above tools (mtop is a good choice) to see that your jobs are distributing properly.

The other caveat with the openMosix cluster is that not all cluster nodes are openMosix members - when using openMosix as the way to distribute jobs, only a subset of computers are available to you. To gain access to the full suite of cluster nodes, you must use the other clustering technology - the grid engine.

Grid Engine

The other clustering technology employed by our cluster is provided by the N1 Grid Engine. Unlike openMosix, the grid engine is not transparent and you must execute special commands in order to utilize the grid engine cluster. Also unlike openMosix, the grid engine doesn't run all jobs it receives simultaneously - rather it runs as many jobs as there are CPUs available and "queues" the others until a CPU becomes available. As such, the grid engine is more "friendly" to all users in that one user cannot swamp the cluster and in practice, the time to complete jobs run serially is not significantly different than if run simultaneously.

To utilize the grid engine, you must "submit" jobs to the grid engine queues. This is most easily done through the bgrun command - usage details for this command are available here

Monitoring jobs in the grid engine system requires the use of some new commands. The most useful are qhost, which lists all members of the grid engine cluster and their current loads; qstat which lists details on the current queue (which jobs are queued or running and where they are running); and qdel which allows you to remove unwanted jobs from the queue (you can only remove your own jobs, of course).

While using the grid engine does involve the use of an extra command, in practice, this should be as easy to use as with openMosix - simply preface your usual command with "bgrun" (eg bgrun mendel instead of just mendel). The benefits of using the grid engine is that unlike openMosix, you will have full access to all members of the cluster and that it is able to better distribute jobs in an efficient manner. Not only will all jobs distribute under the grid engine, it does so in a more load-balanced manner. Furthermore, with the Grid Engine, you have a better chance of not losing your job if the head node goes down.