How to use the SP/2

Table of Contents

The SP/2

The SP/2 consists of a dedicated control workstation and several parallel nodes, 16 to be exact. The nodes are interconnected via IBM's high-speed switch. The nodes and the control workstation are also connected via a conventional ethernet.

Note that the control workstation is NOT connected to the high speed switch!

Node Names

The entire SP/2 is named "sp2". However, you won't find a host named "sp2". Instead, each individual computer which comprises the SP/2 has it's own suffix to the "sp2" root. The control workstation's name is "sp2-cw". The nodes are numbered 1 .. 16, and each of their hostnames is of the form "sp2-%02d", such as sp2-01, sp2-16, etc.

But wait ... there's more

The above hostnames are the names of the ethernet interfaces on each node. The individual nodes also have IP access to the switched network. If you want to run IP across the switch between nodes, you have to use a different hostname, one that tells each node to use it's switch interface. Again, it's another suffix on the "sp2" name. In this case, the suffix is "-n%02d", and the corresponding hostnames are "sp2-n01", "sp2-n11", ...

Whenever I use the term "sp2", I'm either refering to the entire machine, or just the control workstation. Context should make it easy to determine which is intended.

Logging into the SP/2

A Review of Authentication Databases

There are currently several methods for users in the department to be authenticated as themselves.
  1. Departmental Kerberos-5 Database
    On most machines in the department, you are authenticated with the Kerberos-5 database, which also generates an AFS token for you.
  2. AFS Kerberos Database
    Some machines in the deparment use AFS, but don't have support for Kerberos-5. On these systems, AFS may or may not be integrated with the login program. If it is, you login to the system using your AFS password. If not, you need to login to the system using your /etc/passwd password, and then authenticate to AFS with klog and your AFS password.
  3. /etc/passwd
    The tried and true authentication mechanism which only authenticates you to the current host.
Each of these databases can have a different password!

What's Different about the SP/2?

The SP/2 adds an additional authentication database to this mess, the SP/2 Kerberos Database. This database is used when parallel programs are executed on the SP/2.

Logging in

The SP/2 doesn't have kerberos support for the departmental Kerberos-5 database. However, it does does support the AFS Kerberos database. And, AFS is integrated with with SP/2's login program.

So, to login to the SP/2, use your AFS password. If you don't have one, you should visit bolo to set your AFS password.

The ONLY SP/2 node that you can login to is the control workstation! In a bizzare coincidence :-) AFS is only available on the control workstation.

To use the SP/2's parallel facilities (the nodes), you will need to authenticate yourself with the SP/2's Kerberos database. See bolo to set your SP/2 kerberos password. The SP/2 kerberos tools are in /usr/kerberos/bin. Use

Because these kerberos tickets aren't destroyed upon logout, please use kdestroy when you are finished using any parallel tools (aka when you logout).

Using the SP/2

Files

The sp2 has access to all of AFS land. However, the SP/2 already uses the /u directory for its own purposes. Becuase of this, you need to refer to your AFS home directories as /afs/cs.wisc.edu/u/w/h/whoever. The SP/2 uses /u as a way to share user's home directories on the control workstation with the nodes. This is done with NFS. Anything you store in ~whoever (/u/whoever) on the control workstation will be available as ~whoever (/u/whoever) on each node.

Paths

Several tools are needed to compile and use shore on the SP/2. To access these tools add the following directories to your path.

To aid use of SP/2 Kerberos and AFS, you may add the following directories to your path.

However, Beware! These directories contain some executables with identical names. If want to, for instance, change your password, use an absolute pathname. I'd recommend putting the SP/2 kerberos directory before the AFS directory in your path.

In addition, the IBM SP/2 parallel tools themself are in yet another directory

pexec

There is an entire suite of parallel tools to use with the SP/2. I find pexec to be the most useful one for the things I typically do. It allows you to execute normal unix programs, unencumbered by any SP/2-isms, on one or more of the SP/2 nodes.

Pexec allows you to execute something on the nodes ... in parallel! The general form of a pexec command is 

pexec nodes command_to_execute -switches arguments ...
"Nodes" is really a nodespec, which may be a
  1. node number
  2. comma-seperated list of nodespecs
  3. range of node numbers, N-M

Pexec will execute the commands on each node. When all the commands are finished, pexec will "combine" the output of any nodes which have the same output, and then present you the output... 

sp2-cw(bolo) pexec 1-2 uname -a
HOSTS ------------------------------------
sp2-01                                                                      
------------------------------------------
AIX sp2-01 1 4 000090535900

HOSTS ------------------------------------
sp2-02                                                                      
------------------------------------------
AIX sp2-02 1 4 000096065900
Here is an example which demonstrates the output "combining": 
sp2-cw(bolo) pexec 1-5 echo hi
HOSTS ----------------------------------------------------------
sp2-01             sp2-02             sp2-03             sp2-04             
sp2-05                                                                      
----------------------------------------------------------------
hi

Poe

Poe is IBM's Parallel Operating Environment. You have to use Poe if you intend to use MPI, the Message Passing Interface.

However, Poe doesn't allow use of standard unix tools because it consumes resources which ordinary Unix applications may need to run. Examples of this are signals and interval timers. Many applications can be used without modification in this environment, but Shore is not one of them. As a sweeping statement, traditional computationally intensive codes should work without modification. Systems-heavy code, such as Shore will not.

SP/2 Information
Bolo's Home Page
Last Modified: Mon Dec 18 13:06:55 CST 1995
bolo (Josef Burger) <bolo@cs.wisc.edu>