Manuals for GEMS

[ Using the Client Tools ] [ Setting up Storage ] [ Setting up the Server ] [ GEMS API ]

This manual is intended to help users get started with the GEMS tools. GEMS users will typically only need to refer to the client section. Adding storage to an existing system requires reviewing the storage section, and creating a new system requires the server section.

The GEMS API docs are included for users wishing to access the GEMS classes directly in their application. This is particularly useful for Java and Beanshell developers who desire to gain full control of the GEMS clients. Additionally, a Java API is provided for Chirp.

Overview

Setup

Browsing GEMS

> java -jar GEMS.jar &

> GEMSview &

Main window

1. Type the name of your GEMS installation in the Server field.
2. The Account field allows you to specify an authentication mode when downloading files or creating new configs. Check the ACL on the config you wish to download to determine the mode you can use.
3. The left hand panel allows you to enter keys and values to match against configs in the system. Click match to obtain your results.
4. The Preview panel allows you to download all files in the config by clicking on the link. Config titles (use the title tag) are underlined.
5. Additional features below are available when entering the Details panel.
6. When in the Details panel, GEMSview lets you view the Params or the ACL for the config.
7. Right-clicking on the config key allows you to copy the number, which is useful for pasting to the command line.
8. The bottom right panel displays the file information for this config. Right-clicking on a file name allows you to preview this file, download the file, or download all the files from this config.
9. You may delete this config if you have write access by right-clicking on the owner name.
10. To simplify using the GEMSview tool, see the .gemsclient startup file info below.
11. You may put a new config into GEMS by clicking Put.
    • Params for the new config are obtained from the search boxes.
    • The owner for the config is obtained from the Account: box.
    • The storage map for the config is obtained from the current state of the Storage tool (see below).
    • The Access Control List (ACL) is obtained from the current state of the Access Control tool (see below).
    • First, select the path location under which all of the files for the new config reside.
    • Then, use Add and Drop to add files, then set their replica count.
    • Finally, hit Commit before closing the File Selection box, otherwise the Put will be cancelled.

Storage Map

Each cluster in the map has a nickname that you may specify. A cluster consists of a list of patterns. The cluster is made up of Chirp servers in the Chirp Catalog that match one of these patterns. Your data files will only be stored on these clusters. The GEMS Replicator will attempt to place one replica in each cluster before creating additional replicas. You may create stand-alone maps for future use by saving the map to a file. The Default setting is stored in your .gemsclient file.

Access Control

Each user consists of a mode and a name. A wildcard (*) may be used to match multiple users. (unix:* is useful to match all users on your filesystem.) Note that wildcard matching here is similar to the pattern matching in the Storage Map above. A user may have read permission for the config or admin privileges. Admin privileges allow the user to modify or delete the whole config. (In Chirp, read is rl and admin is lrwda.) The config owner and the GEMS system always have admin privileges. You may create stand-alone ACLs for future use by saving the ACL to a file. The Default setting is stored in your .gemsclient file.

Inserting into GEMS

Overview

Technique

• Command-line method:
  This method benefits users performing a one-time GEMSput, or that do not wish to program XML-generating routines to automate the process.
  1. Determine the directory under which all of your files for this config reside. Move to that directory. In the example below, the directory is ~/exp01.
  2. Determine the username that you use when contacting a Chirp system. In this example, this is unix:Sorin. This indicates that the user will be able to authenticate to all Chirp servers in the network by this name. See the Chirp documentation for more information about authentication methods in Chirp.
  3. Determine a set of metadata tags that you will be able to search for later. In this example, the scientist's name is Sorin, and the application is exp.
  4. Determine the files that will be stored and replicated by GEMS. In this case, there are two files, exp.in and exp.out. All files under the current directory may be inserted automatically as described below.
  5. Determine which users will be able to access your files: in this case, all UNIX users that can gain access to the Chirp network will be able to download the whole config. If you would prefer that only you may obtain access, omit this section.
  6. Determine which Chirp servers will be allowed to serve your data. If you do not want all servers in the GEMS installation to be able to serve your data, you must use the XML method or have a working .gemsclient file. Both options are described below.
  7. Type:

     ~exp01> GEMSput --owner unix:sorin
     scientist_lname=sorin app=exp count=1
     --file exp.in --file exp.out
     --acl-entry "unix:*" read
     --acl-entry "hostname:*.university.edu" read

     Note that JAR file users must run GEMSput as

     java -jar GEMS.jar put

     but the arguments remain the same.
  8. To obtain the list of options at any time just pass in the argument --help.
• XML method:
  This method benefits users that are running GEMS on a compute grid, and demonstrates the structure of a GEMS config.
  1. Build your GEMSput XML file. The example below corresponds to the exp01 config above.

     <GEMS> <GEMSput> <config owner="unix:sorin"> <acl> <entry principal="hostname:*.university1.edu" perms="lr" /> <entry principal="hostname:*.university2.edu" perms="lr" /> <entry principal="unix:*" perms="lrwda" /> </acl> <params> <scientist_lname>Sorin</scientist_lname> <app>exp</app> </params> <files> <file path="./" name="exp.in" type="exp input" reps="4" io="i" /> <file path="./" name="exp.out" type="exp output" reps="2" io="o" /> </files> <host>machine.university.edu</host> <storage name="two_universities"> <cluster name="university1"> <pattern>*.university1.edu</pattern> </cluster> <cluster name="university2"> <pattern>*.university2.edu</pattern> </cluster> </storage> </config> </GEMSput> </GEMS>

     As described above, the owner name is required, either here or on the command line. The ACL is specified using Chirp-formatted principals and permission strings. The given ACL indicates that users from sorin's unix file system have full access to this record, and users from two universities have read-only access. Params are entered, where the tag name indicates the param key, and the value is the tag value. A list of files is given: path and name attributes must be specified separately. The type is optional. The number of desired replicas is given as an integer. The input/output status is also optional, but the replication system favors input files, because it assumes that output files are recomputable. A preferred upload site may optionally given: if a Chirp server that is part of the GEMS system is on your local machine, specifying a host here may improve performance, because the data upload process will not need to use the network. <storage> indicates the Storage Map. This specifies which machines are eligible to store your data files. Additionally, the clustered structure of the map ensures that your data is spread over each cluster, and is not simply replicated inside any one cluster. This is very helpful in ensuring data survivability and improves performance when accessing data. name attributes are optional. Each cluster contains one or more patterns. Hosts in the Chirp catalog that match one of the patterns are included in that cluster.

GEMSput Options

Searching in GEMS

Overview

Technique

1. A simple match to obtain all of the config keys that scientist Sorin has submitted:

   > GEMSmatch scientist_lname=Sorin --keys
   141325

   This will simply print the config keys, one per line. Multiple param pairs may be put on the same command line to further restrict the result set.
2. If the user needs to retrieve the param and file information for a known config key in XML, the user may submit:

   > GEMSmatch --config 141325

   which produces the full XML.
3. Non-XML output may be obtained using the command line options below.
4. The above GEMS matches may also be obtained by the direct XML files below:

   <GEMS> <GEMSmatch> <params> <scientist_lname>Sorin</scientist_lname> <app>exp</app> <count>1</count> </params> </GEMSmatch> </GEMS>	Params are searched for as originally specified by the GEMSput XML.
   <GEMS> <GEMSmatch> <config key="24858230" /> </GEMSmatch> </GEMS>	As above, it is faster to simply search for the config key.

5. A typical GEMSmatch XML response is shown below:

   <GEMS> <GEMSmatch_Response> <configs> <config key="24858230" owner="unix:sorin"> <params> <scientist_lname>Sorin</scientist_lname> <app>exp</app> </params> <files> <file path="./" name="exp.in" type="exp input" reps="4" io="i"> <host>machine.university.edu</host> <host>host2.university.edu</host> </file> <file path="./" name="exp.out" type="exp output" reps="2" io="o" > <host>host1.university.edu</host> <host>host2.university.edu</host> </file> </files> </config> </configs> </GEMSmatch_Reponse> </GEMS>

   Note that multiple <config> elements may be returned under the <configs> element. Params are returned as originally specified by the GEMSput XML. Note that each file is now associated with multiple hosts, which will often vary from the original suggested <host> given in the GEMSput XML. These are the hosts from which this file may be retrieved. However, retrieving these files by hand is tedious, see the next tool, GEMSget, to automate this process. Non-XML formatted output may be obtained by using options below.

GEMSmatch Options

Retrieving data from GEMS

Overview

Technique

1. GEMSget users must already know the config key to use to access the config. If you do not know the config key, use GEMSmatch or GEMSview to find it.
2. If the config you want to download is 123, simply execute:

   ~/tmp> GEMSget --config 123.

   This will download all the directories and files in the config and place them under ~/tmp.
3. To download only certain files, indicate each file with --file. When this flag is used, unmentioned files will not be downloaded.

GEMSget options

Deleting data from GEMS

Overview

Technique

1. Deletes a config. Users must have the w permission. GEMSdelete users must already know the config key to use to access the config. If you do not know the config key, use GEMSmatch or GEMSview to find it.
2. If the config you want to download is 123, simply execute:

   > GEMSdelete --config 123.

   This will delete the whole record from the database, and the files in this config will be garbage collected.

GEMSdelete options

Using a .gemsclient file

Overview

<GEMS>
  <GEMSclient>

  <chirp>
   <user>unix:sorin</user> 
   <user>hostname:sorin.university.edu</user> 
  </chirp>

  <acl>
    <entry principal="*.university.edu" perms="lr" />
  </acl> 

   <storage name="two_universities">
    <cluster name="university1">
     <pattern>*.university1.edu</pattern>
    </cluster>
    <cluster name="university2">
     <pattern>*.university2.edu</pattern>
    </cluster>
   </storage>

  <servers>
   <host>gems.university.edu</host>
  </servers>

  <GEMSview>
   <keys>
    <scientist_lname> Sorin </scientist_lname>
   </keys>
  </GEMSview>

 </GEMSclient> 
</GEMS>

Overview

Setup

1. Requirements: You must be able to run the Chirp service, which is a small, relatively portable program that compiles with the gcc tools.
2. First, install and run the Chirp server.
3. Create space for GEMS by creating a directory that is known and accessible to the central GEMS services. The chirp server must report to the Chirp catalog used by GEMSd. Typical installations of GEMS use the /GEMS/ directory, and require hostname access. To accomplish this in chirp, run the server, and then use the client as below, assuming your server is on myhost.university.edu and your GEMSd service and catalog server are on gems.university.edu.

   > chirp_server -r /tmp/chirp -u gems.university.edu
   > chirp myhost.university.edu
   chirp:myhost:/> mkdir /GEMS
   chirp:myhost:/> setacl /GEMS hostname:gems.university.edu admin
   

   That's it!

Overview

Setup

1. Install PostgreSQL. An example installation process is shown below:

   ~/postgres-src# configure --prefix=/opt/pgsql
   ~/postgres-src# make ; make install
   ~/postgres-src# chmod a+rx /opt/pgsql
   ~/postgres-src# chmod a+rx /opt/pgsql/*
   ~/postgres-src# mkdir /opt/pgsql/data
   ~/postgres-src# useradd postgres
   ~/postgres-src# chown postgres /opt/pgsql/data
   ~/postgres-src# su - postgres
   > cd /opt/pgsql
   > bin/initdb data
   > bin/pg_ctl -D data start
   postmaster successfully started

2. Install the Chirp catalog. Ensure that your resource providers report to this catalog. Note that this service need not be run on the same machine that runs GEMSd.
3. Download the latest full GEMS tarball, GEMS-???-src.tar from the Downloads page.
4. Unpack the tarball, for example,

   > tar xf GEMS-???-src.tar

5. Build

   > ant -Ddist.dir=/opt/GEMS dist

where /opt/GEMS is an example of where you might like to install GEMS.
6. We recommend you run GEMS as its own user, named gems.

   # useradd gems

7. Make the GEMS directory world-readable, and let the gems user modify the configuration file:

   # chmod a+rx /opt/GEMS
   # chown gems /opt/GEMS/.gemsdconfig

8. Obtain a suitable PostgreSQL JDBC driver file. Place this file at /opt/GEMS/jdbc/postgresql.jdbc.jar.
9. Login as postgres, or a privileged database user:

   # su - postgres

10. Initialize the GEMSd database:

    > /opt/pgsql/bin/createdb GEMSd

11. Give gems permissions on the database:

    > /opt/pgsql/bin/psql GEMSd
    GEMSd=# create user gems;
    GEMSd=# grant all on database "GEMSd" to gems;
    GEMSd=# \q

12. Login as gems:

    # su - gems

13. Create the GEMSd database:

    > /opt/GEMS/bin/DBcreate

14. Start the server:

    > /opt/GEMS/bin/GEMSd -d -f /opt/GEMS/.gemsdconfig

GEMSd Options

GEMSd Configuration File

<GEMS> 
<GEMSadmin> 
  <Catalog> 
    <host>catalog.university.edu:9097</host> 
    <refresh>5</refresh> 
  </Catalog> 
  <ChirpRoot> 
      /GEMS/ 
  </ChirpRoot> 
  <Metadatabase>  
    <host>gems.university.edu</host> 
    <user>gems</user> 
    <password /> 
  </Metadatabase>  
  <GEMSprincipal> 
     hostname:gems.university.edu 
  </GEMSprincipal>  
  <Threads> 
      ... 
  </Threads> 
  <Groups> 
   <group name="A"> 
    <host name="*.A.university.edu" /> 
   </group>  
   <group name="BCd"> 
    <host name="*.B.university.edu" /> 
    <host name="*.C.university.edu" /> 
    <host name="d.D.university.edu" /> 
   </group> 
  </Groups> 
</GEMSadmin> 
</GEMS>