Manuals for GEMS

[ Using the Client Tools ] [ Setting up Storage ] [ Setting up the Server ] [ GEMS API ]

This manual is intended to help users get started with the GEMS tools. GEMS users will typically only need to refer to the client section. Adding storage to an existing system requires reviewing the storage section, and creating a new system requires the server section.

The GEMS API docs are included for users wishing to access the GEMS classes directly in their application. This is particularly useful for Java and Beanshell developers who desire to gain full control of the GEMS clients. Additionally, a Java API is provided for Chirp.

Overview

Setup

Browsing GEMS

> java -jar GEMS.jar &

> GEMSview &

Main window

1. Type the name of your GEMS installation in the Server field.
2. The Account field allows you to specify an authentication mode when downloading files or creating new configs. Check the ACL on the config you wish to download to determine the mode you can use.
3. The left hand panel allows you to enter keys and values to match against configs in the system. Click match to obtain your results.
4. The Preview panel allows you to download all files in the config by clicking on the link. Config titles (use the title tag) are underlined.
5. Additional features below are available when entering the Details panel.
6. When in the Details panel, GEMSview lets you view the Params or the ACL for the config.
7. Right-clicking on the config key allows you to copy the number, which is useful for pasting to the command line.
8. The bottom right panel displays the file information for this config. Right-clicking on a file name allows you to preview this file, download the file, or download all the files from this config.
9. You may delete this config if you have write access by right-clicking on the owner name.
10. To simplify using the GEMSview tool, see the .gemsclient startup file info below.
11. You may put a new config into GEMS by clicking Put.
    • Params for the new config are obtained from the search boxes.
    • The owner for the config is obtained from the Account: box.
    • The storage map for the config is obtained from the current state of the Storage tool (see below).
    • The Access Control List (ACL) is obtained from the current state of the Access Control tool (see below).
    • First, select the path location under which all of the files for the new config reside.
    • Then, use Add and Drop to add files, then set their replica count.
    • Finally, hit Commit before closing the File Selection box, otherwise the Put will be cancelled.

Storage Map

Each cluster in the map has a nickname that you may specify. A cluster consists of a list of patterns. The cluster is made up of Chirp servers in the Chirp Catalog that match one of these patterns. Your data files will only be stored on these clusters. The GEMS Replicator will attempt to place one replica in each cluster before creating additional replicas. You may create stand-alone maps for future use by saving the map to a file. The Default setting is stored in your .gemsclient file.

Access Control

Each user consists of a mode and a name. A wildcard (*) may be used to match multiple users. (unix:* is useful to match all users on your filesystem.) Note that wildcard matching here is similar to the pattern matching in the Storage Map above. A user may have read permission for the config or admin privileges. Admin privileges allow the user to modify or delete the whole config. (In Chirp, read is rl and admin is lrwda.) The config owner and the GEMS system always have admin privileges. You may create stand-alone ACLs for future use by saving the ACL to a file. The Default setting is stored in your .gemsclient file.

Inserting into GEMS

Overview

Technique

1. Determine the directory under which all of your files for this config reside. Move to that directory. In the example below, the directory is ~/exp01.
2. Determine the username that you use when contacting a Chirp system. In this example, this is unix:Sorin. This indicates that the user will be able to authenticate to all eligible Chirp servers by this name. See the Chirp documentation for more information about authentication methods in Chirp.
3. Determine a set of metadata tags that you will be able to search for later. In this example, the scientist's name is Sorin, and the application is exp.
4. Determine the files that will be stored and replicated by GEMS. In this case, there are two files, exp.in and exp.out. All files under the current directory may be inserted automatically as described below.
5. Determine which users will be able to access your files: in this case, all UNIX users that can gain access to the Chirp network will be able to download the whole config. If you would prefer that only you may obtain access, omit this section.
6. Determine which Chirp servers will be allowed to serve your data. You may construct a storage map and use the --map option to specify the eligible servers.
7. Type:

   ~exp01> GEMSput --owner unix:sorin
   scientist_lname=sorin app=exp count=1
   --file exp.in --file exp.out
   --acl-entry "unix:*" read
   --acl-entry "hostname:*.university.edu" read

   Note that JAR file users must run GEMSput as

   java -jar GEMS.jar put

   but the arguments remain the same.
8. To obtain the list of options at any time just pass in the argument --help.

GEMSput Options

Searching in GEMS

Overview

Technique

1. A simple match to obtain all of the config keys that scientist Sorin has submitted:

   > GEMSmatch scientist_lname=Sorin --keys
   141325

   This will simply print the config keys, one per line. Multiple param pairs may be put on the same command line to further restrict the result set.
2. If the user needs to retrieve the param and file information for a known config key in XML, the user may submit:

   > GEMSmatch --config 141325

   which produces the full XML.
3. Non-XML output may be obtained using the command line options below.
4. A typical GEMSmatch XML response is shown below:

   <GEMS> <GEMSmatch_Response> <configs> <config key="24858230" owner="unix:sorin"> <params> <scientist_lname>Sorin</scientist_lname> <app>exp</app> </params> <files> <file path="./" name="exp.in" type="exp input" reps="4" io="i"> <host>machine.university.edu</host> <host>host2.university.edu</host> </file> <file path="./" name="exp.out" type="exp output" reps="2" io="o" > <host>host1.university.edu</host> <host>host2.university.edu</host> </file> </files> </config> </configs> </GEMSmatch_Reponse> </GEMS>

   Note that multiple <config> elements may be returned under the <configs> element. Params are returned as originally specified by GEMSput. Note that each file is now associated with multiple hosts, which will often vary from the original suggested <host> given to GEMSput. These are the hosts from which this file may be retrieved. However, retrieving these files by hand is tedious, see the next tool, GEMSget, to automate this process. Non-XML formatted output may be obtained by using options below.

GEMSmatch Options

Retrieving data from GEMS

Overview

Technique

1. GEMSget users must already know the config key to use to access the config. If you do not know the config key, use GEMSmatch or GEMSview to find it.
2. If the config you want to download is 123, simply execute:

   ~/tmp> GEMSget --config 123.

   This will download all the directories and files in the config and place them under ~/tmp.
3. To download only certain files, indicate each file with --file. When this flag is used, unmentioned files will not be downloaded.

GEMSget options

Deleting data from GEMS

Overview

Technique

1. Deletes a config. Users must have the w permission. GEMSdelete users must already know the config key to use to access the config. If you do not know the config key, use GEMSmatch or GEMSview to find it.
2. If the config you want to download is 123, simply execute:

   > GEMSdelete --config 123.

   This will delete the whole record from the database, and the files in this config will be garbage collected.

GEMSdelete options

Using a .gemsclient file

Overview

<GEMS>
  <GEMSclient>

  <chirp>
   <user>unix:sorin</user> 
   <user>hostname:sorin.university.edu</user> 
  </chirp>

  <acl>
    <entry principal="*.university.edu" perms="lr" />
  </acl> 

   <storage name="two_universities">
    <cluster name="university1">
     <pattern>*.university1.edu</pattern>
    </cluster>
    <cluster name="university2">
     <pattern>*.university2.edu</pattern>
    </cluster>
   </storage>

  <servers>
   <host>gems.university.edu</host>
  </servers>

  <GEMSview>
   <keys>
    <scientist_lname> Sorin </scientist_lname>
   </keys>
  </GEMSview>

 </GEMSclient> 
</GEMS>

Overview

Setup

1. Requirements: You must be able to run the Chirp service, which is a small, relatively portable program that compiles with the gcc tools.
2. First, install and run the Chirp server.
3. Create space for GEMS by creating a directory that is known and accessible to the central GEMS services. The chirp server must report to the Chirp catalog used by GEMSd. Typical installations of GEMS use the /GEMS/ directory, and require hostname access. To accomplish this in chirp, run the server, and then use the client as below, assuming your server is on myhost.university.edu and your GEMSd service and catalog server are on gems.university.edu.

   > chirp_server -r /tmp/chirp -u gems.university.edu
   > chirp myhost.university.edu
   chirp:myhost:/> mkdir /GEMS
   chirp:myhost:/> setacl /GEMS hostname:gems.university.edu admin
   

   That's it!

Overview

Setup

1. Install PostgreSQL. An example installation process is shown below:

   ~/postgres-src# configure --prefix=/opt/pgsql
   ~/postgres-src# make ; make install
   ~/postgres-src# chmod a+rx /opt/pgsql
   ~/postgres-src# chmod a+rx /opt/pgsql/*
   ~/postgres-src# mkdir /opt/pgsql/data
   ~/postgres-src# useradd postgres
   ~/postgres-src# chown postgres /opt/pgsql/data
   ~/postgres-src# su - postgres
   > cd /opt/pgsql
   > bin/initdb data
   > bin/pg_ctl -D data start
   postmaster successfully started

2. Install the Chirp catalog. Ensure that your resource providers report to this catalog. Note that this service need not be run on the same machine that runs GEMSd.
3. Download the latest full GEMS tarball, GEMS-???-src.tar from the Downloads page.
4. Unpack the tarball, for example,

   > tar xf GEMS-???-src.tar

5. Build

   > ant -Ddist.dir=/opt/GEMS dist

where /opt/GEMS is an example of where you might like to install GEMS.
6. We recommend you run GEMS as its own user, named gems.

   # useradd gems

7. Make the GEMS directory world-readable, and let the gems user modify the configuration file:

   # chmod a+rx /opt/GEMS
   # chown gems /opt/GEMS/.gemsdconfig

8. Obtain a suitable PostgreSQL JDBC driver file. Place this file at /opt/GEMS/jdbc/postgresql.jdbc.jar.
9. Login as postgres, or a privileged database user:

   # su - postgres

10. Initialize the GEMSd database:

    > /opt/pgsql/bin/createdb GEMSd

11. Give gems permissions on the database:

    > /opt/pgsql/bin/psql GEMSd
    GEMSd=# create user gems;
    GEMSd=# grant all on database "GEMSd" to gems;
    GEMSd=# \q

12. Login as gems:

    # su - gems

13. Create the GEMSd database:

    > /opt/GEMS/bin/DBcreate

14. Start the server:

    > /opt/GEMS/bin/GEMSd -d -f /opt/GEMS/.gemsdconfig

15. To stop the server:

    > /opt/GEMS/bin/GEMSadmin --stop

GEMSd Options

GEMSd Configuration File

<GEMS> 
<GEMSadmin> 
  <Catalog> 
    <host>catalog.university.edu:9097</host> 
    <refresh>5</refresh> 
  </Catalog> 
  <ChirpRoot> 
      /GEMS/ 
  </ChirpRoot> 
  <Metadatabase>  
    <host>gems.university.edu</host> 
    <user>gems</user> 
    <password /> 
  </Metadatabase>  
  <GEMSprincipal> 
     hostname:gems.university.edu 
  </GEMSprincipal>  
  <Threads> 
      ... 
  </Threads> 
  <Groups> 
   <group name="A"> 
    <host name="*.A.university.edu" /> 
   </group>  
   <group name="BCd"> 
    <host name="*.B.university.edu" /> 
    <host name="*.C.university.edu" /> 
    <host name="d.D.university.edu" /> 
   </group> 
  </Groups> 
</GEMSadmin> 
</GEMS> 

GEMSd Maintenance

1. The output from the GEMS logging to stdout can get pretty big. Be sure to stop the server and compress the files regularly.