Audit Logging in AstroGrid-D

AstroGrid-D uses the GRAM Audit module to achieve basic job audit logging. (Note: gsissh activity can not be monitored.)

Introduction

The job data is first stored in a PostgreSQL database and then uploaded to Stellaris by a database trigger. To enable audit logging, two steps have to be completed:

1. On every resource, the GRAM audit logging must be enabled (instructions follow below). GRAM Audit Logging is used for host monitoring of globusrun-ws jobs.

2. Once per site (institute) a PostgreSQL database dedicated to audit logging must be set up. See second part of this manual.

Requirements for audit logging are:

GT 4.0.5 or above (test: $GLOBUS_LOCATION/bin/globus-version)
A PostgreSQL database which is accessible for globus user (firewall!) and in which You can define Perl functions (test: createdb auditDatabase; createlang plperlu auditDatabase)

1. Configuration of grid resources for audit logging

Only Globus configuration files need be altered.

Edit (as user globus) the $GLOBUS_LOCATION/container-log4j.properties, which activates GRAM audit logging, and add these lines:


# GRAM				AUDIT
log4j.category.org.globus.exec.service.exec.StateMachine.audit=DEBUG,				AUDIT
log4j.appender.AUDIT=org.globus.exec.utils.audit.AuditDatabaseAppender
log4j.appender.AUDIT.layout=org.apache.log4j.PatternLayout
log4j.additivity.org.globus.exec.service.exec.StateMachine.audit=false

Edit (as user globus) the file $GLOBUS_LOCATION/etc/gram-service/jndi-config.xml which contains the information for connecting the audit database. In the <resource name="auditDatabaseConfiguration"> element change the host name after jdbc:postgresql:// and the password as shown below (the changes are italic).


<resource
...>
  <resourceParams> ...
    <parameter>

<name>url</name><value>jdbc:postgresql://servername/auditDatabase</value> </parameter>

<parameter><name>user</name><value>globus</value></parameter>

<parameter><name>password</name><value>foo</value></parameter> ... </resourceParams> </resource>

where servername is the machine that runs the central database (at AIP it is is.astrogrid-d.org, or astrodata01.gac-grid.org), user name and password are defined according to the first line in update_stellaris.sql. If the database is running locally, 'localhost' doesn't need to be changed.

2. Configuration of a central site database

One audit Database can be used for logging of one or more Grid containers, e.g. a whole cluster or institute.

Install the database table (following http://www.globus.org/toolkit/docs/4.0/execution/wsgram/WS_GRAM_Audit_Logging.html .

Download the file update_stellaris.sql. As user postgres:


createdb auditDatabase

createlang plperlu auditDatabase

psql -d auditDatabase -f update_stellaris.sql

If createlang fails, add the package "postgresql.pl" to your postgres installation.

If createlang still fails, reinstall at first PostgreSQL database version 8.2 from sources:

./configure --with-perl; make; sudo make install.

If PostgreSQL version 7 is used, replace $update_stellaris$ in update_stellaris.sql by single quotes "'".

The first line of Connection Settings in /var/lib/pgsql/data/postgresql.conf has to be changed:

# Connection - Settings -
listen_addresses = '*'
...

The /var/lib/pgsql/data/pg_hba.conf is updated

# TYPE DATABASE        USER      IP-ADDRESS      IP-MASK          METHOD
host   auditDatabase   globus   <IP-Mask>   md5
...

where <IP-Mask> picks out all the hosts that will connect to the server. PostgreSQL takes several styles of these, e.g. an address with a bit-mask: 141.33.4.161/27. Consult the PostgreSQL documentation.

Then start the database with


sudo /etc/init.d/postgresql restart

, and restart the Globus container with


sudo /etc/init.d/globus restart

.

Note that each time the PostgreSQL database is restarted on the server, all the Globus containers of all the hosts connecting to the database must be restarted, in order to re-establish the connection.

3. Pre WS GRAM resource configuration for audit logging

Pre WS GRAM Audit Logging is used for host monitoring of globus-job-run jobs.

Add to $GLOBUS_LOCATION/log4j.properties, these lines to activate GRAM audit logging:


# GRAM AUDIT

log4j.category.org.globus.exec.service.exec.StateMachine.audit=DEBUG, AUDIT

log4j.appender.AUDIT=org.globus.exec.utils.audit.AuditDatabaseAppender

log4j.appender.AUDIT.layout=org.apache.log4j.PatternLayout

log4j.additivity.org.globus.exec.service.exec.StateMachine.audit=false

Create a directory /tmp/globus, and edit the file $GLOBUS_LOCATION/etc/globus-job-manager.conf.


-home "/work1/globus/gt406"

-globus-gatekeeper-host dublin.aip.de

-globus-gatekeeper-port 2119

-globus-gatekeeper-subject "/C=DE/O=GermanGrid/OU=AIP/CN=host/dublin.aip.de"

-globus-host-cputype x86_64

-globus-host-manufacturer unknown

-globus-host-osname Linux

-globus-host-osversion 2.6.18-8.1.14.el5xen

-globus-toolkit-version 4.0.6

-save-logfile on_error

-state-file-dir /work1/globus/gt406/tmp/gram_job_state

-machine-type unknown

-audit-directory /tmp/globus

By default, You need only to edit the last line.

Then restart the Globus container with sudo /etc/init.d/globus restart.

To put the files from the /tmp/globus (permissions drws-wsrwx, e.g. chmod u+s /tmp/globus) directory to Stellaris, use the Perl script update_stellaris.cron (download here) as a cron job.

Test

From now on, the jobs are shown in the Grid timeline.