1. Syslog-based monitoring
Since ROOT version 5.27/04 (SVN revision #33369; the functionality is also available in the special branch 5.26/00-proof) it is possible to configure PROOF in such a way that some information about the activity on the cluster is logged via the syslog machinery. A tool to parse and visualize this information is under development.
1.1 Configuration
Enabling monitoring via syslog requires two steps: i) instructing the syslog system to redirect the 'local5' logs to a log file dedicated to PROOF monitoring; ii) instructing PROOF to send information to syslog.
1.1.1 Defining the log file
The syslog facility used by PROOF is 'local5' . In order to connect this facility to a local file, first choose a location for the file, e.g. /var/log/xpdmon.log (in the following we will use this path in the examples), and create the empty file. We have then to instruct the syslog daemon that logs to local5 should go to tye chosen file; to achieve this we have to edit the syslog config file, usually /etc/syslog.conf to add the following line
# /etc/syslog.conf Configuration file for syslogd.
...
#
# PROOF logging
local5.debug /var/log/xpdmon.log
We have also to tell syslog not to send local5 logs to the default file, usually /var/log/messages. For that we need to find in /etc/syslog.conf the line similar to this
*.info;mail.none;news.none;authpriv.none /var/log/messages
and change it to look as this
*.info;local5,mail.none;news.none;authpriv.none /var/log/messages
To make the changes affective we need to restart the syslog daemon
/etc/init.d/syslog restart
If a full restart is not desired, forcing a re-initialization should be sufficient: this can be done by sending a SIGHUP signal to the daemon
kill -SIGHUP ‘cat /var/run/syslogd.pid‘
1.1.2 The ProofServ.LogToSysLog directive
By default, the proofserv application does not log anything to syslog. To enable sending the information one needs to set a ProofServ.LogToSysLog directive . This is usually done via the xpd.putrc directive of the XrdProofd daemon.
### Log to syslog
xpd.putrc ProofServ.LogToSysLog The directive defines a 2 character string: the first character is a letter which can take the following values:
| a | log all |
| m | log only master |
| w | log only workers |
The second character is a number indicating the log level:
| 0 | log disabled |
| 1 | log start-of-session, main actions (exec, process, dataset handling, cache/package handling, query handling) end-of-session |
| 2 | As 1 but log also all the remaining activity on the input socket (this includes handling of some auxilliary messages) |
| 3 | Log all proofserv logs (i.e. those usually going to the log file) in addition to those obtained with 2 |
The typical setting to monitor the cluster usage is 'm1', i.e. log level 1 on the master.
Example of output obtained from running the "eventproc" tutorial example:
May 4 10:43:55 pcphsft64 proofm-0[14889]: ganis:proofteam 0 0.000 0.000
May 4 10:43:57 pcphsft64 proofm-0[14889]: ganis:proofteam 1012 0.346 0.000 +event.par
May 4 10:43:58 pcphsft64 proofm-0[14889]: ganis:proofteam 1018 0.949 0.020 6 event 0
May 4 10:43:59 pcphsft64 proofm-0[14889]: ganis:proofteam 1018 0.069 0.000 7 event 0
May 4 10:43:59 pcphsft64 proofm-0[14889]: ganis:proofteam 1018 0.001 0.000 8 0
May 4 10:43:59 pcphsft64 proofm-0[14889]: ganis:proofteam 1012 0.055 0.020 ProofEventProc.C
May 4 10:43:59 pcphsft64 proofm-0[14889]: ganis:proofteam 1012 0.009 0.000 ProofEventProc.h
May 4 10:44:58 pcphsft64 proofm-0[14889]: ganis:proofteam 1015 59.365 0.670 0 1000000 596961398 83.530
May 4 10:45:04 pcphsft64 proofm-0[14889]: ganis:proofteam -1 60.819 0.720
1.2 Information posted
We describe in this section the format of the information obtained in the case of level 1. The record hs the following format
date nodename proofid[pid]: user:group what realtime cputime auxinfo
where
- date is the date in the form 'day month time', e.g. 'May 4 11:37:54'
- nodename is the name of the node where the information is recorded
- proofid is a tag indicating the role of the node, e.g. proofm-0 for a top master, or proofw-0.16 for a worker (ordinal 0.16)
- pid is the process ID of the proofserv logging the information
- user is the user name of the session producing the info
- group is the PROOF group of the user producing the info
- what is the message type of the requested action
- realtime is the real time spent executing the action
- cputime is the CPU time spent executing the action
- auxinfo is some auxilliary information which may be present depending on what (see below).
A record with what = 0 indicates the beginning of the session (realtime and cputime are 0.000 in this case). A record with what = -1 indocates the end of the session; realtime and cputime are, in this case, the total real and CPU time spent in execution during the whole session, respectively.
Auxilliary information is present in the following cases:
The monitoring information is sent by the master at the end of the query. The quantities posted are shown in the table:
| what (name) | what (id) | Format | Description |
|---|---|---|---|
| kMESS_CINT | 5 | cmd | Command being executed |
| kPROOF_PROCESS | 1015 | status entries bytes cputime |
status: query status entries: number of entries processed (or cycles performed) bytes: number of bytes read if processing data cputime: total CPU time used by the workers |
|
kPROOF_RETRIEVE kPROOF_REMOVE |
1032 1034 |
queryref | Query reference ID |
| kPROOF_ARCHIVE | 1033 | queryref path | Query reference ID and archiving path |
| kPROOF_CHECKFILE | 1012 | filename | Name of the file being checked |
| kPROOF_CACHE | 1018 | type aux | Cache handling sub-action type; the auxilliary info is described in the table below |
| kPROOF_DATASETS | 1042 | type aux | Dataset handling sub-action type; the auxilliary info is described in the table below |
2. The ProofServ.Monitoring directive
It is possible to configure PROOF in such a way that some information about the queries processed on the cluster are sent to a MonALISA monitoring system or to a SQL database. Starting from 5.29/02 (and 5.28/00c) it is possible to send the information to multiple collectors entering multiple 'ProofServ.Monitoring' directives, separated by a ',' or a '|' or a '\'; for example
ProofServ.Monitoring SQL mysql://localhost:3306 proof.proofquerylog,
+ProofServ.Monitoring: MonaLisa 192.168.2.22 ::::
To enable posting the monitoring information one has to give instruction about which monitoring system has to be used. This is done via the rootrc directive 'ProofServ.Monitoring'. This directive takes a name and up to 9 'const char *' additional configuration arguments for the relevant TProofMonSender plug-in. The TProofMonSender plug-ins available currently are TProofMonSenderML (for a MonALISA backend) and TProofMonSenderSQL (for SQL backends).
Note that these variables can be defined in the XROOTD configuration file via the 'xpd.putrc' directive .
Note that, for ROOT version up to 5.28/00f and 5.30/01, the directives were used to intialize directly the TMonaLisaWriter and TSQLMonitoringWriter plug-ins.
2.1 Configuration
2.1.1 MonALISA
The directive to load the MonALISA writer has the form:
ProofServ.Monitoring MonaLisa
The parameter server specifies the server to whom to send the monitoring UDP packets. If not specified, the hostname is specified in the environment variable APMON_CONFIG, with the default port.
The parameter tag is equivalent to the MonaLisa farm name; for the case of process monitoring it can be a process identifier e.g. a PROOF session ID.
The parameter id is equivalent to the MonaLisa node name; for the case of process monitoring it can be just an identifier to classify the type of jobs e.g. PROOF_PROCESSING. If id is not specified, TMonaLisaWriter tries to set it from environment variables in this order: PROOF_JOB_ID, GRID_JOB_ID, LCG_JOB_ID, ALIEN_MASTERJOB_ID, ALIEN_PROC_ID.
The parameter subid can be used to have finer identifier granularity.
The parameter option is currently only used as local/global switch: if set as "global" the global gMonitoringWriter is set to the instantiated instance of TMonaLisaWriter, so that a unique monitoring writer is used in the session. In PROOF, use "global" to send fine grained processing information (see below).
The sendopts switch is described below.
2.1.2 SQL
The directive to load the SQL writer is:
ProofServ.Monitoring SQL [. [.] [.]] [sendopts:]
The parameter DBMS is the URL used to identify the DB server, in the form
://[:][/]
The parameter user is the user-name to be used to connect to the DBMS.
The parameter passwd is the password associated with the user-name above mentioned.
The fourth parameter indicates the database and the table where to insert the monitoring information. Specifying the tables databases and names is not mandatory. The default for the database is 'proof' and for the tables 'proofquerylog', 'proofquerydsets' and 'proofqueryfiles'. If the summarytable and its database dbs are specified and the other tables are not, then the database dbs is used for all the tables, with default table names.
The sendopts switch is described below.
2.1.3 The sendopts switch
The sendopts string allows to configure the records to be sent and their version (see below); the string contains ':' separated information for each record in the form [{+,-}]RecordTag[RecordVersion]; the '-'('+') prefix disables(enables) the sending of a record (if absent '+' is assumed). Recognized values for the RecordTag are given in the table:
| RecordTag | Description |
|---|---|
| S | Query summary |
| D | Information about processed datasets |
| F | Information about processed files |
For example, if sendopts is set to 'S1:D0:-F', the process will post the query summary record, version 1, and the dataset record, version 0, while detailed information about the processed files will not be sent.
The optional RecordVersion is an integer number specifying the version of the record. For dataset and files information records, the version constrols the amount of information in the record as described below in the relevant sections. For the query summary record, it is a way to provide the possibility to post information as in previous ROOT versions. The following table gives the correspondence:
|
Summary Record Version |
ROOT Versions |
|---|---|
| 0 | |
| 1 | 5.28/00c -> 5.28/00f, 5.30/00, 5.30/01 |
| 2 | > 5.28/00f, > 5.30/01 |
2.2 Information posted
The monitoring information is sent by the master at the end of the query. In this section we give a detailed secription of the records sent under the different configuration options.
The quantities posted in the query summary record are shown in the table:
| Description | Name | Type | ver 0 | ver 1 | ver 2 |
|---|---|---|---|---|---|
| User name | user | XDR_STRING | o | o | o |
| Group name | group | XDR_STRING | o | ||
| Group name | proofgroup | XDR_STRING | o | o | |
| Starting processing time | begin | XDR_STRING | o | o | o |
| End processing time | end | XDR_STRING | o | o | o |
| Total wall time | walltime | XDR_REAL64 | o | o | o |
| Total CPU time | cputime | XDR_REAL64 | o | o | o |
| Number of bytes read | bytesread | XDR_REAL64 | o | o | o |
| Number of events processed | events | XDR_REAL64 | o | o | o |
| Number of workers assigned | workers | XDR_REAL64 | o | o | o |
| Max virtual memory used by workers (kB) | vmemmxw | XDR_REAL64 | o | o | |
| Max resident memory used by workers (kB) | rmemmxw | XDR_REAL64 | o | o | |
| Max virtual memory used during merging (kB) | vmemmxm | XDR_REAL64 | o | o | |
| Max resident memory used during merging (kB) | rmemmxm | XDR_REAL64 | o | o | |
| Dataset name | dataset | XDR_STRING | o | ||
| Number of files in the dataset | numfiles | XDR_REAL64 | o | o | |
| Number of missing files | missfiles | XDR_REAL64 | o | ||
| Query exit status | status | XDR_REAL64 | o | ||
| ROOT versions | rootver | XDR_STRING | o |
The entry is tagged with the unique tag "
The 'vmemmxw' and 'rmemmxw' represent the maximum virtual and resident memory (in kB) used by worker during processing; the 'vmemmxm' and 'rmemmxm' are the maximum values during merging on the master.
For version 1, the user and proof-group names are left in the dataset string only if different from the 'user' and 'proofgroup' of the query; multiple dataset are entered as a comma- separated list; for non-dataset based queries, the name of the class used to define the set of files is entered, e.g. TChain, TDSet or TFileCollection. The string '+++none+++' is used for non-data drive analysis.
The 'status' field contains the exit status of the query (0 = OK; 1 = stopped, 2 = aborted).
The quantities posted in the dataset record are shown in the table:
| Description | Name | Type | ver 0 | ver 1 |
|---|---|---|---|---|
| Dataset name | dsn | XDR_STRING | o | o |
| Query tag | querytag | XDR_STRING | o | o |
| Starting processing time | querybegin | XDR_STRING | o | |
| Number of files in the dataset | numfiles | XDR_REAL64 | o | o |
| Number of missing files | missfiles | XDR_REAL64 | o | o |
The entry is tagged with "dataset_
The quantities posted in the files record are shown in the table:
| Description | Name | Type | ver 0 | ver 1 |
|---|---|---|---|---|
| File basename | lfn | XDR_STRING | o | o |
| Full path (including Url) | path | XDR_STRING | o | o |
| Starting processing time | querybegin | XDR_STRING | o | |
| Availability status | status | XDR_REAL64 | o | o |
The entry is tagged with "file_
The quantities posted in the query summary record are shown in the table:
| Description | Name | Type | ver 0 | ver 1 | ver 2 |
|---|---|---|---|---|---|
| User name | user | VARCHAR(32) | o | o | |
| User name | proofuser | VARCHAR(32) | o | ||
| Group name | group | VARCHAR(32) | o | ||
| Group name | proofgroup | VARCHAR(32) | o | o | |
| Starting processing time | begin | DATETIME | o | o | |
| Starting processing time | querybegin | DATETIME | o | ||
| End processing time | end | DATETIME | o | o | |
| End processing time | queryend | DATETIME | o | ||
| Total wall time | walltime | INT | o | o | o |
| Total CPU time | cputime | FLOAT | o | o | o |
| Number of bytes read | bytesread | BIGINT | o | o | o |
| Number of events processed | events | BIGINT | o | o | o |
| Number of workers assigned | workers | INT | o | o | o |
| Query unique tag | query | VARCHAR(64) | o | o | |
| Max virtual memory used by workers (kB) | vmemmxw | BIGINT | o | o | |
| Max resident memory used by workers (kB) | rmemmxw | BIGINT | o | o | |
| Max virtual memory used during merging (kB) | vmemmxm | BIGINT | o | o | |
| Max resident memory used during merging (kB) | rmemmxm | BIGINT | o | o | |
| Dataset name | dataset | VARCHAR(512) | o | ||
| # of files requested | numfiles | INT | o | o | |
| # of missing files | missfiles | INT | o | ||
| Query exit status | status | INT | o | ||
| ROOT version | rootver | VARCHAR(32) | o |
The 'vmemmxw' and 'rmemmxw' represent the maximum virtual and resident memory (in kB) used by worker during processing; the 'vmemmxm' and 'rmemmxm' are the maximum values during merging on the master.
For version 1, the user and proof-group names are left in the dataset string only if different from the 'user' and 'proofgroup' of the query. Multiple dataset are entered as a comma- separated list. The dataset string is in any case truncated at 512 max characters. For non-dataset based queries, the name of the class used to define the set of files is entered, e.g. TChain, TDSet or TFileCollection. The string '+++none+++' is used for non-data drive analysis.
The 'status' field contains the exit status of the query (0 = OK; 1 = stopped, 2 = aborted).
The table can created in the following way (any name can be used for the table, provided that the correct value is used in the relevant ProofServ.Monitoring directive) :
Version 2:
CREATE TABLE proofquerylog (
id int(11) NOT NULL auto_increment,
proofuser varchar(32) NOT NULL,
proofgroup varchar(32) default NULL,
querybegin datetime default NULL,
queryend datetime default NULL,
walltime int(11) default NULL,
cputime float default NULL,
bytesread bigint(20) default NULL,
events bigint(20) default NULL,
totevents bigint(20) default NULL,
workers int(11) default NULL,
querytag varchar(64) NOT NULL,
vmemmxw bigint(20) default NULL,
rmemmxw bigint(20) default NULL,
vmemmxm bigint(20) default NULL,
rmemmxm bigint(20) default NULL,
numfiles int(11) default NULL,
missfiles int(11) default NULL,
status int(11) default NULL,
rootver varchar(32) NOT NULL,
PRIMARY KEY (id)
)
Version 1:
CREATE TABLE proofquerylog (
id int(11) NOT NULL auto_increment,
proofuser varchar(32) NOT NULL,
proofgroup varchar(32) default NULL,
querybegin datetime default NULL,
queryend datetime default NULL,
walltime int(11) default NULL,
cputime float default NULL,
bytesread bigint(20) default NULL,
events bigint(20) default NULL,
totevents bigint(20) default NULL,
workers int(11) default NULL,
querytag varchar(64) NOT NULL,
vmemmxw bigint(20) default NULL,
rmemmxw bigint(20) default NULL,
vmemmxm bigint(20) default NULL,
rmemmxm bigint(20) default NULL,
numfiles int(11) default NULL,
dataset varchar(512) NOT NULL,
PRIMARY KEY (id)
)
Version 0:
CREATE TABLE proofquerylog (
id int(11) NOT NULL auto_increment,
proofuser varchar(32) NOT NULL,
proofgroup varchar(32) default NULL,
querybegin datetime default NULL,
queryend datetime default NULL,
walltime int(11) default NULL,
cputime float default NULL,
bytesread bigint(20) default NULL,
events bigint(20) default NULL,
totevents bigint(20) default NULL,
workers int(11) default NULL,
PRIMARY KEY (id)
)
The quantities posted in the dataset record are shown in the table:
| Description | Name | Type | ver 0 | ver 1 |
|---|---|---|---|---|
| Dataset name | dsn | VARCHAR(512) | o | o |
| Query unique tag | querytag | VARCHAR(64) | o | o |
| Starting processing time | querybegin | DATETIME | o | |
| # of files requested | numfiles | INT | o | o |
| # of missing files | missfiles | INT | o | o |
The 'querytag' gives the link with the query summary record to lookup other quantities, e.g. the user requiring the dataset. The number of files is the number of files registered to this dataset; the number of missing files is the number of the required files not available or that could not be opened at the moment the query was run.
The table can created in the following way (any name can be used for the table, provided that the correct value is used in the relevant ProofServ.Monitoring directive) :
Version 1:
CREATE TABLE proofquerydsets (
id int(11) NOT NULL auto_increment,
dsn varchar(512) NOT NULL,
querytag varchar(64) NOT NULL,
querybegin datetime default NULL,
numfiles int(11) default NULL,
missfiles int(11) default NULL,
PRIMARY KEY (id),
KEY ix_querytag (querytag)
)
Version 0:
CREATE TABLE proofquerydsets (
id int(11) NOT NULL auto_increment,
dsn varchar(512) NOT NULL,
querytag varchar(64) NOT NULL,
numfiles int(11) default NULL,
missfiles int(11) default NULL,
PRIMARY KEY (id),
KEY ix_querytag (querytag)
)
The quantities posted in the files record are shown in the table:
| Description | Name | Type | ver 0 | ver 1 |
|---|---|---|---|---|
| File basename | lfn | VARCHAR(255) | o | o |
| Full path (including Url) | path | VARCHAR(2048) | o | o |
| Query unique tag | querytag | VARCHAR(64) | o | o |
| Starting processing time | querybegin | DATETIME | o | |
| File availability status | status | enum('Ok','Failed') | o | o |
The 'querytag' gives the link with the query summary record to lookup other quantities, e.g. the user requiring the file. The 'status' tells if the file was successfully analysed (value 1) or not (value 0) by the query.
The table can created in the following way (any name can be used for the table, provided that the correct value is used in the relevant ProofServ.Monitoring directive):
Version 1:
CREATE TABLE proofqueryfiles (
id int(11) NOT NULL auto_increment,
lfn varchar(255) NOT NULL,
path varchar(2048) NOT NULL,
querytag varchar(64) NOT NULL,
querybegin datetime default NULL,
status enum('Ok','Failed') NOT NULL default 'Ok',
PRIMARY KEY (id),
KEY ix_querytag (querytag)
)
Version 0:
CREATE TABLE proofquerydsets (
id int(11) NOT NULL auto_increment,
lfn varchar(255) NOT NULL,
path varchar(2048) NOT NULL,
querytag varchar(64) NOT NULL,
status enum('Ok','Failed') NOT NULL default 'Ok',
PRIMARY KEY (id),
KEY ix_querytag (querytag)
)
2.2.2.4 Configuring the bulk INSERT
The dataset and files records are posted with a bulk INSERT. The maximal size for such an operation is controlled by the max_allowed_packet configuration parameter of 'mysqld', which should default to 16 MB . The TSQLMonitoring plug-in honours this by sending off a bulk chunk as soon as it reaches 90% of the maximal value, set by default to 16 MB. The maximal value can be changed with the directive SQLMonitoringWriter.MaxBulkSize which understands bytes, kilobytes (suffix 'k' or 'K'), megabytes (suffix 'm' or 'M') and gigabytes (suffix 'g' or 'G'). For example, the following
xpd.putrc SQLMonitoringWriter.MaxBulkSize 1048576
xpd.putrc SQLMonitoringWriter.MaxBulkSize 1024k
xpd.putrc SQLMonitoringWriter.MaxBulkSize 1M
are equivalent ways to set the maximal value to 1 MB.
3 Additional monitoring (MonALISA only)
If working with MonALISA, it is possible to send processing progress information to the monitoring server. The following sequence is sent on per worker base:
1) Just before requesting the first packet, an entry with:
| Name | Type | Description | Value |
|---|---|---|---|
| status | XDR_STRING | Status flag | STARTED |
| hostname | XDR_STRING | Worker host name | |
| subid | XDR_STRING | MonaLisa instance subid |
2) After each call to TSelector::Process, an entry with:
| Name | Type | Description | Value |
|---|---|---|---|
| subid | XDR_STRING | MonaLisa instance subid | |
| events | XDR_REAL64 | Number of events processed | |
| processedbytes | XDR_REAL64 | Number of bytes read | |
| realtime | XDR_REAL64 | Real (wall) time from the start | |
| cputime | XDR_REAL64 | CPU time used | |
| totmem | XDR_REAL64 | Total memory used | |
| rssmem | XDR_REAL64 | Resident memory used | |
| shdmem | XDR_REAL64 | Shared memory used | 0 |
| events_str | XDR_STRING | String version of 'events' | |
| processedbytes_str | XDR_STRING | String version of 'processedbytes' | |
| realtime_str | XDR_STRING | String version of 'realtime' | |
| cputime_str | XDR_STRING | String version of 'cputime' | |
| totmem_str | XDR_STRING | String version of 'totmem' | |
| rssmem_str | XDR_STRING | String version of 'rssmem' | |
| shdmem_str | XDR_STRING | String version of 'shdmem' | "0" |
| hostname | XDR_STRING | Worker host name |
3) Just after the last call to TSelector::Process, an entry with:
| Name | Type | Description | Value |
|---|---|---|---|
| status | XDR_STRING | Status flag | DONE |
| hostname | XDR_STRING | Worker host name | |
| subid | XDR_STRING | MonaLisa instance subid |
To enable the sending of this information the option parameter must be set to "global" in the MonALISA plug-in configuration directive.