Configuring a federated cluster (aka multi-level master)

WARNING : page under construction

Introduction

In this section we describe how to set up a federate PROOF cluster, i.e. a cluster which results from grouping together machines which may reside on different geographical domains. The example we going to discuss is schematically displayed in Figure 1.

Figure 1. Schematic view of a multi-level master setup federating two simple PROOF clusters

The two clusters at .domain.one and .domain.two are two simple PROOF clusters which can be accessed directly at master.domain.one and master.domain.two . Both cluster export the /data directory for data access, and have the user PROOF sandboxes under /data/proofbox. The goal of the exercise is to setup a third machine, proofgate.domain , to act as federator in such a way that the client starting a PROOF session at proofgate.domain will have a 6 worker machines in the session.

In this example we federate only the PROOF part. Data serving will be kept separated, with two entry points at master.domain.one and master.domain.two .

In order to achieve out goal we need two configuration files in addition to the ones needed to setup the two simple PROOF cluster. We start, however, by reviewing the configuration of the simple clusters and then we discuss how to join them together.

Submaster configuration

PROOF configuration files

The PROOF configuration is very simple; see proof.domain.one.conf and proof.domain.two.conf. The only addition is the specification of the 'msd' option ('msd' stands for Mass Storage Domain), a string that is used to instruct the packetizer to partition the files to be processed (more later about this).

XROOTD configuration files

Also the XROOTD configuration files are rather standard: xrd.domain.one.cf and xrd.domain.two.cf . We have made use of the all.export, all.role, all.manager new directives to simplify the data serving part. The file assumes that the PROOF configuration files are located under the path defined by the PROOFCFG environment variable.

Testing the setup

Starting xrootd and oldb on the two sub-clusters should setup two working PROOF clusters accessible at master.domain.one and master.domain.two .

Supermaster configuration

PROOF configuration files

The PROOF configuration file is again very simple; we use the keyword 'submaster' to specify the subclusters that we want to federate.

XROOTD configuration files

In the XROOTD configuration file the data serving part is not really configured, as it is not used; however, we change the port used to avoid conflicts with other xrootd related activities. In the PROOF part we just need to specify that the role is 'supermaster' and to give the path to the PROOF config file .

Federating the data serving part

To federate also the data serving part, so that the main entry point for data is root://proofgate.domain, one needs to modify the xrootd configuration files and to start an 'olbd' daemon also on 'proofgate.domain'. The configuration file for proofgate.domain needs to be added a real data serving file as shown in xrd.supermaster-redir.cf. The configuration files for the sub-clusters need to define the master.domain.one and master.domain.two role as 'supervisor' and to define as manager olbd the top olbd , i.e. the one running on proofgate.domain : see xrd-spv.domain.one.cf and xrd-spv.domain.two.cf .