![]() |
ROOT
6.06/09
Reference Guide
|
Using PROOF on Demand is our current recommended way of running a PROOF cluster. The usage of PoD is in particular helpful for the following reasons:
This guide particularly refers to the setup of a static PROOF cluster running on physical hosts: the recommended setup is in practice the same as the ready-to-go Virtual Analysis Facility. If you want to use PROOF on the clouds there is no configuration to go through.
Although PROOF on Demand can run on a cluster of nodes without using a resource management system (using pod-ssh
), it is recommended to setup a dedicated one to benefit from the scheduling in a multiuser environment, or a dedicated queue on an existing one.
As there's a variety of resource management systems, this guide does not cover their setup. The RMS preconfigured for the Virtual Analysis Facility is HTCondor, which we recommend primarily because it has dynamic addition of workers built in.
CernVM-FS should be installed on all machines as the preferred method for software distribution.
Configuration instructions for the latest CernVM-FS can be found here.
A brief step-by-step procedure to install CernVM-FS is hereby described.
cvmfs
package, you don't need the cvmfs-devel
or cvmfs-server
ones.# cvmfs_config setup
Start the autofs
service: how to to this depends on your operating system.
On Ubuntu using Upstart:
# restart autofs
On RHEL-based or older Ubuntus:
# service autofs restart
Prepare a /etc/cvmfs/default.local
file (create it if it does not exists) with the following configuration bits:
``` {.bash} CVMFS_HTTP_PROXY=http://your-proxy-server.domain.ch:3128,DIRECT CVMFS_REPOSITORIES=your-experiment.cern.ch,sft.cern.ch CVMFS_QUOTA_LIMIT=50000 ```
You need to properly specify your closest HTTP caching proxy: separate many of them via commas. The last fallback value, DIRECT
, tells cvmfs to connect directly without using any proxy at all.
Among the list of repositories (comma-separated), always specify sft.cern.ch
and the one containing the software to your experiment (e.g., cms.cern.ch
).
The quota limit is, in Megabytes, the amount of local disk space to use as cache.
# cvmfs_config chksetup OK # cvmfs_config probe Probing /cvmfs/cms.cern.ch... OK Probing /cvmfs/sft.cern.ch... OK
You might need special configurations for some custom software repositories! Special cases are not covered in this guide.
PROOF on Demand is very flexible in handling various cases of network topologies. The best solution would be to allow all TCP communications between the cluster machines.
No other incoming communication is required from the outside.
Latest recommended sshcertauth version is 0.8.5.
If you want your users to connect to the PROOF cluster using their Grid user certificate and private key you might be interested in installing sshcertauth. Please refer to the installation guide for further information.
Latest recommended PROOF on Demand version is 3.12.
On CernVM-FS:
/cvmfs/sft.cern.ch/lcg/external/PoD/3.12
Source code: PoD download page and Installation instructions
PROOF on Demand is required on the head node and on the user's client.
In case your experiment provides a version of PoD on CernVM-FS you can use that one. Experiment-independent versions are available from the PH-SFT cvmfs repository.
Only if you have specific reasons while you want to use a customly built PoD version, download the source code and compile it using the installation instructions.
Please note that CMake and Boost are required to build PoD.
make install
After installing PoD, run:
pod-server getbins
This has to be done only once and downloads the binary packages that will be dynamically transferred to the worker nodes as binary payload, and prevents us from installing PoD on each cluster node.
It is important to do this step now, because in case PoD has been installed in a directory where the user has no write privileges, as in the case of system-wide installations, the user won't be able to download those required packages in the PoD binary directory.
There is no need to "configure" PoD for your specific cluster: it is just enough to install it on your head node.
PoD does not have any system-wide persistent daemon running or any system-wide configuration to be performed. Also, no part of PoD will be ever run as root.
Do not worry about environment or software configuration at this time: there is no system configuration for that. All the environment for your software dependencies will be set via proper scripts from the PoD client.
PoD client configuration and running is properly covered in the appropriate manual page.
The head node only requires TCP ports 22 (SSH) and 443 (HTTPS) to accept connections from the outside. Users will get an authentication "token" from port 443 and all PROOF traffic will be automatically tunneled in a SSH connection on port 22 by PoD.
In case you are not using the HTTPS+SSH token+authentication method, access to the sole port 22 is all you need.