As explained in the benchmark utilities page, it possible to generate some performance histograms and a tree with information which can be used to monitor the performance of a cluster. This can be used to trace possible bottlenecks and therefore it is a useful debugging tool.
The histograms contain the following information:
| Name | Type | Description |
|---|---|---|
| PROOF_PacketsHist | TH1D | "Packets processed per Worker" |
| PROOF_EventsHist | TH1D | "Events processed per Worker" |
| PROOF_ProcPcktHist | TH1I | "Packets being processed per Worker" |
| PROOF_NodeHist | TH1D | "Workers per Fileserving Node" |
| PROOF_LatencyHist | TH2D | "GetPacket Latency per Worker" |
| PROOF_ProcTimeHist | TH2D | "Packet Processing Time per Worker" |
| PROOF_CpuTimeHist | TH2D | "Packet CPU Time per Worker" |
All histograms are dynamically updated, so they can be used in feedback to monitor the worker activity. Starting with ROOT v5.34, the option "fb=stats" in TProof::Process enables feedback for the first three histograms displayed in one canvas.
The tree contains detailed information about 'packet' processing and file opening; the interpretation of the tree requires some expertise, though.
The purpose of this section is to explain how to create and save the relevant information.
1. Creating the performance tree with ROOT >= 5.33
For ROOT >= 5.33 (SVN rev 42382) TProof provides two methods to facilitate the generation and saving of the performance tree:
void TProof::SetPerfTree(const char *pf = "perftree.root", Bool_t withWrks = kFALSE);
A call to SetPerfTree before the query set ups the relevant parameters to create the performance tree; when the query is over the tree is saved to the file specified by the first argument (default 'perftree.root'). The second argument controls whether the start and stop information from the workers should also be recorded (default is not); generation of the performance tree must be re-enabled before each run.
Int_t TProof::SavePerfTree(const char *pf = 0, const char *qref = 0);
This method saves the performance tree for query whose unique reference is 'qref' (by default last query) to be saved to the file indicated by the first argument (default is the path previously set via a call to SetPerfTree or 'perftree.root'). This method is called internally by TProof::Process when the creation and saving of the performance tree is enabled.
2. Creating the performance tree with ROOT
2.1 Setting up the parameters
To create the performance tree and histograms, the following should be done before running the query:
root[] proof->SetParameter("PROOF_StatsHist", "")
root[] proof->SetParameter("PROOF_StatsTrace", "")
or
root[] gEnv->SetValue("Proof.StatsHist",1)
root[] gEnv->SetValue("Proof.StatsTrace",1)
(to save detailed information per worker the following setting needs also to be added
root[] proof->SetParameter("PROOF_SlaveStatsTrace", "")
or
root[] gEnv->SetValue("Proof.SlaveStatsTrace",1)
but this is typically not required).
After processing the query, the output list should contain, in addition to the user output objects, the following objects:
root[] proof->GetOutputList()->ls()
...
OBJ: TTree PROOF_PerfStats PROOF Statistics : 0 at: 0x163ca50
OBJ: TH1D PROOF_PacketsHist Packets processed per Worker : 0 at: 0x163a180
OBJ: TH1I PROOF_ProcPcktHist Packets being processed per Worker : 0 at: 0x163a180
OBJ: TH1D PROOF_EventsHist Events processed per Worker : 0 at: 0x1a93b70
OBJ: TH1D PROOF_NodeHist Slaves per Fileserving Node : 0 at: 0x1639b20
OBJ: TH2D PROOF_LatencyHist GetPacket Latency per Worker : 0 at: 0x1a90c40
OBJ: TH2D PROOF_ProcTimeHist Packet Processing Time per Worker : 0 at: 0x1b07c40
OBJ: TH2D PROOF_CpuTimeHist Packet CPU Time per Worker : 0 at: 0x1b40860
...
root []
The tree "PROOF_PerfStats" contains detailed information about the query.
2.2 Saving the tree
To save the performance tree and histograms into a file one can execute the macro $ROOTSYS/test/ProofBench/SavePerfInfo.C with, as argument, the full path to the file where to save the information:Crea
root[] .x $ROOTSYS/test/ProofBench/SavePerfInfo.C("/tmp/perf.root")