This class provides an interface to process a TTree dataset in parallel with multi-process technology.
The possible usages of the Process method are the following:
For legacy, the following signature is also supported:
For either set of signatures, the processing function is executed as many times as needed by a pool of fNWorkers workers; the number of workers can be passed to the constructor or set via SetNWorkers. It defaults to the number of cores.
A collection containing the result of each execution is returned.
Note: the user is responsible for the deletion of any object that might be created upon execution of func, returned objects included: ROOT::TTreeProcessorMP never deletes what it returns, it simply forgets it.
Note: that the usage of ROOT::TTreeProcessorMP::Process is indicated only when the task to be executed takes more than a few seconds, otherwise the overhead introduced by Process will outrun the benefits of parallel execution on most machines.
func | a lambda expression, an std::function, a loaded macro, a functor class or a function that takes zero arguments (for the first signature) or one (for the second signature). |
args | a standard container (vector, list, deque), an initializer list or a pointer to a TCollection (TList*, TObjArray*, ...). |
Note: the version of ROOT::TTreeProcessorMP::Process that takes a TFileCollection* as argument incurs in the overhead of copying data from the TCollection to an STL container. Only use it when absolutely necessary.
Note: in cases where the function to be executed takes more than zero/one argument but all are fixed except zero/one, the function can be wrapped in a lambda or via std::bind to give it the right signature.
Note: the user should take care of initializing random seeds differently in each process (e.g. using the process id in the seed). Otherwise several parallel executions might generate the same sequence of pseudo-random numbers.
Methods taking 'F func' return the return type of F. Methods taking a TSelector return a 'TList *' with the selector output list; the output list content is owned by the caller.
See tutorials/multicore/mp102_readNtuplesFillHistosAndFit.C and tutorials/multicore/mp103__processSelector.C .
Definition at line 38 of file TTreeProcessorMP.hxx.
Public Member Functions | |
TTreeProcessorMP (const TTreeProcessorMP &)=delete | |
TTreeProcessorMP (unsigned nWorkers=0) | |
~TTreeProcessorMP ()=default | |
unsigned | GetNWorkers () const |
TTreeProcessorMP & | operator= (const TTreeProcessorMP &)=delete |
template<class F > | |
auto | Process (const std::string &fileName, F procFunc, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type |
template<class F > | |
auto | Process (const std::string &fileName, F procFunc, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type |
TList * | Process (const std::string &fileName, TSelector &selector, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) |
TList * | Process (const std::string &fileName, TSelector &selector, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) |
TSelector-based tree processing: dataset as a single file. More... | |
template<class F > | |
auto | Process (const std::vector< std::string > &fileNames, F procFunc, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type |
Process a TTree dataset with a functor: version without entry list. More... | |
template<class F > | |
auto | Process (const std::vector< std::string > &fileNames, F procFunc, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type |
Process a TTree dataset with a functor. More... | |
TList * | Process (const std::vector< std::string > &fileNames, TSelector &selector, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) |
Process a TTree dataset with a selector: version without entry list. More... | |
TList * | Process (const std::vector< std::string > &fileNames, TSelector &selector, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) |
Process a TTree dataset with a selector. More... | |
template<class F > | |
auto | Process (TChain &chain, F procFunc, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type |
template<class F > | |
auto | Process (TChain &files, F procFunc, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type |
TList * | Process (TChain &files, TSelector &selector, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) |
TList * | Process (TChain &files, TSelector &selector, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) |
TSelector-based tree processing: dataset as a TChain. More... | |
template<class F > | |
auto | Process (TFileCollection &collection, F procFunc, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type |
template<class F > | |
auto | Process (TFileCollection &files, F procFunc, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type |
TList * | Process (TFileCollection &files, TSelector &selector, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) |
TList * | Process (TFileCollection &files, TSelector &selector, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) |
TSelector-based tree processing: dataset as a TFileCollection. More... | |
template<class F > | |
auto | Process (TTree &tree, F procFunc, TEntryList &entries, ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type |
template<class F > | |
auto | Process (TTree &tree, F procFunc, ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type |
TList * | Process (TTree &tree, TSelector &selector, TEntryList &entries, ULong64_t nToProcess=0, ULong64_t jFirst=0) |
TSelector-based tree processing: memory resident tree. More... | |
TList * | Process (TTree &tree, TSelector &selector, ULong64_t nToProcess=0, ULong64_t jFirst=0) |
void | SetNWorkers (unsigned n) |
Private Types | |
enum class | ETask : unsigned char { kNoTask , kProcByRange , kProcByFile } |
A collection of the types of tasks that TTreeProcessorMP can execute. More... | |
Private Member Functions | |
template<class T > | |
void | Collect (std::vector< T > &reslist) |
Listen for messages sent by the workers and call the appropriate handler function. More... | |
void | FixLists (std::vector< TObject * > &lists) |
Fix list of lists before merging (to avoid errors about duplicated objects) More... | |
template<class T > | |
void | HandlePoolCode (MPCodeBufPair &msg, TSocket *sender, std::vector< T > &reslist) |
Handle message and reply to the worker. More... | |
void | ReplyToIdle (TSocket *s) |
Reply to a worker who is idle. More... | |
void | Reset () |
Reset TTreeProcessorMP's state. More... | |
Private Member Functions inherited from TMPClient | |
TMPClient (const TMPClient &)=delete | |
TMPClient (unsigned nWorkers=0) | |
Class constructor. More... | |
~TMPClient () | |
Class destructor. More... | |
template<class T > | |
unsigned | Broadcast (unsigned code, const std::vector< T > &objs) |
Send a message with a different object to each server. More... | |
template<class T > | |
unsigned | Broadcast (unsigned code, std::initializer_list< T > &objs) |
Send a message with a different object to each server. More... | |
template<class T > | |
unsigned | Broadcast (unsigned code, T obj, unsigned nMessages=0) |
Send a message containing code and obj to each worker, up to a maximum number of nMessages workers. More... | |
unsigned | Broadcast (unsigned code, unsigned nMessages=0) |
Send a message with the specified code to at most nMessages workers. More... | |
void | DeActivate (TSocket *s) |
DeActivate a certain socket. More... | |
bool | Fork (TMPWorker &server) |
This method forks the ROOT session into fNWorkers children processes. More... | |
bool | GetIsParent () const |
TMonitor & | GetMonitor () |
unsigned | GetNWorkers () const |
void | HandleMPCode (MPCodeBufPair &msg, TSocket *sender) |
Handle messages containing an EMPCode. More... | |
TMPClient & | operator= (const TMPClient &)=delete |
void | ReapWorkers () |
Wait on worker processes and remove their pids from fWorkerPids. More... | |
void | Remove (TSocket *s) |
Remove a certain socket from the monitor. More... | |
void | SetNWorkers (unsigned n) |
Set the number of workers that will be spawned by the next call to Fork() More... | |
Private Attributes | |
unsigned | fNProcessed |
number of arguments already passed to the workers More... | |
unsigned | fNToProcess |
total number of arguments to pass to the workers More... | |
ETask | fTaskType = ETask::kNoTask |
the kind of task that is being executed, if any More... | |
#include <ROOT/TTreeProcessorMP.hxx>
|
strongprivate |
A collection of the types of tasks that TTreeProcessorMP can execute.
It is used to interpret in the right way and properly reply to the messages received (see, for example, TTreeProcessorMP::HandleInput)
Definition at line 179 of file TTreeProcessorMP.hxx.
|
explicit |
|
default |
|
delete |
|
private |
Listen for messages sent by the workers and call the appropriate handler function.
TTreeProcessorMP::HandlePoolCode is called on messages with a code < 1000 and TMPClient::HandleMPCode is called on messages with a code >= 1000.
Definition at line 430 of file TTreeProcessorMP.hxx.
Fix list of lists before merging (to avoid errors about duplicated objects)
Definition at line 323 of file TTreeProcessorMP.cxx.
|
inline |
Definition at line 163 of file TTreeProcessorMP.hxx.
|
private |
Handle message and reply to the worker.
Definition at line 404 of file TTreeProcessorMP.hxx.
|
delete |
auto ROOT::TTreeProcessorMP::Process | ( | const std::string & | fileName, |
F | procFunc, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) | -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type |
Definition at line 363 of file TTreeProcessorMP.hxx.
auto ROOT::TTreeProcessorMP::Process | ( | const std::string & | fileName, |
F | procFunc, | ||
TEntryList & | entries, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) | -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type |
Definition at line 256 of file TTreeProcessorMP.hxx.
TList * ROOT::TTreeProcessorMP::Process | ( | const std::string & | fileName, |
TSelector & | selector, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) |
Definition at line 295 of file TTreeProcessorMP.cxx.
TList * ROOT::TTreeProcessorMP::Process | ( | const std::string & | fileName, |
TSelector & | selector, | ||
TEntryList & | entries, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) |
TSelector-based tree processing: dataset as a single file.
Definition at line 277 of file TTreeProcessorMP.cxx.
auto ROOT::TTreeProcessorMP::Process | ( | const std::vector< std::string > & | fileNames, |
F | procFunc, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) | -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type |
Process a TTree dataset with a functor: version without entry list.
No TEntryList versions of generic processor.
F | functor returning a pointer to TObject or inheriting classes and taking a TTreeReader& (both enforced at compile-time) |
Dataset definition:
[in] | fileNames | vector of strings with the paths of the files with the TTree to process |
[in] | fileName | string with the path of the files with the TTree to process |
[in] | collection | TFileCollection with the files with the TTree to process |
[in] | chain | TChain with the files with the TTree to process |
[in] | tree | TTree to process |
[in] | treeName | Name of the TTree to process |
[in] | nToProcess | Number of entries to process (0 means all) |
[in] | jFirst | First entry to process (0 means the first of the first file) |
Definition at line 353 of file TTreeProcessorMP.hxx.
auto ROOT::TTreeProcessorMP::Process | ( | const std::vector< std::string > & | fileNames, |
F | procFunc, | ||
TEntryList & | entries, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) | -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type |
Process a TTree dataset with a functor.
F | functor returning a pointer to TObject or inheriting classes and taking a TTreeReader& (both enforced at compile-time) |
Dataset definition:
[in] | fileNames | vector of strings with the paths of the files with the TTree to process |
[in] | fileName | string with the path of the files with the TTree to process |
[in] | collection | TFileCollection with the files with the TTree to process |
[in] | chain | TChain with the files with the TTree to process |
[in] | tree | TTree to process |
[in] | entries | TEntryList to filter the dataset |
[in] | treeName | Name of the TTree to process |
[in] | nToProcess | Number of entries to process (0 means all) |
[in] | jFirst | First entry to process (0 means the first of the first file) |
Definition at line 189 of file TTreeProcessorMP.hxx.
TList * ROOT::TTreeProcessorMP::Process | ( | const std::vector< std::string > & | fileNames, |
TSelector & | selector, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) |
Process a TTree dataset with a selector: version without entry list.
No TEntryList versions of selector processor.
Dataset definition:
[in] | fileNames | vector of strings with the paths of the files with the TTree to process |
[in] | fileName | string with the path of the files with the TTree to process |
[in] | collection | TFileCollection with the files with the TTree to process |
[in] | chain | TChain with the files with the TTree to process |
[in] | tree | TTree to process |
[in] | selector | Instance of TSelector to be applied to the dataset |
[in] | treeName | Name of the TTree to process |
[in] | nToProcess | Number of entries to process (0 means all) |
[in] | jFirst | First entry to process (0 means the first of the first file) |
Definition at line 288 of file TTreeProcessorMP.cxx.
TList * ROOT::TTreeProcessorMP::Process | ( | const std::vector< std::string > & | fileNames, |
TSelector & | selector, | ||
TEntryList & | entries, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) |
Process a TTree dataset with a selector.
TSelector-based tree processing: dataset as a vector of files.
Dataset definition:
[in] | fileNames | vector of strings with the paths of the files with the TTree to process |
[in] | fileName | string with the path of the files with the TTree to process |
[in] | collection | TFileCollection with the files with the TTree to process |
[in] | chain | TChain with the files with the TTree to process |
[in] | tree | TTree to process |
[in] | selector | Instance of TSelector to be applied to the dataset |
[in] | entries | TEntryList to filter the dataset |
[in] | treeName | Name of the TTree to process |
[in] | nToProcess | Number of entries to process (0 means all) |
[in] | jFirst | First entry to process (0 means the first of the first file) |
Definition at line 159 of file TTreeProcessorMP.cxx.
auto ROOT::TTreeProcessorMP::Process | ( | TChain & | chain, |
F | procFunc, | ||
TEntryList & | entries, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) | -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type |
Definition at line 280 of file TTreeProcessorMP.hxx.
auto ROOT::TTreeProcessorMP::Process | ( | TChain & | files, |
F | procFunc, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) | -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type |
Definition at line 383 of file TTreeProcessorMP.hxx.
TList * ROOT::TTreeProcessorMP::Process | ( | TChain & | files, |
TSelector & | selector, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) |
Definition at line 309 of file TTreeProcessorMP.cxx.
TList * ROOT::TTreeProcessorMP::Process | ( | TChain & | files, |
TSelector & | selector, | ||
TEntryList & | entries, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) |
TSelector-based tree processing: dataset as a TChain.
Definition at line 263 of file TTreeProcessorMP.cxx.
auto ROOT::TTreeProcessorMP::Process | ( | TFileCollection & | collection, |
F | procFunc, | ||
TEntryList & | entries, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) | -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type |
Definition at line 266 of file TTreeProcessorMP.hxx.
auto ROOT::TTreeProcessorMP::Process | ( | TFileCollection & | files, |
F | procFunc, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) | -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type |
Definition at line 373 of file TTreeProcessorMP.hxx.
TList * ROOT::TTreeProcessorMP::Process | ( | TFileCollection & | files, |
TSelector & | selector, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) |
Definition at line 302 of file TTreeProcessorMP.cxx.
TList * ROOT::TTreeProcessorMP::Process | ( | TFileCollection & | files, |
TSelector & | selector, | ||
TEntryList & | entries, | ||
const std::string & | treeName = "" , |
||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) |
TSelector-based tree processing: dataset as a TFileCollection.
Definition at line 249 of file TTreeProcessorMP.cxx.
auto ROOT::TTreeProcessorMP::Process | ( | TTree & | tree, |
F | procFunc, | ||
TEntryList & | entries, | ||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) | -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type |
Definition at line 295 of file TTreeProcessorMP.hxx.
auto ROOT::TTreeProcessorMP::Process | ( | TTree & | tree, |
F | procFunc, | ||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) | -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type |
Definition at line 393 of file TTreeProcessorMP.hxx.
TList * ROOT::TTreeProcessorMP::Process | ( | TTree & | tree, |
TSelector & | selector, | ||
TEntryList & | entries, | ||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) |
TSelector-based tree processing: memory resident tree.
Definition at line 96 of file TTreeProcessorMP.cxx.
TList * ROOT::TTreeProcessorMP::Process | ( | TTree & | tree, |
TSelector & | selector, | ||
ULong64_t | nToProcess = 0 , |
||
ULong64_t | jFirst = 0 |
||
) |
Definition at line 316 of file TTreeProcessorMP.cxx.
Reply to a worker who is idle.
If still events to process, tell the worker. Otherwise ask for a result
Definition at line 350 of file TTreeProcessorMP.cxx.
Reset TTreeProcessorMP's state.
Definition at line 339 of file TTreeProcessorMP.cxx.
|
inline |
Definition at line 162 of file TTreeProcessorMP.hxx.
|
private |
number of arguments already passed to the workers
Definition at line 173 of file TTreeProcessorMP.hxx.
|
private |
total number of arguments to pass to the workers
Definition at line 174 of file TTreeProcessorMP.hxx.
|
private |
the kind of task that is being executed, if any
Definition at line 185 of file TTreeProcessorMP.hxx.