ROOT::TProcessExecutor Class Reference

This class provides a simple interface to execute the same task multiple times in parallel, possibly with different arguments every time.

ROOT::TProcessExecutor::Map

This class inherits its interfaces from ROOT::TExecutor
. The two possible usages of the Map method are:

• Map(F func, unsigned nTimes): func is executed nTimes with no arguments
• Map(F func, T& args): func is executed on each element of the collection of arguments args

For either signature, func is executed as many times as needed by a pool of fNWorkers workers; the number of workers can be passed to the constructor or set via SetNWorkers. It typically defaults to the number of cores.
A collection containing the result of each execution is returned.
Note: the user is responsible for the deletion of any object that might be created upon execution of func, returned objects included: ROOT::TProcessExecutor never deletes what it returns, it simply forgets it.
Note: that the usage of ROOT::TProcessExecutor::Map is indicated only when the task to be executed takes more than a few seconds, otherwise the overhead introduced by Map will outrun the benefits of parallel execution on most machines.

Parameters

func	a callable object, such as a lambda expression, an std::function, a functor object or a function that takes zero arguments (for the first signature) or one (for the second signature).
args	a standard vector, a ROOT::TSeq of integer type or an initializer list for the second signature. An integer only for the first.

Note: in cases where the function to be executed takes more than zero/one argument but all are fixed except zero/one, the function can be wrapped in a lambda or via std::bind to give it the right signature.
Note: the user should take care of initializing random seeds differently in each process (e.g. using the process id in the seed). Otherwise several parallel executions might generate the same sequence of pseudo-random numbers.

Return value:

An std::vector. The elements in the container will be the objects returned by func.

Examples:

root[] ROOT::TProcessExecutor pool; auto hists = pool.Map(CreateHisto, 10);
root[] ROOT::TProcessExecutor pool(2); auto squares = pool.Map([](int a) { return a*a; }, {1,2,3});

ROOT::TProcessExecutor::MapReduce

This set of methods behaves exactly like Map, but takes an additional function as a third argument. This function is applied to the set of objects returned by the corresponding Map execution to "squash" them into a single object.

Examples:

root[] ROOT::TProcessExecutor pool; auto ten = pool.MapReduce([]() { return 1; }, 10, [](const std::vector<int> &v) { return std::accumulate(v.begin(), v.end(), 0); })
root[] ROOT::TProcessExecutor pool; auto hist = pool.MapReduce(CreateAndFillHists, 10, PoolUtils::ReduceObjects);

Definition at line 37 of file TProcessExecutor.hxx.

Public Member Functions
	TProcessExecutor (const TProcessExecutor &)=delete
	TProcessExecutor (unsigned nWorkers=0)
	Class constructor.
	~TProcessExecutor ()=default
unsigned	GetPoolSize () const
	Return the number of pooled parallel workers.
template<class F , class T , class R , class Cond = noReferenceCond<F, T>>
auto	MapReduce (F func, const std::vector< T > &args, R redfunc) -> typename std::result_of< F(T)>::type
	Execute a function in parallel over the elements of an immutable vector (Map) and accumulate the results into a single value (Reduce).
template<class F , class T , class R , class Cond = noReferenceCond<F, T>>
auto	MapReduce (F func, std::vector< T > &args, R redfunc) -> typename std::result_of< F(T)>::type
	Execute a function in parallel over the elements of a vector (Map) and accumulate the results into a single value (Reduce).
template<class F , class R , class Cond = noReferenceCond<F>>
auto	MapReduce (F func, unsigned nTimes, R redfunc) -> typename std::result_of< F()>::type
	Execute a function nTimes in parallel (Map) and accumulate the results into a single value (Reduce).
TProcessExecutor &	operator= (const TProcessExecutor &)=delete
void	SetNWorkers (unsigned n)
Public Member Functions inherited from ROOT::TExecutorCRTP< TProcessExecutor >
	TExecutorCRTP ()=default
	TExecutorCRTP (const TExecutorCRTP &)=delete
auto	Map (F func, const std::vector< T > &args) -> std::vector< typename std::result_of< F(T)>::type >
	Execute a function over the elements of an immutable vector.
auto	Map (F func, ROOT::TSeq< INTEGER > args) -> std::vector< typename std::result_of< F(INTEGER)>::type >
	Execute a function over a sequence of indexes.
auto	Map (F func, std::initializer_list< T > args) -> std::vector< typename std::result_of< F(T)>::type >
	Execute a function over the elements of an initializer_list.
auto	Map (F func, std::vector< T > &args) -> std::vector< typename std::result_of< F(T)>::type >
	Execute a function over the elements of a vector.
auto	Map (F func, unsigned nTimes) -> std::vector< typename std::result_of< F()>::type >
	Execute a function without arguments several times.
T *	MapReduce (F func, const std::vector< T * > &args)
	Execute a function over the TObject-inheriting elements of an immutable vector (Map) and merge the objects into a single one (Reduce).
auto	MapReduce (F func, const std::vector< T > &args, R redfunc) -> typename std::result_of< F(T)>::type
	Execute a function over the elements of an immutable vector (Map) and accumulate the results into a single value (Reduce).
auto	MapReduce (F func, ROOT::TSeq< INTEGER > args, R redfunc) -> typename std::result_of< F(INTEGER)>::type
	Execute a function over a sequence of indexes (Map) and accumulate the results into a single value (Reduce).
auto	MapReduce (F func, std::initializer_list< T > args, R redfunc) -> typename std::result_of< F(T)>::type
	Execute a function over the elements of an initializer_list (Map) and accumulate the results into a single value (Reduce).
T *	MapReduce (F func, std::vector< T * > &args)
	Execute a function over the TObject-inheriting elements of a vector (Map) and merge the objects into a single one (Reduce).
auto	MapReduce (F func, std::vector< T > &args, R redfunc) -> typename std::result_of< F(T)>::type
	Execute a function over the elements of a vector (Map) and accumulate the results into a single value (Reduce).
auto	MapReduce (F func, unsigned nTimes, R redfunc) -> typename std::result_of< F()>::type
	Execute a function without arguments several times (Map) and accumulate the results into a single value (Reduce).
TExecutorCRTP &	operator= (const TExecutorCRTP &)=delete
T *	Reduce (const std::vector< T * > &mergeObjs)
	"Reduce" an std::vector into a single object by using the object's Merge method.
auto	Reduce (const std::vector< T > &objs, R redfunc) -> decltype(redfunc(objs))
	"Reduce" an std::vector into a single object by passing a function as the second argument defining the reduction operation.

Private Types
enum class	ETask : unsigned char {   kNoTask , kMap , kMapWithArg , kMapRed ,   kMapRedWithArg }
	A collection of the types of tasks that TProcessExecutor can execute. More...

Private Member Functions
template<class T >
void	Collect (std::vector< T > &reslist)
	Listen for messages sent by the workers and call the appropriate handler function.
template<class T >
void	HandlePoolCode (MPCodeBufPair &msg, TSocket *sender, std::vector< T > &reslist)
	Handle message and reply to the worker.
template<class F , class T , class Cond = noReferenceCond<F, T>>
auto	MapImpl (F func, const std::vector< T > &args) -> std::vector< typename std::result_of< F(T)>::type >
	Execute a function over the elements of an immutable vector in parallel Implementation of the Map method.
template<class F , class INTEGER , class Cond = noReferenceCond<F, INTEGER>>
auto	MapImpl (F func, ROOT::TSeq< INTEGER > args) -> std::vector< typename std::result_of< F(INTEGER)>::type >
	Execute a function over a sequence of indexes in parallel.
template<class F , class T , class Cond = noReferenceCond<F, T>>
auto	MapImpl (F func, std::vector< T > &args) -> std::vector< typename std::result_of< F(T)>::type >
	Execute a function over the elements of a vector in parallel Implementation of the Map method.
template<class F , class Cond = noReferenceCond<F>>
auto	MapImpl (F func, unsigned nTimes) -> std::vector< typename std::result_of< F()>::type >
	Execute a function without arguments several times in parallel.
void	ReplyToFuncResult (TSocket *s)
	Reply to a worker who just sent a result.
void	ReplyToIdle (TSocket *s)
	Reply to a worker who is idle.
void	Reset ()
	Reset TProcessExecutor's state.
Private Member Functions inherited from TMPClient
	TMPClient (const TMPClient &)=delete
	TMPClient (unsigned nWorkers=0)
	Class constructor.
	~TMPClient ()
	Class destructor.
template<class T >
unsigned	Broadcast (unsigned code, const std::vector< T > &objs)
	Send a message with a different object to each server.
template<class T >
unsigned	Broadcast (unsigned code, std::initializer_list< T > &objs)
	Send a message with a different object to each server.
template<class T >
unsigned	Broadcast (unsigned code, T obj, unsigned nMessages=0)
	Send a message containing code and obj to each worker, up to a maximum number of nMessages workers.
unsigned	Broadcast (unsigned code, unsigned nMessages=0)
	Send a message with the specified code to at most nMessages workers.
void	DeActivate (TSocket *s)
	DeActivate a certain socket.
bool	Fork (TMPWorker &server)
	This method forks the ROOT session into fNWorkers children processes.
bool	GetIsParent () const
TMonitor &	GetMonitor ()
unsigned	GetNWorkers () const
void	HandleMPCode (MPCodeBufPair &msg, TSocket *sender)
	Handle messages containing an EMPCode.
TMPClient &	operator= (const TMPClient &)=delete
void	ReapWorkers ()
	Wait on worker processes and remove their pids from fWorkerPids.
void	Remove (TSocket *s)
	Remove a certain socket from the monitor.
void	SetNWorkers (unsigned n)
	Set the number of workers that will be spawned by the next call to Fork()

Private Attributes
unsigned	fNProcessed
	number of arguments already passed to the workers
unsigned	fNToProcess
	total number of arguments to pass to the workers
ETask	fTaskType = ETask::kNoTask
	the kind of task that is being executed, if any
friend	TExecutorCRTP

Additional Inherited Members
Public Types inherited from ROOT::TExecutorCRTP< TProcessExecutor >
using	noReferenceCond = typename std::enable_if<"Function can't return a reference" &&!(std::is_reference< typename std::result_of< F(T...)>::type >::value)>::type
	type definition in used in templated functions for not allowing mapping functions that return references.

#include <ROOT/TProcessExecutor.hxx>

Inheritance diagram for ROOT::TProcessExecutor:

Member Enumeration Documentation

ETask

enum class ROOT::TProcessExecutor::ETask : unsigned char

strongprivate

A collection of the types of tasks that TProcessExecutor can execute.

It is used to interpret in the right way and properly reply to the messages received (see, for example, TProcessExecutor::HandleInput)

Enumerator
kNoTask	no task is being executed
kMap	a Map method with no arguments is being executed
kMapWithArg	a Map method with arguments is being executed
kMapRed	a MapReduce method with no arguments is being executed
kMapRedWithArg	a MapReduce method with arguments is being executed

Definition at line 98 of file TProcessExecutor.hxx.

Constructor & Destructor Documentation

TProcessExecutor() [1/2]

ROOT::TProcessExecutor::TProcessExecutor ( unsigned  nWorkers = 0 )

explicit

Class constructor.

Parameters

nWorkers

Number of times this ROOT session will be forked, i.e. the number of workers that will be spawned.

Definition at line 90 of file TProcessExecutor.cxx.

~TProcessExecutor()

ROOT::TProcessExecutor::~TProcessExecutor ( )

default

TProcessExecutor() [2/2]

ROOT::TProcessExecutor::TProcessExecutor ( const TProcessExecutor &  )

delete

Member Function Documentation

Collect()

template<class T >

void ROOT::TProcessExecutor::Collect ( std::vector< T > &  reslist )

private

Listen for messages sent by the workers and call the appropriate handler function.

TProcessExecutor::HandlePoolCode is called on messages with a code < 1000 and TMPClient::HandleMPCode is called on messages with a code >= 1000.

Definition at line 411 of file TProcessExecutor.hxx.

GetPoolSize()

unsigned ROOT::TProcessExecutor::GetPoolSize ( ) const

inline

Return the number of pooled parallel workers.

Returns

The number of workers in the pool.

Definition at line 71 of file TProcessExecutor.hxx.

HandlePoolCode()

template<class T >

void ROOT::TProcessExecutor::HandlePoolCode ( MPCodeBufPair &  msg, TSocket *  sender, std::vector< T > &  reslist  )

private

Handle message and reply to the worker.

Definition at line 382 of file TProcessExecutor.hxx.

MapImpl() [1/4]

template<class F , class T , class Cond >

auto ROOT::TProcessExecutor::MapImpl ( F  func, const std::vector< T > &  args  ) -> std::vector<typename std::result_of<F(T)>::type>

private

Execute a function over the elements of an immutable vector in parallel Implementation of the Map method.

Parameters

func	Function to be executed on the elements of the vector passed as second parameter.
args	Vector of elements passed as an argument to func.

Returns

A vector with the results of the function calls.

Definition at line 204 of file TProcessExecutor.hxx.

MapImpl() [2/4]

template<class F , class INTEGER , class Cond >

auto ROOT::TProcessExecutor::MapImpl ( F  func, ROOT::TSeq< INTEGER >  args  ) -> std::vector<typename std::result_of<F(INTEGER)>::type>

private

Execute a function over a sequence of indexes in parallel.

Implementation of the Map method.

Parameters

func	Function to be executed. Must take an element of the sequence passed assecond argument as a parameter.
args	Sequence of indexes to execute func on.

Returns

A vector with the results of the function calls.

Definition at line 249 of file TProcessExecutor.hxx.

MapImpl() [3/4]

template<class F , class T , class Cond >

auto ROOT::TProcessExecutor::MapImpl ( F  func, std::vector< T > &  args  ) -> std::vector<typename std::result_of<F(T)>::type>

private

Execute a function over the elements of a vector in parallel Implementation of the Map method.

Parameters

func	Function to be executed on the elements of the vector passed as second parameter.
args	Vector of elements passed as an argument to func.

Returns

A vector with the results of the function calls.

Definition at line 159 of file TProcessExecutor.hxx.

MapImpl() [4/4]

template<class F , class Cond >

auto ROOT::TProcessExecutor::MapImpl ( F  func, unsigned  nTimes  ) -> std::vector<typename std::result_of<F()>::type>

private

Execute a function without arguments several times in parallel.

Implementation of the Map method.

Parameters

func	Function to be executed.
nTimes	Number of times function should be called.

Returns

A vector with the results of the function calls. Functions that take arguments can be executed (with fixed arguments) by wrapping them in a lambda or with std::bind.

Definition at line 118 of file TProcessExecutor.hxx.

MapReduce() [1/3]

template<class F , class T , class R , class Cond >

auto ROOT::TProcessExecutor::MapReduce	(	F	func,
		const std::vector< T > &	args,
		R	redfunc
	)		-> typename std::result_of<F(T)>::type

Execute a function in parallel over the elements of an immutable vector (Map) and accumulate the results into a single value (Reduce).

Benefits from partial reduction into nChunks intermediate results.

Benefits from partial reduction into nChunks intermediate results if the execution policy is multithreaded. Otherwise, it ignores the nChunks argument and performs a normal MapReduce operation.

Parameters

func	Function to be executed. Must take an element of the sequence passed assecond argument as a parameter.
args	Immutable vector, whose elements are passed as an argument to func.
redfunc	Reduction function to combine the results of the calls to func into partial results, and these into a final result. Must return the same type as func.
nChunks	Number of chunks to split the input data for processing.

Returns

A value result of "reducing" the vector returned by the Map operation into a single object.

Definition at line 343 of file TProcessExecutor.hxx.

MapReduce() [2/3]

template<class F , class T , class R , class Cond >

auto ROOT::TProcessExecutor::MapReduce	(	F	func,
		std::vector< T > &	args,
		R	redfunc
	)		-> typename std::result_of<F(T)>::type

Execute a function in parallel over the elements of a vector (Map) and accumulate the results into a single value (Reduce).

Benefits from partial reduction into nChunks intermediate results.

Benefits from partial reduction into nChunks intermediate results if the execution policy is multithreaded. Otherwise, it ignores the nChunks argument and performs a normal MapReduce operation.

Parameters

func	Function to be executed. Must take an element of the sequence passed assecond argument as a parameter.
args	Vector of elements passed as an argument to func.
redfunc	Reduction function to combine the results of the calls to func into partial results, and these into a final result. Must return the same type as func.
nChunks	Number of chunks to split the input data for processing.

Returns

A value result of "reducing" the vector returned by the Map operation into a single object.

Definition at line 301 of file TProcessExecutor.hxx.

MapReduce() [3/3]

template<class F , class R , class Cond >

auto ROOT::TProcessExecutor::MapReduce	(	F	func,
		unsigned	nTimes,
		R	redfunc
	)		-> typename std::result_of<F()>::type

Execute a function nTimes in parallel (Map) and accumulate the results into a single value (Reduce).

Parameters

func	Function to be executed.
nTimes	Number of times function should be called.

Returns

A vector with the results of the function calls.

Parameters

redfunc

Reduction function to combine the results of the calls to func. Must return the same type as func.

Returns

A value result of "reducing" the vector returned by the Map operation into a single object.

Definition at line 261 of file TProcessExecutor.hxx.

operator=()

TProcessExecutor & ROOT::TProcessExecutor::operator= ( const TProcessExecutor &  )

delete

ReplyToFuncResult()

void ROOT::TProcessExecutor::ReplyToFuncResult ( TSocket *  s )

private

Reply to a worker who just sent a result.

If another argument to process exists, tell the worker. Otherwise send a shutdown order.

Definition at line 108 of file TProcessExecutor.cxx.

ReplyToIdle()

void ROOT::TProcessExecutor::ReplyToIdle ( TSocket *  s )

private

Reply to a worker who is idle.

If another argument to process exists, tell the worker. Otherwise ask for a result

Definition at line 126 of file TProcessExecutor.cxx.

Reset()

void ROOT::TProcessExecutor::Reset ( )

private

Reset TProcessExecutor's state.

Definition at line 97 of file TProcessExecutor.cxx.

SetNWorkers()

void ROOT::TProcessExecutor::SetNWorkers ( unsigned  n )

inline

Definition at line 65 of file TProcessExecutor.hxx.

Member Data Documentation

fNProcessed

unsigned ROOT::TProcessExecutor::fNProcessed

private

number of arguments already passed to the workers

Definition at line 92 of file TProcessExecutor.hxx.

fNToProcess

unsigned ROOT::TProcessExecutor::fNToProcess

private

total number of arguments to pass to the workers

Definition at line 93 of file TProcessExecutor.hxx.

fTaskType

ETask ROOT::TProcessExecutor::fTaskType = ETask::kNoTask

private

the kind of task that is being executed, if any

Definition at line 106 of file TProcessExecutor.hxx.

TExecutorCRTP

friend ROOT::TProcessExecutor::TExecutorCRTP

private

Definition at line 38 of file TProcessExecutor.hxx.

Libraries for ROOT::TProcessExecutor:

---

The documentation for this class was generated from the following files:

• core/multiproc/inc/ROOT/TProcessExecutor.hxx
• core/multiproc/src/TProcessExecutor.cxx