This class provides a simple interface to execute the same task multiple times in parallel, possibly with different arguments every time.

ROOT::TProcessExecutor::Map

This class inherits its interfaces from ROOT::TExecutor
. The two possible usages of the Map method are:

Map(F func, unsigned nTimes): func is executed nTimes with no arguments
Map(F func, T& args): func is executed on each element of the collection of arguments args

For either signature, func is executed as many times as needed by a pool of fNWorkers workers; the number of workers can be passed to the constructor or set via SetNWorkers. It typically defaults to the number of cores.
A collection containing the result of each execution is returned.
Note: the user is responsible for the deletion of any object that might be created upon execution of func, returned objects included: ROOT::TProcessExecutor never deletes what it returns, it simply forgets it.
Note: that the usage of ROOT::TProcessExecutor::Map is indicated only when the task to be executed takes more than a few seconds, otherwise the overhead introduced by Map will outrun the benefits of parallel execution on most machines.

Parameters

func	a callable object, such as a lambda expression, an std::function, a functor object or a function that takes zero arguments (for the first signature) or one (for the second signature).
args	a standard vector, a ROOT::TSeq of integer type or an initializer list for the second signature. An integer only for the first.

Note: in cases where the function to be executed takes more than zero/one argument but all are fixed except zero/one, the function can be wrapped in a lambda or via std::bind to give it the right signature.
Note: the user should take care of initializing random seeds differently in each process (e.g. using the process id in the seed). Otherwise several parallel executions might generate the same sequence of pseudo-random numbers.

Return value:

An std::vector. The elements in the container will be the objects returned by func.

Examples:

root[] ROOT::TProcessExecutor pool; auto hists = pool.Map(CreateHisto, 10);

root[] ROOT::TProcessExecutor pool(2); auto squares = pool.Map([](int a) { return a*a; }, {1,2,3});

a

#define a(i)

Definition RSha256.hxx:99

ROOT::TExecutorCRTP::Map

auto Map(F func, unsigned nTimes) -> std::vector< InvokeResult_t< F > >

Execute a function without arguments several times.

Definition TExecutorCRTP.hxx:185

ROOT::TProcessExecutor

This class provides a simple interface to execute the same task multiple times in parallel,...

Definition TProcessExecutor.hxx:37

ROOT::TProcessExecutor::MapReduce

This set of methods behaves exactly like Map, but takes an additional function as a third argument. This function is applied to the set of objects returned by the corresponding Map execution to "squash" them into a single object.

Examples:

root[] ROOT::TProcessExecutor pool; auto ten = pool.MapReduce([]() { return 1; }, 10, [](const std::vector<int> &v) { return std::accumulate(v.begin(), v.end(), 0); })

root[] ROOT::TProcessExecutor pool; auto hist = pool.MapReduce(CreateAndFillHists, 10, PoolUtils::ReduceObjects);

ROOT::ExecutorUtils::ReduceObjects

Merge collection of TObjects.

Definition PoolUtils.h:35

ROOT::TProcessExecutor::MapReduce

auto MapReduce(F func, unsigned nTimes, R redfunc) -> InvokeResult_t< F >

Execute a function nTimes in parallel (Map) and accumulate the results into a single value (Reduce).

Definition TProcessExecutor.hxx:262

v

@ v

Definition rootcling_impl.cxx:3554

Definition at line 37 of file TProcessExecutor.hxx.

Public Member Functions
	TProcessExecutor (const TProcessExecutor &)=delete
	TProcessExecutor (unsigned nWorkers=0)
	Class constructor.
	~TProcessExecutor ()=default
unsigned	GetPoolSize () const
	Return the number of pooled parallel workers.
auto	Map (F func, unsigned nTimes) -> std::vector< InvokeResult_t< F > >
	Execute a function without arguments several times.
auto	MapReduce (F func, unsigned nTimes, R redfunc) -> InvokeResult_t< F >
	Execute a function without arguments several times (Map) and accumulate the results into a single value (Reduce).
template<class F, class T, class R, class Cond = validMapReturnCond<F, T>>
auto	MapReduce (F func, const std::vector< T > &args, R redfunc) -> InvokeResult_t< F, T >
	Execute a function in parallel over the elements of an immutable vector (Map) and accumulate the results into a single value (Reduce).
template<class F, class T, class R, class Cond = validMapReturnCond<F, T>>
auto	MapReduce (F func, std::vector< T > &args, R redfunc) -> InvokeResult_t< F, T >
	Execute a function in parallel over the elements of a vector (Map) and accumulate the results into a single value (Reduce).
template<class F, class R, class Cond = validMapReturnCond<F>>
auto	MapReduce (F func, unsigned nTimes, R redfunc) -> InvokeResult_t< F >
	Execute a function nTimes in parallel (Map) and accumulate the results into a single value (Reduce).
TProcessExecutor &	operator= (const TProcessExecutor &)=delete
T *	Reduce (const std::vector< T * > &mergeObjs)
	"Reduce" an std::vector into a single object by using the object's Merge method.
void	SetNWorkers (unsigned n)

Protected Types
using	InvokeResult_t
using	validMapReturnCond
	type definition used in templated functions for not allowing mapping functions that return references or void.

Private Types
enum class	ETask : unsigned char { kNoTask , kMap , kMapWithArg , kMapRed , kMapRedWithArg }
	A collection of the types of tasks that TProcessExecutor can execute. More...

Private Member Functions
template<class T>
unsigned	Broadcast (unsigned code, const std::vector< T > &objs)
	Send a message with a different object to each server.
template<class T>
unsigned	Broadcast (unsigned code, std::initializer_list< T > &objs)
	Send a message with a different object to each server.
template<class T>
unsigned	Broadcast (unsigned code, T obj, unsigned nMessages=0)
	Send a message containing code and obj to each worker, up to a maximum number of nMessages workers.
unsigned	Broadcast (unsigned code, unsigned nMessages=0)
	Send a message with the specified code to at most nMessages workers.
template<class T>
void	Collect (std::vector< T > &reslist)
	Listen for messages sent by the workers and call the appropriate handler function.
void	DeActivate (TSocket *s)
	DeActivate a certain socket.
TProcessExecutor &	Derived ()
bool	Fork (TMPWorker &server)
	This method forks the ROOT session into fNWorkers children processes.
bool	GetIsParent () const
TMonitor &	GetMonitor ()
unsigned	GetNWorkers () const
void	HandleMPCode (MPCodeBufPair &msg, TSocket *sender)
	Handle messages containing an EMPCode.
template<class T>
void	HandlePoolCode (MPCodeBufPair &msg, TSocket *sender, std::vector< T > &reslist)
	Handle message and reply to the worker.
template<class F, class T, class Cond = validMapReturnCond<F, T>>
auto	MapImpl (F func, const std::vector< T > &args) -> std::vector< InvokeResult_t< F, T > >
	Execute a function over the elements of an immutable vector in parallel Implementation of the Map method.
template<class F, class INTEGER, class Cond = validMapReturnCond<F, INTEGER>>
auto	MapImpl (F func, ROOT::TSeq< INTEGER > args) -> std::vector< InvokeResult_t< F, INTEGER > >
	Execute a function over a sequence of indexes in parallel.
template<class F, class T, class Cond = validMapReturnCond<F, T>>
auto	MapImpl (F func, std::vector< T > &args) -> std::vector< InvokeResult_t< F, T > >
	Execute a function over the elements of a vector in parallel Implementation of the Map method.
template<class F, class Cond = validMapReturnCond<F>>
auto	MapImpl (F func, unsigned nTimes) -> std::vector< InvokeResult_t< F > >
	Execute a function without arguments several times in parallel.
void	ReapWorkers ()
	Wait on worker processes and remove their pids from fWorkerPids.
void	Remove (TSocket *s)
	Remove a certain socket from the monitor.
void	ReplyToFuncResult (TSocket *s)
	Reply to a worker who just sent a result.
void	ReplyToIdle (TSocket *s)
	Reply to a worker who is idle.
void	Reset ()
	Reset TProcessExecutor's state.

Private Attributes
bool	fIsParent
	This is true if this is the parent/client process, false if this is a child/worker process.
TMonitor	fMon
	This object manages the sockets and detect socket events via TMonitor::Select.
unsigned	fNProcessed
	number of arguments already passed to the workers
unsigned	fNToProcess
	total number of arguments to pass to the workers
unsigned	fNWorkers
	The number of workers that should be spawned upon forking.
ETask	fTaskType = ETask::kNoTask
	the kind of task that is being executed, if any
std::vector< pid_t >	fWorkerPids
	A vector containing the PIDs of children processes/workers.
friend	TExecutorCRTP

#include <ROOT/TProcessExecutor.hxx>

Inheritance diagram for ROOT::TProcessExecutor:

Member Typedef Documentation

◆ InvokeResult_t

using ROOT::TExecutorCRTP< TProcessExecutor >::InvokeResult_t

protectedinherited

Definition at line 107 of file TExecutorCRTP.hxx.

◆ validMapReturnCond

using ROOT::TExecutorCRTP< TProcessExecutor >::validMapReturnCond

protectedinherited

type definition used in templated functions for not allowing mapping functions that return references or void.

The resulting vector elements must be assignable, references aren't.

Definition at line 112 of file TExecutorCRTP.hxx.

Member Enumeration Documentation

◆ ETask

enum class ROOT::TProcessExecutor::ETask : unsigned char

strongprivate

A collection of the types of tasks that TProcessExecutor can execute.

It is used to interpret in the right way and properly reply to the messages received (see, for example, TProcessExecutor::HandleInput)

Enumerator
kNoTask	no task is being executed
kMap	a Map method with no arguments is being executed
kMapWithArg	a Map method with arguments is being executed
kMapRed	a MapReduce method with no arguments is being executed
kMapRedWithArg	a MapReduce method with arguments is being executed

Definition at line 99 of file TProcessExecutor.hxx.

Constructor & Destructor Documentation

◆ TProcessExecutor() [1/2]

ROOT::TProcessExecutor::TProcessExecutor ( unsigned nWorkers = 0 )

explicit

Class constructor.

Parameters

nWorkers Number of times this ROOT session will be forked, i.e. the number of workers that will be spawned.

Definition at line 89 of file TProcessExecutor.cxx.

◆ ~TProcessExecutor()

ROOT::TProcessExecutor::~TProcessExecutor ( )

default

◆ TProcessExecutor() [2/2]

ROOT::TProcessExecutor::TProcessExecutor ( const TProcessExecutor & )

delete

Member Function Documentation

◆ Broadcast() [1/4]

template<class T>

unsigned TMPClient::Broadcast	(	unsigned	code,
		const std::vector< T > &	args )

inherited

Send a message with a different object to each server.

Sockets can either be in an "active" or "non-active" state. This method activates all the sockets through which the client is connected to the workers, and deactivates them when a message is sent to the corresponding worker. This way the sockets pertaining to workers who have been left idle will be the only ones in the active list (TSocket::GetMonitor()->GetListOfActives()) after execution.

Parameters

code	the code of the message to send (e.g. EMPCode)
args	a vector containing the different messages to be sent. If the size of the vector is smaller than the number of workers, a message will be sent only to the first args.size() workers. If the size of the args vector is bigger than the number of workers, only the first fNWorkers arguments will be sent.

Returns: the number of messages successfully sent

Definition at line 73 of file TMPClient.h.

◆ Broadcast() [2/4]

template<class T>

unsigned TMPClient::Broadcast	(	unsigned	code,
		std::initializer_list< T > &	args )

inherited

Send a message with a different object to each server.

See TMPClient::Broadcast(unsigned code, const std::vector<T> &args) for more informations.

Definition at line 100 of file TMPClient.h.

◆ Broadcast() [3/4]

template<class T>

unsigned TMPClient::Broadcast	(	unsigned	code,
		T	obj,
		unsigned	nMessages = 0 )

inherited

Send a message containing code and obj to each worker, up to a maximum number of nMessages workers.

See Broadcast(unsigned code, unsigned nMessages) for more informations.

Parameters

code	the code of the message to send (e.g. EMPCode)
obj	the object to send
nMessages	the maximum number of messages to send. If nMessages == 0, send a message to every worker.

Returns: the number of messages successfully sent

Definition at line 120 of file TMPClient.h.

◆ Broadcast() [4/4]

unsigned TMPClient::Broadcast	(	unsigned	code,
		unsigned	nMessages = 0 )

inherited

Send a message with the specified code to at most nMessages workers.

Sockets can either be in an "active" or "non-active" state. This method activates all the sockets through which the client is connected to the workers, and deactivates them when a message is sent to the corresponding worker. This way the sockets pertaining to workers who have been left idle will be the only ones in the active list (TSocket::GetMonitor()->GetListOfActives()) after execution.

Parameters

code	the code to send (e.g. EMPCode)
nMessages	the maximum number of messages to send. If nMessages == 0 \|\| nMessage > fNWorkers, send a message to every worker.

Returns: the number of messages successfully sent

Definition at line 253 of file TMPClient.cxx.

◆ Collect()

template<class T>

void ROOT::TProcessExecutor::Collect ( std::vector< T > & reslist )

private

Listen for messages sent by the workers and call the appropriate handler function.

TProcessExecutor::HandlePoolCode is called on messages with a code < 1000 and TMPClient::HandleMPCode is called on messages with a code >= 1000.

Definition at line 412 of file TProcessExecutor.hxx.

◆ DeActivate()

void TMPClient::DeActivate ( TSocket * s )

inherited

DeActivate a certain socket.

This does not remove it from the monitor: it will be reactivated by the next call to Broadcast() (or possibly other methods that are specified to do so).
A socket should be DeActivated when the corresponding worker is done for now and we want to stop listening to this worker's socket. If the worker is done forever, Remove() should be used instead.

Parameters

s	the socket to be deactivated

Definition at line 286 of file TMPClient.cxx.

◆ Derived()

TProcessExecutor & ROOT::TExecutorCRTP< TProcessExecutor >::Derived ( )

inlineprivateinherited

Definition at line 156 of file TExecutorCRTP.hxx.

◆ Fork()

bool TMPClient::Fork ( TMPWorker & server )

inherited

This method forks the ROOT session into fNWorkers children processes.

The ROOT sessions spawned in this way will not have graphical capabilities and will not read from standard input, but will be connected to the original (interactive) session through TSockets. The children processes' PIDs are added to the fWorkerPids vector. The parent session can then communicate with the children using the Broadcast and MPSend methods, and receive messages through MPRecv.

Parameters

server A pointer to an instance of the class that will take control of the subprocesses after forking. Applications should implement their own class inheriting from TMPWorker. Behaviour can be customized overriding TMPWorker::HandleInput.

Returns: true if Fork succeeded, false otherwise

Definition at line 128 of file TMPClient.cxx.

◆ GetIsParent()

bool TMPClient::GetIsParent ( ) const

inlineinherited

Definition at line 37 of file TMPClient.h.

◆ GetMonitor()

TMonitor & TMPClient::GetMonitor ( )

inlineinherited

Definition at line 36 of file TMPClient.h.

◆ GetNWorkers()

unsigned TMPClient::GetNWorkers ( ) const

inlineinherited

Definition at line 40 of file TMPClient.h.

◆ GetPoolSize()

unsigned ROOT::TProcessExecutor::GetPoolSize ( ) const

inline

Return the number of pooled parallel workers.

Returns: The number of workers in the pool.

Definition at line 72 of file TProcessExecutor.hxx.

◆ HandleMPCode()

void TMPClient::HandleMPCode	(	MPCodeBufPair &	msg,
		TSocket *	s )

inherited

Handle messages containing an EMPCode.

This method should be called upon receiving a message with a code >= 1000 (i.e. EMPCode). It handles the most generic types of messages.
Classes inheriting from TMPClient should implement a similar method to handle message codes specific to the application they're part of.

Parameters

msg	the MPCodeBufPair returned by a MPRecv call
s	a pointer to the socket from which the message has been received is passed. This way HandleMPCode knows which socket to reply on.

Definition at line 334 of file TMPClient.cxx.

◆ HandlePoolCode()

template<class T>

void ROOT::TProcessExecutor::HandlePoolCode	(	MPCodeBufPair &	msg,
		TSocket *	sender,
		std::vector< T > &	reslist )

private

Handle message and reply to the worker.

Definition at line 383 of file TProcessExecutor.hxx.

◆ Map()

auto ROOT::TExecutorCRTP< TProcessExecutor >::Map	(	F	func,
		unsigned	nTimes )-> std::vector< InvokeResult_t< F > >

inherited

Execute a function without arguments several times.

Parameters

func	Function to be executed.
nTimes	Number of times function should be called.

Returns: A vector with the results of the function calls. Functions that take arguments can be executed (with fixed arguments) by wrapping them in a lambda or with std::bind.

Definition at line 123 of file TExecutorCRTP.hxx.

◆ MapImpl() [1/4]

template<class F, class T, class Cond>

auto ROOT::TProcessExecutor::MapImpl	(	F	func,
		const std::vector< T > &	args ) -> std::vector< InvokeResult_t< F, T > >

private

Execute a function over the elements of an immutable vector in parallel Implementation of the Map method.

Definition at line 205 of file TProcessExecutor.hxx.

◆ MapImpl() [2/4]

template<class F, class INTEGER, class Cond>

auto ROOT::TProcessExecutor::MapImpl	(	F	func,
		ROOT::TSeq< INTEGER >	args ) -> std::vector< InvokeResult_t< F, INTEGER > >

private

Execute a function over a sequence of indexes in parallel.

Implementation of the Map method.

Definition at line 250 of file TProcessExecutor.hxx.

◆ MapImpl() [3/4]

template<class F, class T, class Cond>

auto ROOT::TProcessExecutor::MapImpl	(	F	func,
		std::vector< T > &	args ) -> std::vector< InvokeResult_t< F, T > >

private

Execute a function over the elements of a vector in parallel Implementation of the Map method.

Definition at line 160 of file TProcessExecutor.hxx.

◆ MapImpl() [4/4]

template<class F, class Cond>

auto ROOT::TProcessExecutor::MapImpl	(	F	func,
		unsigned	nTimes ) -> std::vector< InvokeResult_t< F > >

private

Execute a function without arguments several times in parallel.

Implementation of the Map method.

Definition at line 119 of file TProcessExecutor.hxx.

◆ MapReduce() [1/4]

auto ROOT::TExecutorCRTP< TProcessExecutor >::MapReduce	(	F	func,
		unsigned	nTimes,
		R	redfunc )-> InvokeResult_t< F >

inherited

Execute a function without arguments several times (Map) and accumulate the results into a single value (Reduce).

Parameters

func	Function to be executed.
nTimes	Number of times function should be called.

Returns: A vector with the results of the function calls.

Parameters

redfunc Reduction function to combine the results of the calls to func. Must return the same type as func.

Returns: A value result of "reducing" the vector returned by the Map operation into a single object.

Definition at line 137 of file TExecutorCRTP.hxx.

◆ MapReduce() [2/4]

template<class F, class T, class R, class Cond>

auto ROOT::TProcessExecutor::MapReduce	(	F	func,
		const std::vector< T > &	args,
		R	redfunc ) -> InvokeResult_t< F, T >

Execute a function in parallel over the elements of an immutable vector (Map) and accumulate the results into a single value (Reduce).

Benefits from partial reduction into nChunks intermediate results.

Definition at line 344 of file TProcessExecutor.hxx.

◆ MapReduce() [3/4]

template<class F, class T, class R, class Cond>

auto ROOT::TProcessExecutor::MapReduce	(	F	func,
		std::vector< T > &	args,
		R	redfunc ) -> InvokeResult_t< F, T >

Execute a function in parallel over the elements of a vector (Map) and accumulate the results into a single value (Reduce).

Benefits from partial reduction into nChunks intermediate results.

Definition at line 302 of file TProcessExecutor.hxx.

◆ MapReduce() [4/4]

template<class F, class R, class Cond>

auto ROOT::TProcessExecutor::MapReduce	(	F	func,
		unsigned	nTimes,
		R	redfunc ) -> InvokeResult_t< F >

Execute a function nTimes in parallel (Map) and accumulate the results into a single value (Reduce).

Definition at line 262 of file TProcessExecutor.hxx.

◆ operator=()

TProcessExecutor & ROOT::TProcessExecutor::operator= ( const TProcessExecutor & )

delete

◆ ReapWorkers()

void TMPClient::ReapWorkers ( )

inherited

Wait on worker processes and remove their pids from fWorkerPids.

A blocking waitpid is called, but this should actually not block execution since ReapWorkers should only be called when all workers have already quit. ReapWorkers is then called not to leave zombie processes hanging around, and to clean-up fWorkerPids.

Definition at line 313 of file TMPClient.cxx.

◆ Reduce()

T * ROOT::TExecutorCRTP< TProcessExecutor >::Reduce ( const std::vector< T * > & mergeObjs )

inherited

"Reduce" an std::vector into a single object by using the object's Merge method.

Parameters

mergeObjs A vector of ROOT objects implementing the Merge method

Returns: An object result of merging the vector elements into one.

Definition at line 151 of file TExecutorCRTP.hxx.

◆ Remove()

void TMPClient::Remove ( TSocket * s )

inherited

Remove a certain socket from the monitor.

A socket should be Removed from the monitor when the corresponding worker is done forever. For example HandleMPCode() calls this method on sockets pertaining to workers which sent an MPCode::kShutdownNotice.
If the worker is done for now, DeActivate should be used instead.

Parameters

s	the socket to be removed from the monitor fMon

Definition at line 300 of file TMPClient.cxx.

◆ ReplyToFuncResult()

void ROOT::TProcessExecutor::ReplyToFuncResult ( TSocket * s )

private

Reply to a worker who just sent a result.

If another argument to process exists, tell the worker. Otherwise send a shutdown order.

Definition at line 107 of file TProcessExecutor.cxx.

◆ ReplyToIdle()

void ROOT::TProcessExecutor::ReplyToIdle ( TSocket * s )

friend ROOT::TProcessExecutor::TExecutorCRTP

private

Definition at line 38 of file TProcessExecutor.hxx.

The documentation for this class was generated from the following files:

core/multiproc/inc/ROOT/TProcessExecutor.hxx
core/multiproc/src/TProcessExecutor.cxx

ROOT::TProcessExecutor::Map

Return value:

Examples:

ROOT::TProcessExecutor::MapReduce

Examples:

Public Member Functions

Protected Types

Private Types

Private Member Functions

Private Attributes

Member Typedef Documentation

◆ InvokeResult_t

◆ validMapReturnCond

Member Enumeration Documentation

◆ ETask

Constructor & Destructor Documentation

◆ TProcessExecutor() [1/2]

◆ ~TProcessExecutor()

◆ TProcessExecutor() [2/2]

Member Function Documentation

◆ Broadcast() [1/4]

◆ Broadcast() [2/4]

◆ Broadcast() [3/4]

◆ Broadcast() [4/4]

◆ Collect()

◆ DeActivate()

◆ Derived()

◆ Fork()

◆ GetIsParent()

◆ GetMonitor()

◆ GetNWorkers()

◆ GetPoolSize()

◆ HandleMPCode()

◆ HandlePoolCode()

◆ Map()

◆ MapImpl() [1/4]

◆ MapImpl() [2/4]

◆ MapImpl() [3/4]

◆ MapImpl() [4/4]

◆ MapReduce() [1/4]

◆ MapReduce() [2/4]

◆ MapReduce() [3/4]

◆ MapReduce() [4/4]

◆ operator=()

◆ ReapWorkers()

◆ Reduce()

◆ Remove()

◆ ReplyToFuncResult()

◆ ReplyToIdle()

◆ Reset()

◆ SetNWorkers()

Member Data Documentation

◆ fIsParent

◆ fMon

◆ fNProcessed

◆ fNToProcess

◆ fNWorkers

◆ fTaskType

◆ fWorkerPids

◆ TExecutorCRTP