Class to perform cross validation, splitting the dataloader into folds.

Use html for explicit line breaking
Markdown links? class reference?

ce->BookMethod(dataloader, options);

ce->Evaluate();

Cross-evaluation will generate a new training and a test set dynamically from from K folds. These K folds are generated by splitting the input training set. The input test set is currently ignored.

This means that when you specify your DataSet you should include all events in your training set. One way of doing this would be the following:

dataloader->AddTree( signalTree, "cls1" );
dataloader->AddTree( background, "cls2" );
dataloader->PrepareTrainingAndTestTree( "", "", "nTest_cls1=1:nTest_cls2=1" );

Split Expression

See CVSplit documentation?

Definition at line 124 of file CrossValidation.h.

Public Member Functions
	CrossValidation (TString jobName, TMVA::DataLoader dataloader, TFile outputFile, TString options)

	CrossValidation (TString jobName, TMVA::DataLoader *dataloader, TString options)

	~CrossValidation ()

void	Evaluate ()
	Does training, test set evaluation and performance evaluation of using cross-evalution.

Factory &	GetFactory ()

UInt_t	GetNumFolds ()

const std::vector< CrossValidationResult > &	GetResults () const

TString	GetSplitExpr ()

void	InitOptions ()

void	ParseOptions ()
	Method to parse the internal option string.

void	SetNumFolds (UInt_t i)

void	SetSplitExpr (TString splitExpr)

Public Member Functions inherited from TMVA::Envelope
	~Envelope ()
	Default destructor.

virtual void	BookMethod (TString methodname, TString methodtitle, TString options="")
	Method to book the machine learning method to perform the algorithm.

virtual void	BookMethod (Types::EMVA method, TString methodtitle, TString options="")
	Method to book the machine learning method to perform the algorithm.

DataLoader *	GetDataLoader ()
	Method to get the pointer to TMVA::DataLoader object.

TFile *	GetFile ()
	Method to get the pointer to TFile object.

std::vector< OptionMap > &	GetMethods ()
	Method get the Booked methods in a option map object.

Bool_t	HasMethod (TString methodname, TString methodtitle)
	function to check methods booked

Bool_t	IsModelPersistence ()
	Method to see if the algorithm model is saved in xml or serialized files.

Bool_t	IsSilentFile ()
	Method to see if a file is available to save results.

Bool_t	IsVerbose ()
	Method to see if the algorithm should print extra information.

void	SetDataLoader (DataLoader *dalaloader)
	Method to set the pointer to TMVA::DataLoader object.

void	SetFile (TFile *file)
	Method to set the pointer to TFile object, with a writable file.

void	SetModelPersistence (Bool_t status=kTRUE)
	Method enable model persistence, then algorithms model is saved in xml or serialized files.

void	SetVerbose (Bool_t status)
	Method enable print extra information in the algorithms.

Public Member Functions inherited from TMVA::Configurable
	Configurable (const TString &theOption="")
	constructor

virtual	~Configurable ()
	default destructor

void	AddOptionsXMLTo (void *parent) const
	write options to XML file

template<class T >
void	AddPreDefVal (const T &)

template<class T >
void	AddPreDefVal (const TString &optname, const T &)

void	CheckForUnusedOptions () const
	checks for unused options in option string

template<class T >
TMVA::OptionBase *	DeclareOptionRef (T &ref, const TString &name, const TString &desc)

template<class T >
OptionBase *	DeclareOptionRef (T &ref, const TString &name, const TString &desc="")

template<class T >
TMVA::OptionBase *	DeclareOptionRef (T *&ref, Int_t size, const TString &name, const TString &desc)

template<class T >
OptionBase *	DeclareOptionRef (T *&ref, Int_t size, const TString &name, const TString &desc="")

const char *	GetConfigDescription () const

const char *	GetConfigName () const

const TString &	GetOptions () const

MsgLogger &	Log () const

void	PrintOptions () const
	prints out the options set in the options string and the defaults

void	ReadOptionsFromStream (std::istream &istr)
	read option back from the weight file

void	ReadOptionsFromXML (void *node)

void	SetConfigDescription (const char *d)

void	SetConfigName (const char *n)

void	SetMsgType (EMsgType t)

void	SetOptions (const TString &s)

void	WriteOptionsToStream (std::ostream &o, const TString &prefix) const
	write options to output stream (e.g. in writing the MVA weight files

Public Member Functions inherited from TNamed
	TNamed ()

	TNamed (const char name, const char title)

	TNamed (const TNamed &named)
	TNamed copy ctor.

	TNamed (const TString &name, const TString &title)

virtual	~TNamed ()
	TNamed destructor.

virtual void	Clear (Option_t *option="")
	Set name and title to empty strings ("").

virtual TObject *	Clone (const char *newname="") const
	Make a clone of an object using the Streamer facility.

virtual Int_t	Compare (const TObject *obj) const
	Compare two TNamed objects.

virtual void	Copy (TObject &named) const
	Copy this to obj.

virtual void	FillBuffer (char *&buffer)
	Encode TNamed into output buffer.

virtual const char *	GetName () const
	Returns name of object.

virtual const char *	GetTitle () const
	Returns title of object.

virtual ULong_t	Hash () const
	Return hash value for this object.

virtual Bool_t	IsSortable () const

virtual void	ls (Option_t *option="") const
	List TNamed name and title.

TNamed &	operator= (const TNamed &rhs)
	TNamed assignment operator.

virtual void	Print (Option_t *option="") const
	Print TNamed name and title.

virtual void	SetName (const char *name)
	Set the name of the TNamed.

virtual void	SetNameTitle (const char name, const char title)
	Set all the TNamed parameters (name and title).

virtual void	SetTitle (const char *title="")
	Set the title of the TNamed.

virtual Int_t	Sizeof () const
	Return size of the TNamed part of the TObject.

Public Member Functions inherited from TObject
	TObject ()
	TObject constructor.

	TObject (const TObject &object)
	TObject copy ctor.

virtual	~TObject ()
	TObject destructor.

void	AbstractMethod (const char *method) const
	Use this method to implement an "abstract" method that you don't want to leave purely abstract.

virtual void	AppendPad (Option_t *option="")
	Append graphics object to current pad.

virtual void	Browse (TBrowser *b)
	Browse object. May be overridden for another default action.

ULong_t	CheckedHash ()
	Check and record whether this class has a consistent Hash/RecursiveRemove setup (*) and then return the regular Hash value for this object.

virtual const char *	ClassName () const
	Returns name of class to which the object belongs.

virtual void	Delete (Option_t *option="")
	Delete this object.

virtual Int_t	DistancetoPrimitive (Int_t px, Int_t py)
	Computes distance from point (px,py) to the object.

virtual void	Draw (Option_t *option="")
	Default Draw method for all objects.

virtual void	DrawClass () const
	Draw class inheritance tree of the class to which this object belongs.

virtual TObject *	DrawClone (Option_t *option="") const
	Draw a clone of this object in the current selected pad for instance with: `gROOT->SetSelectedPad(gPad)`.

virtual void	Dump () const
	Dump contents of object on stdout.

virtual void	Error (const char method, const char msgfmt,...) const
	Issue error message.

virtual void	Execute (const char method, const char params, Int_t *error=0)
	Execute method on this object with the given parameter string, e.g.

virtual void	Execute (TMethod method, TObjArray params, Int_t *error=0)
	Execute method on this object with parameters stored in the TObjArray.

virtual void	ExecuteEvent (Int_t event, Int_t px, Int_t py)
	Execute action corresponding to an event at (px,py).

virtual void	Fatal (const char method, const char msgfmt,...) const
	Issue fatal error message.

virtual TObject *	FindObject (const char *name) const
	Must be redefined in derived classes.

virtual TObject *	FindObject (const TObject *obj) const
	Must be redefined in derived classes.

virtual Option_t *	GetDrawOption () const
	Get option used by the graphics system to draw this object.

virtual const char *	GetIconName () const
	Returns mime type name of object.

virtual char *	GetObjectInfo (Int_t px, Int_t py) const
	Returns string containing info about the object at position (px,py).

virtual Option_t *	GetOption () const

virtual UInt_t	GetUniqueID () const
	Return the unique object id.

virtual Bool_t	HandleTimer (TTimer *timer)
	Execute action in response of a timer timing out.

Bool_t	HasInconsistentHash () const
	Return true is the type of this object is known to have an inconsistent setup for Hash and RecursiveRemove (i.e.

virtual void	Info (const char method, const char msgfmt,...) const
	Issue info message.

virtual Bool_t	InheritsFrom (const char *classname) const
	Returns kTRUE if object inherits from class "classname".

virtual Bool_t	InheritsFrom (const TClass *cl) const
	Returns kTRUE if object inherits from TClass cl.

virtual void	Inspect () const
	Dump contents of this object in a graphics canvas.

void	InvertBit (UInt_t f)

virtual Bool_t	IsEqual (const TObject *obj) const
	Default equal comparison (objects are equal if they have the same address in memory).

virtual Bool_t	IsFolder () const
	Returns kTRUE in case object contains browsable objects (like containers or lists of other objects).

R__ALWAYS_INLINE Bool_t	IsOnHeap () const

R__ALWAYS_INLINE Bool_t	IsZombie () const

void	MayNotUse (const char *method) const
	Use this method to signal that a method (defined in a base class) may not be called in a derived class (in principle against good design since a child class should not provide less functionality than its parent, however, sometimes it is necessary).

virtual Bool_t	Notify ()
	This method must be overridden to handle object notification.

void	Obsolete (const char method, const char asOfVers, const char *removedFromVers) const
	Use this method to declare a method obsolete.

void	operator delete (void *ptr)
	Operator delete.

void	operator delete[] (void *ptr)
	Operator delete [].

void *	operator new (size_t sz)

void *	operator new (size_t sz, void *vp)

void *	operator new[] (size_t sz)

void *	operator new[] (size_t sz, void *vp)

TObject &	operator= (const TObject &rhs)
	TObject assignment operator.

virtual void	Paint (Option_t *option="")
	This method must be overridden if a class wants to paint itself.

virtual void	Pop ()
	Pop on object drawn in a pad to the top of the display list.

virtual Int_t	Read (const char *name)
	Read contents of object with specified name from the current directory.

virtual void	RecursiveRemove (TObject *obj)
	Recursively remove this object from a list.

void	ResetBit (UInt_t f)

virtual void	SaveAs (const char filename="", Option_t option="") const
	Save this object in the file specified by filename.

virtual void	SavePrimitive (std::ostream &out, Option_t *option="")
	Save a primitive as a C++ statement(s) on output stream "out".

void	SetBit (UInt_t f)

void	SetBit (UInt_t f, Bool_t set)
	Set or unset the user status bits as specified in f.

virtual void	SetDrawOption (Option_t *option="")
	Set drawing option for object.

virtual void	SetUniqueID (UInt_t uid)
	Set the unique object id.

virtual void	SysError (const char method, const char msgfmt,...) const
	Issue system error message.

R__ALWAYS_INLINE Bool_t	TestBit (UInt_t f) const

Int_t	TestBits (UInt_t f) const

virtual void	UseCurrentStyle ()
	Set current style settings in this object This function is called when either TCanvas::UseCurrentStyle or TROOT::ForceStyle have been invoked.

virtual void	Warning (const char method, const char msgfmt,...) const
	Issue warning message.

virtual Int_t	Write (const char *name=0, Int_t option=0, Int_t bufsize=0)
	Write this object to the current directory.

virtual Int_t	Write (const char *name=0, Int_t option=0, Int_t bufsize=0) const
	Write this object to the current directory.

Private Member Functions
CrossValidationFoldResult	ProcessFold (UInt_t iFold, const OptionMap &methodInfo)
	Evaluates each fold in turn.

Private Attributes
Types::EAnalysisType	fAnalysisType

TString	fAnalysisTypeStr

Bool_t	fCorrelations

TString	fCvFactoryOptions

Bool_t	fDrawProgressBar

std::unique_ptr< Factory >	fFactory

std::unique_ptr< Factory >	fFoldFactory

Bool_t	fFoldFileOutput

Bool_t	fFoldStatus
	If true: generate output file for each fold.

TString	fJobName
	If true: dataset is prepared.

UInt_t	fNumFolds

UInt_t	fNumWorkerProcs
	Number of folds to prepare.

TString	fOutputEnsembling

TString	fOutputFactoryOptions
	Number of processes to use for fold evaluation.

TFile *	fOutputFile
	How to combine output of individual folds.

std::vector< CrossValidationResult >	fResults

Bool_t	fROC

Bool_t	fSilent

std::unique_ptr< CvSplitKFolds >	fSplit

TString	fSplitExprString

TString	fSplitTypeStr

TString	fTransformations

Bool_t	fVerbose

TString	fVerboseLevel

Additional Inherited Members
Public Types inherited from TObject
enum	{ kIsOnHeap = 0x01000000 , kNotDeleted = 0x02000000 , kZombie = 0x04000000 , kInconsistent = 0x08000000 , kBitMask = 0x00ffffff }

enum	{ kSingleKey = BIT(0) , kOverwrite = BIT(1) , kWriteDelete = BIT(2) }

enum	EDeprecatedStatusBits { kObjInCanvas = BIT(3) }

enum	EStatusBits { kCanDelete = BIT(0) , kMustCleanup = BIT(3) , kIsReferenced = BIT(4) , kHasUUID = BIT(5) , kCannotPick = BIT(6) , kNoContextMenu = BIT(8) , kInvalidObject = BIT(13) }

Static Public Member Functions inherited from TObject
static Long_t	GetDtorOnly ()
	Return destructor only flag.

static Bool_t	GetObjectStat ()
	Get status of object stat flag.

static void	SetDtorOnly (void *obj)
	Set destructor only flag.

static void	SetObjectStat (Bool_t stat)
	Turn on/off tracking of objects in the TObjectTable.

Protected Types inherited from TObject
enum	{ kOnlyPrepStep = BIT(3) }

Protected Member Functions inherited from TMVA::Envelope
	Envelope (const TString &name, DataLoader dataloader=nullptr, TFile file=nullptr, const TString options="")
	timer to measute the time.

DataInputHandler &	GetDataLoaderDataInput ()
	Utility method to get TMVA::DataInputHandler reference from the DataLoader.

DataSetInfo &	GetDataLoaderDataSetInfo ()
	Utility method to get TMVA::DataSetInfo reference from the DataLoader.

DataSetManager *	GetDataLoaderDataSetManager ()
	Utility method to get TMVA::DataSetManager pointer from the DataLoader.

TDirectory *	RootBaseDir ()
	Utility method to get base dir directory from current file.

void	WriteDataInformation (TMVA::DataSetInfo &fDataSetInfo, TMVA::Types::EAnalysisType fAnalysisType)
	method to save Train/Test information into the output file.

Protected Member Functions inherited from TMVA::Configurable
void	EnableLooseOptions (Bool_t b=kTRUE)

const TString &	GetReferenceFile () const

Bool_t	LooseOptionCheckingEnabled () const

void	ResetSetFlag ()
	resets the IsSet flag for all declare options to be called before options are read from stream

void	WriteOptionsReferenceToFile ()
	write complete options to output stream

Protected Member Functions inherited from TObject
virtual void	DoError (int level, const char location, const char fmt, va_list va) const
	Interface to ErrorHandler (protected).

void	MakeZombie ()

Protected Attributes inherited from TMVA::Envelope
std::shared_ptr< DataLoader >	fDataLoader
	Booked method information.

std::shared_ptr< TFile >	fFile
	data

UInt_t	fJobs
	procpool object

std::vector< OptionMap >	fMethods

Bool_t	fModelPersistence
	file to save the results

Bool_t	fSilentFile
	List of transformations to test.

TStopwatch	fTimer
	number of jobs to run some high level algorithm in parallel

TString	fTransformations
	flag for extra information

Bool_t	fVerbose
	flag to save the trained model

TProcPool	fWorkers
	if true dont produce file output

Protected Attributes inherited from TMVA::Configurable
MsgLogger *	fLogger

Protected Attributes inherited from TNamed
TString	fName

TString	fTitle

#include <TMVA/CrossValidation.h>

Inheritance diagram for TMVA::CrossValidation:

[legend]

Constructor & Destructor Documentation

◆ CrossValidation() [1/2]

TMVA::CrossValidation::CrossValidation	(	TString	jobName,
		TMVA::DataLoader *	dataloader,
		TString	options
	)

explicit

Definition at line 310 of file CrossValidation.cxx.

◆ CrossValidation() [2/2]

TMVA::CrossValidation::CrossValidation	(	TString	jobName,
		TMVA::DataLoader *	dataloader,
		TFile *	outputFile,
		TString	options
	)

explicit

Definition at line 279 of file CrossValidation.cxx.

◆ ~CrossValidation()

TMVA::CrossValidation::~CrossValidation ( )

default

Member Function Documentation

◆ Evaluate()

void TMVA::CrossValidation::Evaluate ( )

virtual

Does training, test set evaluation and performance evaluation of using cross-evalution.

Implements TMVA::Envelope.

Definition at line 588 of file CrossValidation.cxx.

◆ GetFactory()

Factory & TMVA::CrossValidation::GetFactory ( )

inline

Definition at line 140 of file CrossValidation.h.

◆ GetNumFolds()

UInt_t TMVA::CrossValidation::GetNumFolds ( )

inline

Definition at line 137 of file CrossValidation.h.

◆ GetResults()

const std::vector< TMVA::CrossValidationResult > & TMVA::CrossValidation::GetResults ( ) const

Definition at line 700 of file CrossValidation.cxx.

◆ GetSplitExpr()

TString TMVA::CrossValidation::GetSplitExpr ( )

inline

Definition at line 138 of file CrossValidation.h.

◆ InitOptions()

void TMVA::CrossValidation::InitOptions ( )

Definition at line 323 of file CrossValidation.cxx.

◆ ParseOptions()

void TMVA::CrossValidation::ParseOptions ( )

virtual

Method to parse the internal option string.

Reimplemented from TMVA::Envelope.

Definition at line 380 of file CrossValidation.cxx.

◆ ProcessFold()

TMVA::CrossValidationFoldResult TMVA::CrossValidation::ProcessFold	(	UInt_t	iFold,
		const OptionMap &	methodInfo
	)

private

Evaluates each fold in turn.

Prepares train and test data sets
Trains method
Evalutes on test set
Stores the evaluation internally

Parameters

iFold fold to evaluate

Definition at line 507 of file CrossValidation.cxx.

◆ SetNumFolds()

void TMVA::CrossValidation::SetNumFolds ( UInt_t i )

Definition at line 474 of file CrossValidation.cxx.

◆ SetSplitExpr()

void TMVA::CrossValidation::SetSplitExpr ( TString splitExpr )

Definition at line 487 of file CrossValidation.cxx.

Member Data Documentation

◆ fAnalysisType

Types::EAnalysisType TMVA::CrossValidation::fAnalysisType

private

Definition at line 149 of file CrossValidation.h.

◆ fAnalysisTypeStr

TString TMVA::CrossValidation::fAnalysisTypeStr

private

Definition at line 150 of file CrossValidation.h.

◆ fCorrelations

Bool_t TMVA::CrossValidation::fCorrelations

private

Definition at line 152 of file CrossValidation.h.

◆ fCvFactoryOptions

TString TMVA::CrossValidation::fCvFactoryOptions

private

Definition at line 153 of file CrossValidation.h.

◆ fDrawProgressBar

Bool_t TMVA::CrossValidation::fDrawProgressBar

private

Definition at line 154 of file CrossValidation.h.

◆ fFactory

std::unique_ptr<Factory> TMVA::CrossValidation::fFactory

private

Definition at line 173 of file CrossValidation.h.

◆ fFoldFactory

std::unique_ptr<Factory> TMVA::CrossValidation::fFoldFactory

private

Definition at line 172 of file CrossValidation.h.

◆ fFoldFileOutput

Bool_t TMVA::CrossValidation::fFoldFileOutput

private

Definition at line 155 of file CrossValidation.h.

◆ fFoldStatus

Bool_t TMVA::CrossValidation::fFoldStatus

private

If true: generate output file for each fold.

Definition at line 156 of file CrossValidation.h.

◆ fJobName

TString TMVA::CrossValidation::fJobName

private

If true: dataset is prepared.

Definition at line 157 of file CrossValidation.h.

◆ fNumFolds

UInt_t TMVA::CrossValidation::fNumFolds

private

Definition at line 158 of file CrossValidation.h.

◆ fNumWorkerProcs

UInt_t TMVA::CrossValidation::fNumWorkerProcs

private

Number of folds to prepare.

Definition at line 159 of file CrossValidation.h.

◆ fOutputEnsembling

TString TMVA::CrossValidation::fOutputEnsembling

private

Definition at line 162 of file CrossValidation.h.

◆ fOutputFactoryOptions

TString TMVA::CrossValidation::fOutputFactoryOptions

private

Number of processes to use for fold evaluation.

(Default, no parallel evaluation)

Definition at line 161 of file CrossValidation.h.

◆ fOutputFile

TFile* TMVA::CrossValidation::fOutputFile

private

How to combine output of individual folds.

Definition at line 163 of file CrossValidation.h.

◆ fResults

std::vector<CrossValidationResult> TMVA::CrossValidation::fResults

private

Definition at line 166 of file CrossValidation.h.

◆ fROC

Bool_t TMVA::CrossValidation::fROC

private

Definition at line 167 of file CrossValidation.h.

◆ fSilent

Bool_t TMVA::CrossValidation::fSilent

private

Definition at line 164 of file CrossValidation.h.

◆ fSplit

std::unique_ptr<CvSplitKFolds> TMVA::CrossValidation::fSplit

private

Definition at line 174 of file CrossValidation.h.

◆ fSplitExprString

TString TMVA::CrossValidation::fSplitExprString

private

Definition at line 165 of file CrossValidation.h.

◆ fSplitTypeStr

TString TMVA::CrossValidation::fSplitTypeStr

private

Definition at line 151 of file CrossValidation.h.

◆ fTransformations

TString TMVA::CrossValidation::fTransformations

private

Definition at line 168 of file CrossValidation.h.

◆ fVerbose

Bool_t TMVA::CrossValidation::fVerbose

private

Definition at line 169 of file CrossValidation.h.

◆ fVerboseLevel

TString TMVA::CrossValidation::fVerboseLevel

private

Definition at line 170 of file CrossValidation.h.

Libraries for TMVA::CrossValidation:

[legend]

The documentation for this class was generated from the following files:

tmva/tmva/inc/TMVA/CrossValidation.h
tmva/tmva/src/CrossValidation.cxx

Split Expression

Public Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Constructor & Destructor Documentation

◆ CrossValidation() [1/2]

◆ CrossValidation() [2/2]

◆ ~CrossValidation()

Member Function Documentation

◆ Evaluate()

◆ GetFactory()

◆ GetNumFolds()

◆ GetResults()

◆ GetSplitExpr()

◆ InitOptions()

◆ ParseOptions()

◆ ProcessFold()

◆ SetNumFolds()

◆ SetSplitExpr()

Member Data Documentation

◆ fAnalysisType

◆ fAnalysisTypeStr

◆ fCorrelations

◆ fCvFactoryOptions

◆ fDrawProgressBar

◆ fFactory

◆ fFoldFactory

◆ fFoldFileOutput

◆ fFoldStatus

◆ fJobName

◆ fNumFolds

◆ fNumWorkerProcs

◆ fOutputEnsembling

◆ fOutputFactoryOptions

◆ fOutputFile

◆ fResults

◆ fROC

◆ fSilent

◆ fSplit

◆ fSplitExprString

◆ fSplitTypeStr

◆ fTransformations

◆ fVerbose

◆ fVerboseLevel