template<typename AReal>
class TMVA::DNN::TReference< AReal >

The reference architecture class.

Class template that contains the reference implementation of the low-level interface for the DNN implementation. The reference implementation uses the TMatrixT class template to represent matrices.

Template Parameters

AReal The floating point type used to represent scalars.

Definition at line 37 of file Reference.h.

Public Types
using	Matrix_t = TMatrixT< AReal >

using	Scalar_t = AReal

Static Public Member Functions
Forward Propagation
Low-level functions required for the forward propagation of activations through the network.
static void	MultiplyTranspose (TMatrixT< Scalar_t > &output, const TMatrixT< Scalar_t > &input, const TMatrixT< Scalar_t > &weights)
	Matrix-multiply `input` with the transpose of and write the results into `output`. More...

static void	AddRowWise (TMatrixT< Scalar_t > &output, const TMatrixT< Scalar_t > &biases)
	Add the vectors biases row-wise to the matrix output. More...

Backward Propagation
Low-level functions required for the forward propagation of activations through the network.
static void	Backward (TMatrixT< Scalar_t > &activationGradientsBackward, TMatrixT< Scalar_t > &weightGradients, TMatrixT< Scalar_t > &biasGradients, TMatrixT< Scalar_t > &df, const TMatrixT< Scalar_t > &activationGradients, const TMatrixT< Scalar_t > &weights, const TMatrixT< Scalar_t > &activationBackward)
	Perform the complete backward propagation step. More...

static void	ScaleAdd (TMatrixT< Scalar_t > &A, const TMatrixT< Scalar_t > &B, Scalar_t beta=1.0)
	Adds a the elements in matrix B scaled by c to the elements in the matrix A. More...

static void	Copy (TMatrixT< Scalar_t > &A, const TMatrixT< Scalar_t > &B)

Activation Functions
For each activation function, the low-level interface contains two routines. One that applies the acitvation function to a matrix and one that evaluate the derivatives of the activation function at the elements of a given matrix and writes the results into the result matrix.
static void	Identity (TMatrixT< AReal > &B)

static void	IdentityDerivative (TMatrixT< AReal > &B, const TMatrixT< AReal > &A)

static void	Relu (TMatrixT< AReal > &B)

static void	ReluDerivative (TMatrixT< AReal > &B, const TMatrixT< AReal > &A)

static void	Sigmoid (TMatrixT< AReal > &B)

static void	SigmoidDerivative (TMatrixT< AReal > &B, const TMatrixT< AReal > &A)

static void	Tanh (TMatrixT< AReal > &B)

static void	TanhDerivative (TMatrixT< AReal > &B, const TMatrixT< AReal > &A)

static void	SymmetricRelu (TMatrixT< AReal > &B)

static void	SymmetricReluDerivative (TMatrixT< AReal > &B, const TMatrixT< AReal > &A)

static void	SoftSign (TMatrixT< AReal > &B)

static void	SoftSignDerivative (TMatrixT< AReal > &B, const TMatrixT< AReal > &A)

static void	Gauss (TMatrixT< AReal > &B)

static void	GaussDerivative (TMatrixT< AReal > &B, const TMatrixT< AReal > &A)

Loss Functions
Loss functions compute a scalar value given the `output` of the network for a given training input and the expected network prediction `Y` that quantifies the quality of the prediction. For each function also a routing that computes the gradients (suffixed by Gradients) must be provided for the starting of the backpropagation algorithm.
static AReal	MeanSquaredError (const TMatrixT< AReal > &Y, const TMatrixT< AReal > &output)

static void	MeanSquaredErrorGradients (TMatrixT< AReal > &dY, const TMatrixT< AReal > &Y, const TMatrixT< AReal > &output)

static AReal	CrossEntropy (const TMatrixT< AReal > &Y, const TMatrixT< AReal > &output)
	Sigmoid transformation is implicitly applied, thus `output` should hold the linear activations of the last layer in the net. More...

static void	CrossEntropyGradients (TMatrixT< AReal > &dY, const TMatrixT< AReal > &Y, const TMatrixT< AReal > &output)

static AReal	SoftmaxCrossEntropy (const TMatrixT< AReal > &Y, const TMatrixT< AReal > &output)
	Softmax transformation is implicitly applied, thus `output` should hold the linear activations of the last layer in the net. More...

static void	SoftmaxCrossEntropyGradients (TMatrixT< AReal > &dY, const TMatrixT< AReal > &Y, const TMatrixT< AReal > &output)

Output Functions
Output functions transform the activations `output` of the output layer in the network to a valid prediction `YHat` for the desired usage of the network, e.g. the identity function for regression or the sigmoid transformation for two-class classification.
static void	Sigmoid (TMatrixT< AReal > &YHat, const TMatrixT< AReal > &)

static void	Softmax (TMatrixT< AReal > &YHat, const TMatrixT< AReal > &)

Regularization
For each regularization type two functions are required, one named `<Type>Regularization` that evaluates the corresponding regularization functional for a given weight matrix and the `Add<Type>RegularizationGradients`, that adds the regularization component in the gradients to the provided matrix.
static AReal	L1Regularization (const TMatrixT< AReal > &W)

static void	AddL1RegularizationGradients (TMatrixT< AReal > &A, const TMatrixT< AReal > &W, AReal weightDecay)

static AReal	L2Regularization (const TMatrixT< AReal > &W)

static void	AddL2RegularizationGradients (TMatrixT< AReal > &A, const TMatrixT< AReal > &W, AReal weightDecay)

Initialization
For each initialization method, one function in the low-level interface is provided. The naming scheme is Initialize<Type> for a given initialization method Type.
static void	InitializeGauss (TMatrixT< AReal > &A)

static void	InitializeUniform (TMatrixT< AReal > &A)

static void	InitializeIdentity (TMatrixT< AReal > &A)

static void	InitializeZero (TMatrixT< AReal > &A)

Dropout
static void	Dropout (TMatrixT< AReal > &A, AReal dropoutProbability)
	Apply dropout with activation probability `p` to the given matrix `A` and scale the result by reciprocal of `p`. More...

#include <TMVA/DNN/Architectures/Reference.h>

Member Typedef Documentation

◆ Matrix_t

template<typename AReal >

using TMVA::DNN::TReference< AReal >::Matrix_t = TMatrixT<AReal>

Definition at line 42 of file Reference.h.

◆ Scalar_t

template<typename AReal >

using TMVA::DNN::TReference< AReal >::Scalar_t = AReal

Definition at line 41 of file Reference.h.

Member Function Documentation

◆ AddL1RegularizationGradients()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::AddL1RegularizationGradients	(	TMatrixT< AReal > &	A,
		const TMatrixT< AReal > &	W,
		AReal	weightDecay
	)

static

Definition at line 44 of file Regularization.cxx.

◆ AddL2RegularizationGradients()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::AddL2RegularizationGradients	(	TMatrixT< AReal > &	A,
		const TMatrixT< AReal > &	W,
		AReal	weightDecay
	)

static

Definition at line 82 of file Regularization.cxx.

◆ AddRowWise()

template<typename Scalar_t >

void TMVA::DNN::TReference< Scalar_t >::AddRowWise	(	TMatrixT< Scalar_t > &	output,
		const TMatrixT< Scalar_t > &	biases
	)

static

Add the vectors biases row-wise to the matrix output.

Definition at line 33 of file Propagation.cxx.

◆ Backward()

template<typename Scalar_t >

void TMVA::DNN::TReference< Scalar_t >::Backward	(	TMatrixT< Scalar_t > &	activationGradientsBackward,
		TMatrixT< Scalar_t > &	weightGradients,
		TMatrixT< Scalar_t > &	biasGradients,
		TMatrixT< Scalar_t > &	df,
		const TMatrixT< Scalar_t > &	activationGradients,
		const TMatrixT< Scalar_t > &	weights,
		const TMatrixT< Scalar_t > &	activationBackward
	)

static

Perform the complete backward propagation step.

If the provided activationGradientsBackward matrix is not empty, compute the gradients of the objective function with respect to the activations of the previous layer (backward direction). Also compute the weight and the bias gradients. Modifies the values in df and thus produces only a valid result, if it is applied the first time after the corresponding forward propagation has been per- formed.

Definition at line 44 of file Propagation.cxx.

◆ Copy()

template<typename Scalar_t >

void TMVA::DNN::TReference< Scalar_t >::Copy	(	TMatrixT< Scalar_t > &	A,
		const TMatrixT< Scalar_t > &	B
	)

static

Definition at line 95 of file Propagation.cxx.

◆ CrossEntropy()

template<typename AReal >

AReal TMVA::DNN::TReference< AReal >::CrossEntropy	(	const TMatrixT< AReal > &	Y,
		const TMatrixT< AReal > &	output
	)

static

Sigmoid transformation is implicitly applied, thus output should hold the linear activations of the last layer in the net.

Definition at line 59 of file LossFunctions.cxx.

◆ CrossEntropyGradients()

template<typename AReal >

void TMVA::DNN::TReference< AReal >::CrossEntropyGradients	(	TMatrixT< AReal > &	dY,
		const TMatrixT< AReal > &	Y,
		const TMatrixT< AReal > &	output
	)

static

Definition at line 80 of file LossFunctions.cxx.

◆ Dropout()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::Dropout	(	TMatrixT< AReal > &	A,
		AReal	dropoutProbability
	)

static

Apply dropout with activation probability p to the given matrix A and scale the result by reciprocal of p.

Definition at line 29 of file Dropout.cxx.

◆ Gauss()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::Gauss ( TMatrixT< AReal > & B )

inlinestatic

Definition at line 206 of file ActivationFunctions.cxx.

◆ GaussDerivative()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::GaussDerivative	(	TMatrixT< AReal > &	B,
		const TMatrixT< AReal > &	A
	)

inlinestatic

Definition at line 222 of file ActivationFunctions.cxx.

◆ Identity()

template<typename AReal >

static void TMVA::DNN::TReference< AReal >::Identity ( TMatrixT< AReal > & B )

static

◆ IdentityDerivative()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::IdentityDerivative	(	TMatrixT< AReal > &	B,
		const TMatrixT< AReal > &	A
	)

static

Definition at line 27 of file ActivationFunctions.cxx.

◆ InitializeGauss()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::InitializeGauss ( TMatrixT< AReal > & A )

static

Definition at line 27 of file Initialization.cxx.

◆ InitializeIdentity()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::InitializeIdentity ( TMatrixT< AReal > & A )

static

Definition at line 65 of file Initialization.cxx.

◆ InitializeUniform()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::InitializeUniform ( TMatrixT< AReal > & A )

static

Definition at line 46 of file Initialization.cxx.

◆ InitializeZero()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::InitializeZero ( TMatrixT< AReal > & A )

static

Definition at line 83 of file Initialization.cxx.

◆ L1Regularization()

template<typename AReal >

Real_t TMVA::DNN::TReference< Real_t >::L1Regularization ( const TMatrixT< AReal > & W )

static

Definition at line 26 of file Regularization.cxx.

◆ L2Regularization()

template<typename AReal >

Real_t TMVA::DNN::TReference< Real_t >::L2Regularization ( const TMatrixT< AReal > & W )

static

Definition at line 64 of file Regularization.cxx.

◆ MeanSquaredError()

template<typename AReal >

AReal TMVA::DNN::TReference< AReal >::MeanSquaredError	(	const TMatrixT< AReal > &	Y,
		const TMatrixT< AReal > &	output
	)

static

Definition at line 25 of file LossFunctions.cxx.

◆ MeanSquaredErrorGradients()

template<typename AReal >

void TMVA::DNN::TReference< AReal >::MeanSquaredErrorGradients	(	TMatrixT< AReal > &	dY,
		const TMatrixT< AReal > &	Y,
		const TMatrixT< AReal > &	output
	)

static

Definition at line 45 of file LossFunctions.cxx.

◆ MultiplyTranspose()

template<typename Scalar_t >

void TMVA::DNN::TReference< Scalar_t >::MultiplyTranspose	(	TMatrixT< Scalar_t > &	output,
		const TMatrixT< Scalar_t > &	input,
		const TMatrixT< Scalar_t > &	weights
	)

static

Matrix-multiply input with the transpose of and write the results into output.

Definition at line 25 of file Propagation.cxx.

◆ Relu()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::Relu ( TMatrixT< AReal > & B )

static

Definition at line 43 of file ActivationFunctions.cxx.

◆ ReluDerivative()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::ReluDerivative	(	TMatrixT< AReal > &	B,
		const TMatrixT< AReal > &	A
	)

inlinestatic

Definition at line 58 of file ActivationFunctions.cxx.

◆ ScaleAdd()

template<typename Scalar_t >

void TMVA::DNN::TReference< Scalar_t >::ScaleAdd	(	TMatrixT< Scalar_t > &	A,
		const TMatrixT< Scalar_t > &	B,
		Scalar_t	beta = `1.0`
	)

static

Adds a the elements in matrix B scaled by c to the elements in the matrix A.

This is required for the weight update in the gradient descent step.

Definition at line 83 of file Propagation.cxx.

◆ Sigmoid() [1/2]

template<typename AReal >

static void TMVA::DNN::TReference< AReal >::Sigmoid ( TMatrixT< AReal > & B )

static

◆ Sigmoid() [2/2]

template<typename AReal >

void TMVA::DNN::TReference< AReal >::Sigmoid	(	TMatrixT< AReal > &	YHat,
		const TMatrixT< AReal > &	A
	)

static

Definition at line 21 of file OutputFunctions.cxx.

◆ SigmoidDerivative()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::SigmoidDerivative	(	TMatrixT< AReal > &	B,
		const TMatrixT< AReal > &	A
	)

inlinestatic

Definition at line 92 of file ActivationFunctions.cxx.

◆ Softmax()

template<typename AReal >

void TMVA::DNN::TReference< AReal >::Softmax	(	TMatrixT< AReal > &	YHat,
		const TMatrixT< AReal > &	A
	)

static

Definition at line 37 of file OutputFunctions.cxx.

◆ SoftmaxCrossEntropy()

template<typename AReal >

AReal TMVA::DNN::TReference< AReal >::SoftmaxCrossEntropy	(	const TMatrixT< AReal > &	Y,
		const TMatrixT< AReal > &	output
	)

static

Softmax transformation is implicitly applied, thus output should hold the linear activations of the last layer in the net.

Definition at line 102 of file LossFunctions.cxx.

◆ SoftmaxCrossEntropyGradients()

template<typename AReal >

void TMVA::DNN::TReference< AReal >::SoftmaxCrossEntropyGradients	(	TMatrixT< AReal > &	dY,
		const TMatrixT< AReal > &	Y,
		const TMatrixT< AReal > &	output
	)

static

Definition at line 125 of file LossFunctions.cxx.

◆ SoftSign()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::SoftSign ( TMatrixT< AReal > & B )

inlinestatic

Definition at line 173 of file ActivationFunctions.cxx.

◆ SoftSignDerivative()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::SoftSignDerivative	(	TMatrixT< AReal > &	B,
		const TMatrixT< AReal > &	A
	)

inlinestatic

Definition at line 189 of file ActivationFunctions.cxx.

◆ SymmetricRelu()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::SymmetricRelu ( TMatrixT< AReal > & B )

inlinestatic

Definition at line 142 of file ActivationFunctions.cxx.

◆ SymmetricReluDerivative()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::SymmetricReluDerivative	(	TMatrixT< AReal > &	B,
		const TMatrixT< AReal > &	A
	)

inlinestatic

Definition at line 157 of file ActivationFunctions.cxx.

◆ Tanh()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::Tanh ( TMatrixT< AReal > & B )

inlinestatic

Definition at line 109 of file ActivationFunctions.cxx.

◆ TanhDerivative()

template<typename AReal >

void TMVA::DNN::TReference< Real_t >::TanhDerivative	(	TMatrixT< AReal > &	B,
		const TMatrixT< AReal > &	A
	)

inlinestatic

Definition at line 125 of file ActivationFunctions.cxx.

The documentation for this class was generated from the following files:

tmva/tmva/inc/TMVA/DNN/Architectures/Reference.h
tmva/tmva/src/DNN/Architectures/Reference/ActivationFunctions.cxx
tmva/tmva/src/DNN/Architectures/Reference/Dropout.cxx
tmva/tmva/src/DNN/Architectures/Reference/Initialization.cxx
tmva/tmva/src/DNN/Architectures/Reference/LossFunctions.cxx
tmva/tmva/src/DNN/Architectures/Reference/OutputFunctions.cxx
tmva/tmva/src/DNN/Architectures/Reference/Propagation.cxx
tmva/tmva/src/DNN/Architectures/Reference/Regularization.cxx

template<typename AReal> class TMVA::DNN::TReference< AReal >

Public Types

Static Public Member Functions

Member Typedef Documentation

◆ Matrix_t

◆ Scalar_t

Member Function Documentation

◆ AddL1RegularizationGradients()

◆ AddL2RegularizationGradients()

◆ AddRowWise()

◆ Backward()

◆ Copy()

◆ CrossEntropy()

◆ CrossEntropyGradients()

◆ Dropout()

◆ Gauss()

◆ GaussDerivative()

◆ Identity()

◆ IdentityDerivative()

◆ InitializeGauss()

◆ InitializeIdentity()

◆ InitializeUniform()

◆ InitializeZero()

◆ L1Regularization()

◆ L2Regularization()

◆ MeanSquaredError()

◆ MeanSquaredErrorGradients()

◆ MultiplyTranspose()

◆ Relu()

◆ ReluDerivative()

◆ ScaleAdd()

◆ Sigmoid() [1/2]

◆ Sigmoid() [2/2]

◆ SigmoidDerivative()

◆ Softmax()

◆ SoftmaxCrossEntropy()

◆ SoftmaxCrossEntropyGradients()

◆ SoftSign()

◆ SoftSignDerivative()

◆ SymmetricRelu()

◆ SymmetricReluDerivative()

◆ Tanh()

◆ TanhDerivative()

template<typename AReal>
class TMVA::DNN::TReference< AReal >