TMVA_CNN_Classification.C File Reference

Detailed Description

TMVA Classification Example Using a Convolutional Neural Network

This is an example of using a CNN in TMVA. We do classification using a toy image data set that is generated when running the example macro

 
Running with nthreads  = 4
DataSetInfo              : [dataset] : Added class "Signal"
                         : Add Tree sig_tree of type Signal with 1000 events
DataSetInfo              : [dataset] : Added class "Background"
                         : Add Tree bkg_tree of type Background with 1000 events
Factory                  : Booking method: ␛[1mBDT␛[0m
                         : 
                         : Rebuilding Dataset dataset
                         : Building event vectors for type 2 Signal
                         : Dataset[dataset] :  create input formulas for tree sig_tree
                         : Using variable vars[0] from array expression vars of size 256
                         : Building event vectors for type 2 Background
                         : Dataset[dataset] :  create input formulas for tree bkg_tree
                         : Using variable vars[0] from array expression vars of size 256
DataSetFactory           : [dataset] : Number of events in input trees
                         : 
                         : 
                         : Number of training and testing events
                         : ---------------------------------------------------------------------------
                         : Signal     -- training events            : 800
                         : Signal     -- testing events             : 200
                         : Signal     -- training and testing events: 1000
                         : Background -- training events            : 800
                         : Background -- testing events             : 200
                         : Background -- training and testing events: 1000
                         : 
Factory                  : Booking method: ␛[1mTMVA_DNN_CPU␛[0m
                         : 
                         : Parsing option string: 
                         : ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIER:Layout=DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,MaxEpochs=10,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.:Architecture=CPU"
                         : The following options are set:
                         : - By User:
                         :     <none>
                         : - Default:
                         :     Boost_num: "0" [Number of times the classifier will be boosted]
                         : Parsing option string: 
                         : ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIER:Layout=DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,MaxEpochs=10,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.:Architecture=CPU"
                         : The following options are set:
                         : - By User:
                         :     V: "True" [Verbose output (short form of "VerbosityLevel" below - overrides the latter one)]
                         :     VarTransform: "None" [List of variable transformations performed before training, e.g., "D_Background,P_Signal,G,N_AllClasses" for: "Decorrelation, PCA-transformation, Gaussianisation, Normalisation, each for the given class of events ('AllClasses' denotes all events of all classes, if no class indication is given, 'All' is assumed)"]
                         :     H: "False" [Print method-specific help message]
                         :     Layout: "DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,DENSE|1|LINEAR" [Layout of the network.]
                         :     ErrorStrategy: "CROSSENTROPY" [Loss function: Mean squared error (regression) or cross entropy (binary classification).]
                         :     WeightInitialization: "XAVIER" [Weight initialization strategy]
                         :     Architecture: "CPU" [Which architecture to perform the training on.]
                         :     TrainingStrategy: "LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,MaxEpochs=10,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0." [Defines the training strategies.]
                         : - Default:
                         :     VerbosityLevel: "Default" [Verbosity level]
                         :     CreateMVAPdfs: "False" [Create PDFs for classifier outputs (signal and background)]
                         :     IgnoreNegWeightsInTraining: "False" [Events with negative weights are ignored in the training (but are included for testing and performance evaluation)]
                         :     InputLayout: "0|0|0" [The Layout of the input]
                         :     BatchLayout: "0|0|0" [The Layout of the batch]
                         :     RandomSeed: "0" [Random seed used for weight initialization and batch shuffling]
                         :     ValidationSize: "20%" [Part of the training data to use for validation. Specify as 0.2 or 20% to use a fifth of the data set as validation set. Specify as 100 to use exactly 100 events. (Default: 20%)]
                         : Will now use the CPU architecture with BLAS and IMT support !
Factory                  : Booking method: ␛[1mTMVA_CNN_CPU␛[0m
                         : 
                         : Parsing option string: 
                         : ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIER:InputLayout=1|16|16:Layout=CONV|10|3|3|1|1|1|1|RELU,BNORM,CONV|10|3|3|1|1|1|1|RELU,MAXPOOL|2|2|1|1,RESHAPE|FLAT,DENSE|100|RELU,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,MaxEpochs=10,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.0:Architecture=CPU"
                         : The following options are set:
                         : - By User:
                         :     <none>
                         : - Default:
                         :     Boost_num: "0" [Number of times the classifier will be boosted]
                         : Parsing option string: 
                         : ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:WeightInitialization=XAVIER:InputLayout=1|16|16:Layout=CONV|10|3|3|1|1|1|1|RELU,BNORM,CONV|10|3|3|1|1|1|1|RELU,MAXPOOL|2|2|1|1,RESHAPE|FLAT,DENSE|100|RELU,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,MaxEpochs=10,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.0:Architecture=CPU"
                         : The following options are set:
                         : - By User:
                         :     V: "True" [Verbose output (short form of "VerbosityLevel" below - overrides the latter one)]
                         :     VarTransform: "None" [List of variable transformations performed before training, e.g., "D_Background,P_Signal,G,N_AllClasses" for: "Decorrelation, PCA-transformation, Gaussianisation, Normalisation, each for the given class of events ('AllClasses' denotes all events of all classes, if no class indication is given, 'All' is assumed)"]
                         :     H: "False" [Print method-specific help message]
                         :     InputLayout: "1|16|16" [The Layout of the input]
                         :     Layout: "CONV|10|3|3|1|1|1|1|RELU,BNORM,CONV|10|3|3|1|1|1|1|RELU,MAXPOOL|2|2|1|1,RESHAPE|FLAT,DENSE|100|RELU,DENSE|1|LINEAR" [Layout of the network.]
                         :     ErrorStrategy: "CROSSENTROPY" [Loss function: Mean squared error (regression) or cross entropy (binary classification).]
                         :     WeightInitialization: "XAVIER" [Weight initialization strategy]
                         :     Architecture: "CPU" [Which architecture to perform the training on.]
                         :     TrainingStrategy: "LearningRate=1e-3,Momentum=0.9,Repetitions=1,ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,MaxEpochs=10,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.0" [Defines the training strategies.]
                         : - Default:
                         :     VerbosityLevel: "Default" [Verbosity level]
                         :     CreateMVAPdfs: "False" [Create PDFs for classifier outputs (signal and background)]
                         :     IgnoreNegWeightsInTraining: "False" [Events with negative weights are ignored in the training (but are included for testing and performance evaluation)]
                         :     BatchLayout: "0|0|0" [The Layout of the batch]
                         :     RandomSeed: "0" [Random seed used for weight initialization and batch shuffling]
                         :     ValidationSize: "20%" [Part of the training data to use for validation. Specify as 0.2 or 20% to use a fifth of the data set as validation set. Specify as 100 to use exactly 100 events. (Default: 20%)]
                         : Will now use the CPU architecture with BLAS and IMT support !
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 reshape (Reshape)           (None, 16, 16, 1)         0         
                                                                 
 conv2d (Conv2D)             (None, 16, 16, 10)        100       
                                                                 
 batch_normalization (Batch  (None, 16, 16, 10)        40        
 Normalization)                                                  
                                                                 
 conv2d_1 (Conv2D)           (None, 16, 16, 10)        910       
                                                                 
 max_pooling2d (MaxPooling2  (None, 15, 15, 10)        0         
 D)                                                              
                                                                 
 flatten (Flatten)           (None, 2250)              0         
                                                                 
 dense (Dense)               (None, 256)               576256    
                                                                 
 dense_1 (Dense)             (None, 2)                 514       
                                                                 
=================================================================
Total params: 577820 (2.20 MB)
Trainable params: 577800 (2.20 MB)
Non-trainable params: 20 (80.00 Byte)
_________________________________________________________________
Factory                  : Booking method: ␛[1mPyKeras␛[0m
                         : 
                         : Setting up tf.keras
                         : Using TensorFlow version 2
                         : Use Keras version from TensorFlow : tf.keras
                         : Applying GPU option:  gpu_options.allow_growth=True
                         :  Loading Keras Model 
                         : Loaded model from file: model_cnn.h5
Factory                  : Booking method: ␛[1mPyTorch␛[0m
                         : 
                         : Using PyTorch - setting special configuration options 
                         : Using PyTorch version 2
                         :  Setup PyTorch Model for training
                         :  Executing user initialization code from  /github/home/ROOT-CI/build/tutorials/tmva/PyTorch_Generate_CNN_Model.py
running Torch code defining the model....
The PyTorch CNN model is created and saved as PyTorchModelCNN.pt
custom objects for loading model :  {'optimizer': <class 'torch.optim.adam.Adam'>, 'criterion': BCELoss(), 'train_func': <function fit at 0x7fe090045c10>, 'predict_func': <function predict at 0x7fe090045ca0>}
                         : Loaded pytorch train function: 
                         : Loaded pytorch optimizer: 
                         : Loaded pytorch loss function: 
                         : Loaded pytorch predict function: 
                         : Loaded model from file: PyTorchModelCNN.pt
Factory                  : ␛[1mTrain all methods␛[0m
Factory                  : Train method: BDT for Classification
                         : 
BDT                      : #events: (reweighted) sig: 800 bkg: 800
                         : #events: (unweighted) sig: 800 bkg: 800
                         : Training 200 Decision Trees ... patience please
                         : Elapsed time for training with 1600 events: 0.815 sec         
BDT                      : [dataset] : Evaluation of BDT on training sample (1600 events)
                         : Elapsed time for evaluation of 1600 events: 0.0126 sec       
                         : Creating xml weight file: ␛[0;36mdataset/weights/TMVA_CNN_Classification_BDT.weights.xml␛[0m
                         : Creating standalone class: ␛[0;36mdataset/weights/TMVA_CNN_Classification_BDT.class.C␛[0m
                         : TMVA_CNN_ClassificationOutput.root:/dataset/Method_BDT/BDT
Factory                  : Training finished
                         : 
Factory                  : Train method: TMVA_DNN_CPU for Classification
                         : 
                         : Start of deep neural network training on CPU using MT,  nthreads = 4
                         : 
                         : *****   Deep Learning Network *****
DEEP NEURAL NETWORK:   Depth = 8  Input = ( 1, 1, 256 )  Batch size = 100  Loss function = C
   Layer 0   DENSE Layer:   ( Input =   256 , Width =   100 )  Output = (  1 ,   100 ,   100 )   Activation Function = Relu
   Layer 1   BATCH NORM Layer:    Input/Output = ( 100 , 100 , 1 )    Norm dim =   100  axis = -1
 
   Layer 2   DENSE Layer:   ( Input =   100 , Width =   100 )  Output = (  1 ,   100 ,   100 )   Activation Function = Relu
   Layer 3   BATCH NORM Layer:    Input/Output = ( 100 , 100 , 1 )    Norm dim =   100  axis = -1
 
   Layer 4   DENSE Layer:   ( Input =   100 , Width =   100 )  Output = (  1 ,   100 ,   100 )   Activation Function = Relu
   Layer 5   BATCH NORM Layer:    Input/Output = ( 100 , 100 , 1 )    Norm dim =   100  axis = -1
 
   Layer 6   DENSE Layer:   ( Input =   100 , Width =   100 )  Output = (  1 ,   100 ,   100 )   Activation Function = Relu
   Layer 7   DENSE Layer:   ( Input =   100 , Width =     1 )  Output = (  1 ,   100 ,     1 )   Activation Function = Identity
                         : Using 1280 events for training and 320 for testing
                         : Compute initial loss  on the validation data 
                         : Training phase 1 of 1:  Optimizer ADAM (beta1=0.9,beta2=0.999,eps=1e-07) Learning rate = 0.001 regularization 0 minimum error = 55.9242
                         : --------------------------------------------------------------
                         :      Epoch |   Train Err.   Val. Err.  t(s)/epoch   t(s)/Loss   nEvents/s Conv. Steps
                         : --------------------------------------------------------------
                         :    Start epoch iteration ...
                         :          1 Minimum Test error found - save the configuration 
                         :          1 |     0.903846    0.878002   0.0267645  0.00254996     49556.9           0
                         :          2 Minimum Test error found - save the configuration 
                         :          2 |     0.662094    0.802488   0.0240634  0.00235519     55278.6           0
                         :          3 Minimum Test error found - save the configuration 
                         :          3 |     0.590091    0.766646   0.0240102  0.00229931     55271.8           0
                         :          4 Minimum Test error found - save the configuration 
                         :          4 |     0.517593    0.707145   0.0238882  0.00229118     55563.3           0
                         :          5 |     0.450089    0.728706    0.023327  0.00150656     54994.2           1
                         :          6 Minimum Test error found - save the configuration 
                         :          6 |     0.395614    0.705426   0.0240404  0.00232628     55263.5           0
                         :          7 Minimum Test error found - save the configuration 
                         :          7 |     0.351755    0.687489   0.0242198   0.0023066     54761.5           0
                         :          8 |     0.291487    0.738776   0.0233498  0.00146654     54836.4           1
                         :          9 |     0.265619    0.767674   0.0230724  0.00150569     55641.4           2
                         :         10 |     0.220697    0.732345   0.0239599  0.00157137     53598.9           3
                         : 
                         : Elapsed time for training with 1600 events: 0.26 sec         
                         : Evaluate deep neural network on CPU using batches with size = 100
                         : 
TMVA_DNN_CPU             : [dataset] : Evaluation of TMVA_DNN_CPU on training sample (1600 events)
                         : Elapsed time for evaluation of 1600 events: 0.0083 sec       
                         : Creating xml weight file: ␛[0;36mdataset/weights/TMVA_CNN_Classification_TMVA_DNN_CPU.weights.xml␛[0m
                         : Creating standalone class: ␛[0;36mdataset/weights/TMVA_CNN_Classification_TMVA_DNN_CPU.class.C␛[0m
Factory                  : Training finished
                         : 
Factory                  : Train method: TMVA_CNN_CPU for Classification
                         : 
                         : Start of deep neural network training on CPU using MT,  nthreads = 4
                         : 
                         : *****   Deep Learning Network *****
DEEP NEURAL NETWORK:   Depth = 7  Input = ( 1, 16, 16 )  Batch size = 100  Loss function = C
   Layer 0   CONV LAYER:   ( W = 16 ,  H = 16 ,  D = 10 )    Filter ( W = 3 ,  H = 3 )    Output = ( 100 , 10 , 10 , 256 )     Activation Function = Relu
   Layer 1   BATCH NORM Layer:    Input/Output = ( 10 , 256 , 100 )   Norm dim =    10  axis = 1
 
   Layer 2   CONV LAYER:   ( W = 16 ,  H = 16 ,  D = 10 )    Filter ( W = 3 ,  H = 3 )    Output = ( 100 , 10 , 10 , 256 )     Activation Function = Relu
   Layer 3   POOL Layer:   ( W = 15 ,  H = 15 ,  D = 10 )    Filter ( W = 2 ,  H = 2 )    Output = ( 100 , 10 , 10 , 225 ) 
   Layer 4   RESHAPE Layer     Input = ( 10 , 15 , 15 )  Output = ( 1 , 100 , 2250 ) 
   Layer 5   DENSE Layer:   ( Input =  2250 , Width =   100 )  Output = (  1 ,   100 ,   100 )   Activation Function = Relu
   Layer 6   DENSE Layer:   ( Input =   100 , Width =     1 )  Output = (  1 ,   100 ,     1 )   Activation Function = Identity
                         : Using 1280 events for training and 320 for testing
                         : Compute initial loss  on the validation data 
                         : Training phase 1 of 1:  Optimizer ADAM (beta1=0.9,beta2=0.999,eps=1e-07) Learning rate = 0.001 regularization 0 minimum error = 92.9105
                         : --------------------------------------------------------------
                         :      Epoch |   Train Err.   Val. Err.  t(s)/epoch   t(s)/Loss   nEvents/s Conv. Steps
                         : --------------------------------------------------------------
                         :    Start epoch iteration ...
                         :          1 Minimum Test error found - save the configuration 
                         :          1 |      1.76787    0.714832    0.531554   0.0403641     2443.04           0
                         :          2 |     0.781803    0.736042    0.524755   0.0401652     2476.32           1
                         :          3 Minimum Test error found - save the configuration 
                         :          3 |     0.709358    0.704204    0.476771   0.0425366     2763.49           0
                         :          4 Minimum Test error found - save the configuration 
                         :          4 |     0.695135    0.691598    0.516922   0.0430165     2532.15           0
                         :          5 Minimum Test error found - save the configuration 
                         :          5 |     0.685968     0.69011    0.552385   0.0426368      2354.1           0
                         :          6 Minimum Test error found - save the configuration 
                         :          6 |     0.680002    0.684613    0.556716   0.0430292     2336.05           0
                         :          7 Minimum Test error found - save the configuration 
                         :          7 |     0.677171    0.683112    0.512614   0.0450889     2566.71           0
                         :          8 |     0.671493    0.686612    0.484962    0.042857     2714.29           1
                         :          9 Minimum Test error found - save the configuration 
                         :          9 |     0.666061    0.682254    0.539859   0.0418702     2409.69           0
                         :         10 Minimum Test error found - save the configuration 
                         :         10 |     0.654266    0.657698     0.55653   0.0426019     2334.96           0
                         : 
                         : Elapsed time for training with 1600 events: 5.31 sec         
                         : Evaluate deep neural network on CPU using batches with size = 100
                         : 
TMVA_CNN_CPU             : [dataset] : Evaluation of TMVA_CNN_CPU on training sample (1600 events)
                         : Elapsed time for evaluation of 1600 events: 0.192 sec       
                         : Creating xml weight file: ␛[0;36mdataset/weights/TMVA_CNN_Classification_TMVA_CNN_CPU.weights.xml␛[0m
                         : Creating standalone class: ␛[0;36mdataset/weights/TMVA_CNN_Classification_TMVA_CNN_CPU.class.C␛[0m
Factory                  : Training finished
                         : 
Factory                  : Train method: PyKeras for Classification
                         : 
                         : 
                         : ␛[1m================================================================␛[0m
                         : ␛[1mH e l p   f o r   M V A   m e t h o d   [ PyKeras ] :␛[0m
                         : 
                         : Keras is a high-level API for the Theano and Tensorflow packages.
                         : This method wraps the training and predictions steps of the Keras
                         : Python package for TMVA, so that dataloading, preprocessing and
                         : evaluation can be done within the TMVA system. To use this Keras
                         : interface, you have to generate a model with Keras first. Then,
                         : this model can be loaded and trained in TMVA.
                         : 
                         : 
                         : <Suppress this message by specifying "!H" in the booking option>
                         : ␛[1m================================================================␛[0m
                         : 
                         : Split TMVA training data in 1280 training events and 320 validation events
                         : Training Model Summary
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 reshape (Reshape)           (None, 16, 16, 1)         0         
                                                                 
 conv2d (Conv2D)             (None, 16, 16, 10)        100       
                                                                 
 batch_normalization (Batch  (None, 16, 16, 10)        40        
 Normalization)                                                  
                                                                 
 conv2d_1 (Conv2D)           (None, 16, 16, 10)        910       
                                                                 
 max_pooling2d (MaxPooling2  (None, 15, 15, 10)        0         
 D)                                                              
                                                                 
 flatten (Flatten)           (None, 2250)              0         
                                                                 
 dense (Dense)               (None, 256)               576256    
                                                                 
 dense_1 (Dense)             (None, 2)                 514       
                                                                 
=================================================================
Total params: 577820 (2.20 MB)
Trainable params: 577800 (2.20 MB)
Non-trainable params: 20 (80.00 Byte)
_________________________________________________________________
                         : Option SaveBestOnly: Only model weights with smallest validation loss will be stored
Epoch 1/10
 
 1/13 [=>............................] - ETA: 8s - loss: 0.9262 - accuracy: 0.4900␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 5/13 [==========>...................] - ETA: 0s - loss: 1.7146 - accuracy: 0.4860␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 9/13 [===================>..........] - ETA: 0s - loss: 1.3857 - accuracy: 0.4978␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - ETA: 0s - loss: 1.1909 - accuracy: 0.5133
Epoch 1: val_loss improved from inf to 0.94225, saving model to trained_model_cnn.h5
␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - 1s 44ms/step - loss: 1.1909 - accuracy: 0.5133 - val_loss: 0.9422 - val_accuracy: 0.4938
Epoch 2/10
 
 1/13 [=>............................] - ETA: 0s - loss: 0.7429 - accuracy: 0.5700␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 5/13 [==========>...................] - ETA: 0s - loss: 0.7388 - accuracy: 0.5300␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 9/13 [===================>..........] - ETA: 0s - loss: 0.7186 - accuracy: 0.5378␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - ETA: 0s - loss: 0.7110 - accuracy: 0.5375
Epoch 2: val_loss improved from 0.94225 to 0.70134, saving model to trained_model_cnn.h5
␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - 0s 21ms/step - loss: 0.7110 - accuracy: 0.5375 - val_loss: 0.7013 - val_accuracy: 0.5406
Epoch 3/10
 
 1/13 [=>............................] - ETA: 0s - loss: 0.6979 - accuracy: 0.5000␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 5/13 [==========>...................] - ETA: 0s - loss: 0.6866 - accuracy: 0.5500␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 9/13 [===================>..........] - ETA: 0s - loss: 0.6827 - accuracy: 0.5533␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - ETA: 0s - loss: 0.6800 - accuracy: 0.5641
Epoch 3: val_loss did not improve from 0.70134
␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - 0s 18ms/step - loss: 0.6800 - accuracy: 0.5641 - val_loss: 0.7128 - val_accuracy: 0.5125
Epoch 4/10
 
 1/13 [=>............................] - ETA: 0s - loss: 0.6873 - accuracy: 0.5400␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 5/13 [==========>...................] - ETA: 0s - loss: 0.6629 - accuracy: 0.6200␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 9/13 [===================>..........] - ETA: 0s - loss: 0.6617 - accuracy: 0.6278␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - ETA: 0s - loss: 0.6623 - accuracy: 0.6313
Epoch 4: val_loss improved from 0.70134 to 0.67519, saving model to trained_model_cnn.h5
␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - 0s 21ms/step - loss: 0.6623 - accuracy: 0.6313 - val_loss: 0.6752 - val_accuracy: 0.6000
Epoch 5/10
 
 1/13 [=>............................] - ETA: 0s - loss: 0.6438 - accuracy: 0.7100␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 5/13 [==========>...................] - ETA: 0s - loss: 0.6511 - accuracy: 0.6640␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 9/13 [===================>..........] - ETA: 0s - loss: 0.6488 - accuracy: 0.6556␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - ETA: 0s - loss: 0.6475 - accuracy: 0.6609
Epoch 5: val_loss improved from 0.67519 to 0.66816, saving model to trained_model_cnn.h5
␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - 0s 21ms/step - loss: 0.6475 - accuracy: 0.6609 - val_loss: 0.6682 - val_accuracy: 0.6094
Epoch 6/10
 
 1/13 [=>............................] - ETA: 0s - loss: 0.6312 - accuracy: 0.6800␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 5/13 [==========>...................] - ETA: 0s - loss: 0.6457 - accuracy: 0.6580␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 9/13 [===================>..........] - ETA: 0s - loss: 0.6416 - accuracy: 0.6589␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - ETA: 0s - loss: 0.6349 - accuracy: 0.6789
Epoch 6: val_loss improved from 0.66816 to 0.66435, saving model to trained_model_cnn.h5
␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - 0s 21ms/step - loss: 0.6349 - accuracy: 0.6789 - val_loss: 0.6644 - val_accuracy: 0.6281
Epoch 7/10
 
 1/13 [=>............................] - ETA: 0s - loss: 0.6033 - accuracy: 0.7800␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 5/13 [==========>...................] - ETA: 0s - loss: 0.6196 - accuracy: 0.7200␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 9/13 [===================>..........] - ETA: 0s - loss: 0.6164 - accuracy: 0.7222␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - ETA: 0s - loss: 0.6187 - accuracy: 0.7102
Epoch 7: val_loss improved from 0.66435 to 0.65774, saving model to trained_model_cnn.h5
␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - 0s 20ms/step - loss: 0.6187 - accuracy: 0.7102 - val_loss: 0.6577 - val_accuracy: 0.6281
Epoch 8/10
 
 1/13 [=>............................] - ETA: 0s - loss: 0.5940 - accuracy: 0.6900␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 5/13 [==========>...................] - ETA: 0s - loss: 0.6023 - accuracy: 0.7240␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 9/13 [===================>..........] - ETA: 0s - loss: 0.6056 - accuracy: 0.7044␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - ETA: 0s - loss: 0.5990 - accuracy: 0.7148
Epoch 8: val_loss did not improve from 0.65774
␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - 0s 18ms/step - loss: 0.5990 - accuracy: 0.7148 - val_loss: 0.6591 - val_accuracy: 0.5938
Epoch 9/10
 
 1/13 [=>............................] - ETA: 0s - loss: 0.5997 - accuracy: 0.6600␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 5/13 [==========>...................] - ETA: 0s - loss: 0.5831 - accuracy: 0.7100␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 9/13 [===================>..........] - ETA: 0s - loss: 0.5930 - accuracy: 0.6978␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - ETA: 0s - loss: 0.5922 - accuracy: 0.6945
Epoch 9: val_loss improved from 0.65774 to 0.64599, saving model to trained_model_cnn.h5
␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - 0s 22ms/step - loss: 0.5922 - accuracy: 0.6945 - val_loss: 0.6460 - val_accuracy: 0.6562
Epoch 10/10
 
 1/13 [=>............................] - ETA: 0s - loss: 0.5617 - accuracy: 0.8100␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 5/13 [==========>...................] - ETA: 0s - loss: 0.5669 - accuracy: 0.7300␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
 9/13 [===================>..........] - ETA: 0s - loss: 0.5692 - accuracy: 0.7233␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - ETA: 0s - loss: 0.5663 - accuracy: 0.7281
Epoch 10: val_loss improved from 0.64599 to 0.63946, saving model to trained_model_cnn.h5
␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈␈
13/13 [==============================] - 0s 21ms/step - loss: 0.5663 - accuracy: 0.7281 - val_loss: 0.6395 - val_accuracy: 0.6687
                         : Getting training history for item:0 name = 'loss'
                         : Getting training history for item:1 name = 'accuracy'
                         : Getting training history for item:2 name = 'val_loss'
                         : Getting training history for item:3 name = 'val_accuracy'
                         : Elapsed time for training with 1600 events: 3.7 sec         
                         : Setting up tf.keras
                         : Using TensorFlow version 2
                         : Use Keras version from TensorFlow : tf.keras
                         : Applying GPU option:  gpu_options.allow_growth=True
                         : Disabled TF eager execution when evaluating model 
                         :  Loading Keras Model 
                         : Loaded model from file: trained_model_cnn.h5
PyKeras                  : [dataset] : Evaluation of PyKeras on training sample (1600 events)
                         : Elapsed time for evaluation of 1600 events: 0.205 sec       
                         : Creating xml weight file: ␛[0;36mdataset/weights/TMVA_CNN_Classification_PyKeras.weights.xml␛[0m
                         : Creating standalone class: ␛[0;36mdataset/weights/TMVA_CNN_Classification_PyKeras.class.C␛[0m
Factory                  : Training finished
                         : 
Factory                  : Train method: PyTorch for Classification
                         : 
                         : 
                         : ␛[1m================================================================␛[0m
                         : ␛[1mH e l p   f o r   M V A   m e t h o d   [ PyTorch ] :␛[0m
                         : 
                         : PyTorch is a scientific computing package supporting
                         : automatic differentiation. This method wraps the training
                         : and predictions steps of the PyTorch Python package for
                         : TMVA, so that dataloading, preprocessing and evaluation
                         : can be done within the TMVA system. To use this PyTorch
                         : interface, you need to generatea model with PyTorch first.
                         : Then, this model can be loaded and trained in TMVA.
                         : 
                         : 
                         : <Suppress this message by specifying "!H" in the booking option>
                         : ␛[1m================================================================␛[0m
                         : 
                         : Split TMVA training data in 1280 training events and 320 validation events
                         : Print Training Model Architecture
RecursiveScriptModule(
  original_name=Sequential
  (0): RecursiveScriptModule(original_name=Reshape)
  (1): RecursiveScriptModule(original_name=Conv2d)
  (2): RecursiveScriptModule(original_name=ReLU)
  (3): RecursiveScriptModule(original_name=BatchNorm2d)
  (4): RecursiveScriptModule(original_name=Conv2d)
  (5): RecursiveScriptModule(original_name=ReLU)
  (6): RecursiveScriptModule(original_name=MaxPool2d)
  (7): RecursiveScriptModule(original_name=Flatten)
  (8): RecursiveScriptModule(original_name=Linear)
  (9): RecursiveScriptModule(original_name=ReLU)
  (10): RecursiveScriptModule(original_name=Linear)
  (11): RecursiveScriptModule(original_name=Sigmoid)
)
                         : Option SaveBestOnly: Only model weights with smallest validation loss will be stored
[1, 4] train loss: 1.324
[1, 8] train loss: 0.730
[1, 12] train loss: 0.698
[1] val loss: 0.699
[2, 4] train loss: 0.706
[2, 8] train loss: 0.696
[2, 12] train loss: 0.691
[2] val loss: 0.683
[3, 4] train loss: 0.693
[3, 8] train loss: 0.693
[3, 12] train loss: 0.683
[3] val loss: 0.680
[4, 4] train loss: 0.683
[4, 8] train loss: 0.685
[4, 12] train loss: 0.667
[4] val loss: 0.651
[5, 4] train loss: 0.675
[5, 8] train loss: 0.671
[5, 12] train loss: 0.651
[5] val loss: 0.649
[6, 4] train loss: 0.630
[6, 8] train loss: 0.618
[6, 12] train loss: 0.587
[6] val loss: 0.774
[7, 4] train loss: 0.581
[7, 8] train loss: 0.574
[7, 12] train loss: 0.586
[7] val loss: 0.725
[8, 4] train loss: 0.564
[8, 8] train loss: 0.564
[8, 12] train loss: 0.469
[8] val loss: 0.599
[9, 4] train loss: 0.457
[9, 8] train loss: 0.499
[9, 12] train loss: 0.420
[9] val loss: 0.682
[10, 4] train loss: 0.440
[10, 8] train loss: 0.411
[10, 12] train loss: 0.358
[10] val loss: 0.642
Finished Training on 10 Epochs!
                         : Elapsed time for training with 1600 events: 2.06 sec         
PyTorch                  : [dataset] : Evaluation of PyTorch on training sample (1600 events)
                         : Elapsed time for evaluation of 1600 events: 0.0349 sec       
                         : Creating xml weight file: ␛[0;36mdataset/weights/TMVA_CNN_Classification_PyTorch.weights.xml␛[0m
                         : Creating standalone class: ␛[0;36mdataset/weights/TMVA_CNN_Classification_PyTorch.class.C␛[0m
Factory                  : Training finished
                         : 
                         : Ranking input variables (method specific)...
BDT                      : Ranking result (top variable is best ranked)
                         : --------------------------------------
                         : Rank : Variable  : Variable Importance
                         : --------------------------------------
                         :    1 : vars      : 1.240e-02
                         :    2 : vars      : 1.221e-02
                         :    3 : vars      : 1.186e-02
                         :    4 : vars      : 1.117e-02
                         :    5 : vars      : 9.855e-03
                         :    6 : vars      : 9.609e-03
                         :    7 : vars      : 9.250e-03
                         :    8 : vars      : 9.248e-03
                         :    9 : vars      : 9.177e-03
                         :   10 : vars      : 9.171e-03
                         :   11 : vars      : 8.978e-03
                         :   12 : vars      : 8.479e-03
                         :   13 : vars      : 8.338e-03
                         :   14 : vars      : 8.153e-03
                         :   15 : vars      : 8.144e-03
                         :   16 : vars      : 8.122e-03
                         :   17 : vars      : 8.058e-03
                         :   18 : vars      : 7.922e-03
                         :   19 : vars      : 7.907e-03
                         :   20 : vars      : 7.763e-03
                         :   21 : vars      : 7.599e-03
                         :   22 : vars      : 7.335e-03
                         :   23 : vars      : 7.318e-03
                         :   24 : vars      : 7.317e-03
                         :   25 : vars      : 7.268e-03
                         :   26 : vars      : 7.156e-03
                         :   27 : vars      : 7.044e-03
                         :   28 : vars      : 7.017e-03
                         :   29 : vars      : 7.017e-03
                         :   30 : vars      : 6.972e-03
                         :   31 : vars      : 6.931e-03
                         :   32 : vars      : 6.768e-03
                         :   33 : vars      : 6.756e-03
                         :   34 : vars      : 6.749e-03
                         :   35 : vars      : 6.614e-03
                         :   36 : vars      : 6.572e-03
                         :   37 : vars      : 6.546e-03
                         :   38 : vars      : 6.475e-03
                         :   39 : vars      : 6.434e-03
                         :   40 : vars      : 6.428e-03
                         :   41 : vars      : 6.379e-03
                         :   42 : vars      : 6.361e-03
                         :   43 : vars      : 6.336e-03
                         :   44 : vars      : 6.326e-03
                         :   45 : vars      : 6.315e-03
                         :   46 : vars      : 6.294e-03
                         :   47 : vars      : 6.263e-03
                         :   48 : vars      : 6.234e-03
                         :   49 : vars      : 6.228e-03
                         :   50 : vars      : 6.214e-03
                         :   51 : vars      : 6.209e-03
                         :   52 : vars      : 6.156e-03
                         :   53 : vars      : 6.120e-03
                         :   54 : vars      : 5.994e-03
                         :   55 : vars      : 5.934e-03
                         :   56 : vars      : 5.925e-03
                         :   57 : vars      : 5.853e-03
                         :   58 : vars      : 5.806e-03
                         :   59 : vars      : 5.777e-03
                         :   60 : vars      : 5.775e-03
                         :   61 : vars      : 5.685e-03
                         :   62 : vars      : 5.677e-03
                         :   63 : vars      : 5.660e-03
                         :   64 : vars      : 5.585e-03
                         :   65 : vars      : 5.584e-03
                         :   66 : vars      : 5.577e-03
                         :   67 : vars      : 5.561e-03
                         :   68 : vars      : 5.538e-03
                         :   69 : vars      : 5.494e-03
                         :   70 : vars      : 5.461e-03
                         :   71 : vars      : 5.455e-03
                         :   72 : vars      : 5.410e-03
                         :   73 : vars      : 5.405e-03
                         :   74 : vars      : 5.401e-03
                         :   75 : vars      : 5.401e-03
                         :   76 : vars      : 5.383e-03
                         :   77 : vars      : 5.368e-03
                         :   78 : vars      : 5.306e-03
                         :   79 : vars      : 5.306e-03
                         :   80 : vars      : 5.255e-03
                         :   81 : vars      : 5.173e-03
                         :   82 : vars      : 5.068e-03
                         :   83 : vars      : 5.031e-03
                         :   84 : vars      : 4.996e-03
                         :   85 : vars      : 4.974e-03
                         :   86 : vars      : 4.971e-03
                         :   87 : vars      : 4.971e-03
                         :   88 : vars      : 4.951e-03
                         :   89 : vars      : 4.928e-03
                         :   90 : vars      : 4.927e-03
                         :   91 : vars      : 4.863e-03
                         :   92 : vars      : 4.845e-03
                         :   93 : vars      : 4.799e-03
                         :   94 : vars      : 4.759e-03
                         :   95 : vars      : 4.750e-03
                         :   96 : vars      : 4.656e-03
                         :   97 : vars      : 4.638e-03
                         :   98 : vars      : 4.513e-03
                         :   99 : vars      : 4.498e-03
                         :  100 : vars      : 4.493e-03
                         :  101 : vars      : 4.478e-03
                         :  102 : vars      : 4.427e-03
                         :  103 : vars      : 4.373e-03
                         :  104 : vars      : 4.334e-03
                         :  105 : vars      : 4.332e-03
                         :  106 : vars      : 4.331e-03
                         :  107 : vars      : 4.301e-03
                         :  108 : vars      : 4.292e-03
                         :  109 : vars      : 4.279e-03
                         :  110 : vars      : 4.271e-03
                         :  111 : vars      : 4.255e-03
                         :  112 : vars      : 4.218e-03
                         :  113 : vars      : 4.210e-03
                         :  114 : vars      : 4.199e-03
                         :  115 : vars      : 4.189e-03
                         :  116 : vars      : 4.184e-03
                         :  117 : vars      : 4.122e-03
                         :  118 : vars      : 4.122e-03
                         :  119 : vars      : 4.089e-03
                         :  120 : vars      : 4.074e-03
                         :  121 : vars      : 3.933e-03
                         :  122 : vars      : 3.918e-03
                         :  123 : vars      : 3.860e-03
                         :  124 : vars      : 3.837e-03
                         :  125 : vars      : 3.828e-03
                         :  126 : vars      : 3.816e-03
                         :  127 : vars      : 3.812e-03
                         :  128 : vars      : 3.780e-03
                         :  129 : vars      : 3.775e-03
                         :  130 : vars      : 3.755e-03
                         :  131 : vars      : 3.708e-03
                         :  132 : vars      : 3.666e-03
                         :  133 : vars      : 3.655e-03
                         :  134 : vars      : 3.639e-03
                         :  135 : vars      : 3.639e-03
                         :  136 : vars      : 3.635e-03
                         :  137 : vars      : 3.615e-03
                         :  138 : vars      : 3.613e-03
                         :  139 : vars      : 3.603e-03
                         :  140 : vars      : 3.577e-03
                         :  141 : vars      : 3.532e-03
                         :  142 : vars      : 3.514e-03
                         :  143 : vars      : 3.511e-03
                         :  144 : vars      : 3.495e-03
                         :  145 : vars      : 3.492e-03
                         :  146 : vars      : 3.479e-03
                         :  147 : vars      : 3.465e-03
                         :  148 : vars      : 3.431e-03
                         :  149 : vars      : 3.401e-03
                         :  150 : vars      : 3.376e-03
                         :  151 : vars      : 3.315e-03
                         :  152 : vars      : 3.295e-03
                         :  153 : vars      : 3.255e-03
                         :  154 : vars      : 3.237e-03
                         :  155 : vars      : 3.233e-03
                         :  156 : vars      : 3.222e-03
                         :  157 : vars      : 3.200e-03
                         :  158 : vars      : 3.152e-03
                         :  159 : vars      : 3.131e-03
                         :  160 : vars      : 3.089e-03
                         :  161 : vars      : 3.049e-03
                         :  162 : vars      : 3.047e-03
                         :  163 : vars      : 3.030e-03
                         :  164 : vars      : 3.018e-03
                         :  165 : vars      : 2.989e-03
                         :  166 : vars      : 2.952e-03
                         :  167 : vars      : 2.911e-03
                         :  168 : vars      : 2.883e-03
                         :  169 : vars      : 2.875e-03
                         :  170 : vars      : 2.848e-03
                         :  171 : vars      : 2.835e-03
                         :  172 : vars      : 2.832e-03
                         :  173 : vars      : 2.825e-03
                         :  174 : vars      : 2.806e-03
                         :  175 : vars      : 2.798e-03
                         :  176 : vars      : 2.713e-03
                         :  177 : vars      : 2.690e-03
                         :  178 : vars      : 2.678e-03
                         :  179 : vars      : 2.628e-03
                         :  180 : vars      : 2.608e-03
                         :  181 : vars      : 2.573e-03
                         :  182 : vars      : 2.568e-03
                         :  183 : vars      : 2.462e-03
                         :  184 : vars      : 2.451e-03
                         :  185 : vars      : 2.442e-03
                         :  186 : vars      : 2.434e-03
                         :  187 : vars      : 2.434e-03
                         :  188 : vars      : 2.430e-03
                         :  189 : vars      : 2.428e-03
                         :  190 : vars      : 2.420e-03
                         :  191 : vars      : 2.380e-03
                         :  192 : vars      : 2.378e-03
                         :  193 : vars      : 2.371e-03
                         :  194 : vars      : 2.246e-03
                         :  195 : vars      : 2.218e-03
                         :  196 : vars      : 2.182e-03
                         :  197 : vars      : 2.127e-03
                         :  198 : vars      : 2.104e-03
                         :  199 : vars      : 2.062e-03
                         :  200 : vars      : 2.002e-03
                         :  201 : vars      : 1.859e-03
                         :  202 : vars      : 1.792e-03
                         :  203 : vars      : 1.778e-03
                         :  204 : vars      : 1.765e-03
                         :  205 : vars      : 1.702e-03
                         :  206 : vars      : 1.570e-03
                         :  207 : vars      : 1.496e-03
                         :  208 : vars      : 1.348e-03
                         :  209 : vars      : 1.311e-03
                         :  210 : vars      : 1.146e-03
                         :  211 : vars      : 9.801e-04
                         :  212 : vars      : 0.000e+00
                         :  213 : vars      : 0.000e+00
                         :  214 : vars      : 0.000e+00
                         :  215 : vars      : 0.000e+00
                         :  216 : vars      : 0.000e+00
                         :  217 : vars      : 0.000e+00
                         :  218 : vars      : 0.000e+00
                         :  219 : vars      : 0.000e+00
                         :  220 : vars      : 0.000e+00
                         :  221 : vars      : 0.000e+00
                         :  222 : vars      : 0.000e+00
                         :  223 : vars      : 0.000e+00
                         :  224 : vars      : 0.000e+00
                         :  225 : vars      : 0.000e+00
                         :  226 : vars      : 0.000e+00
                         :  227 : vars      : 0.000e+00
                         :  228 : vars      : 0.000e+00
                         :  229 : vars      : 0.000e+00
                         :  230 : vars      : 0.000e+00
                         :  231 : vars      : 0.000e+00
                         :  232 : vars      : 0.000e+00
                         :  233 : vars      : 0.000e+00
                         :  234 : vars      : 0.000e+00
                         :  235 : vars      : 0.000e+00
                         :  236 : vars      : 0.000e+00
                         :  237 : vars      : 0.000e+00
                         :  238 : vars      : 0.000e+00
                         :  239 : vars      : 0.000e+00
                         :  240 : vars      : 0.000e+00
                         :  241 : vars      : 0.000e+00
                         :  242 : vars      : 0.000e+00
                         :  243 : vars      : 0.000e+00
                         :  244 : vars      : 0.000e+00
                         :  245 : vars      : 0.000e+00
                         :  246 : vars      : 0.000e+00
                         :  247 : vars      : 0.000e+00
                         :  248 : vars      : 0.000e+00
                         :  249 : vars      : 0.000e+00
                         :  250 : vars      : 0.000e+00
                         :  251 : vars      : 0.000e+00
                         :  252 : vars      : 0.000e+00
                         :  253 : vars      : 0.000e+00
                         :  254 : vars      : 0.000e+00
                         :  255 : vars      : 0.000e+00
                         :  256 : vars      : 0.000e+00
                         : --------------------------------------
                         : No variable ranking supplied by classifier: TMVA_DNN_CPU
                         : No variable ranking supplied by classifier: TMVA_CNN_CPU
                         : No variable ranking supplied by classifier: PyKeras
                         : No variable ranking supplied by classifier: PyTorch
TH1.Print Name  = TrainingHistory_TMVA_DNN_CPU_trainingError, Entries= 0, Total sum= 4.64888
TH1.Print Name  = TrainingHistory_TMVA_DNN_CPU_valError, Entries= 0, Total sum= 7.5147
TH1.Print Name  = TrainingHistory_TMVA_CNN_CPU_trainingError, Entries= 0, Total sum= 7.98913
TH1.Print Name  = TrainingHistory_TMVA_CNN_CPU_valError, Entries= 0, Total sum= 6.93107
TH1.Print Name  = TrainingHistory_PyKeras_'accuracy', Entries= 0, Total sum= 6.43359
TH1.Print Name  = TrainingHistory_PyKeras_'loss', Entries= 0, Total sum= 6.90294
TH1.Print Name  = TrainingHistory_PyKeras_'val_accuracy', Entries= 0, Total sum= 5.93125
TH1.Print Name  = TrainingHistory_PyKeras_'val_loss', Entries= 0, Total sum= 6.96643
Factory                  : === Destroy and recreate all methods via weight files for testing ===
                         : 
                         : Reading weight file: ␛[0;36mdataset/weights/TMVA_CNN_Classification_BDT.weights.xml␛[0m
                         : Reading weight file: ␛[0;36mdataset/weights/TMVA_CNN_Classification_TMVA_DNN_CPU.weights.xml␛[0m
                         : Reading weight file: ␛[0;36mdataset/weights/TMVA_CNN_Classification_TMVA_CNN_CPU.weights.xml␛[0m
                         : Reading weight file: ␛[0;36mdataset/weights/TMVA_CNN_Classification_PyKeras.weights.xml␛[0m
                         : Reading weight file: ␛[0;36mdataset/weights/TMVA_CNN_Classification_PyTorch.weights.xml␛[0m
Factory                  : ␛[1mTest all methods␛[0m
Factory                  : Test method: BDT for Classification performance
                         : 
BDT                      : [dataset] : Evaluation of BDT on testing sample (400 events)
                         : Elapsed time for evaluation of 400 events: 0.00391 sec       
Factory                  : Test method: TMVA_DNN_CPU for Classification performance
                         : 
                         : Evaluate deep neural network on CPU using batches with size = 400
                         : 
TMVA_DNN_CPU             : [dataset] : Evaluation of TMVA_DNN_CPU on testing sample (400 events)
                         : Elapsed time for evaluation of 400 events: 0.00199 sec       
Factory                  : Test method: TMVA_CNN_CPU for Classification performance
                         : 
                         : Evaluate deep neural network on CPU using batches with size = 400
                         : 
TMVA_CNN_CPU             : [dataset] : Evaluation of TMVA_CNN_CPU on testing sample (400 events)
                         : Elapsed time for evaluation of 400 events: 0.0552 sec       
Factory                  : Test method: PyKeras for Classification performance
                         : 
                         : Setting up tf.keras
                         : Using TensorFlow version 2
                         : Use Keras version from TensorFlow : tf.keras
                         : Applying GPU option:  gpu_options.allow_growth=True
                         : Disabled TF eager execution when evaluating model 
                         :  Loading Keras Model 
                         : Loaded model from file: trained_model_cnn.h5
PyKeras                  : [dataset] : Evaluation of PyKeras on testing sample (400 events)
                         : Elapsed time for evaluation of 400 events: 0.165 sec       
Factory                  : Test method: PyTorch for Classification performance
                         : 
                         :  Setup PyTorch Model for training
                         :  Executing user initialization code from  /github/home/ROOT-CI/build/tutorials/tmva/PyTorch_Generate_CNN_Model.py
running Torch code defining the model....
The PyTorch CNN model is created and saved as PyTorchModelCNN.pt
custom objects for loading model :  {'optimizer': <class 'torch.optim.adam.Adam'>, 'criterion': BCELoss(), 'train_func': <function fit at 0x7fdf458b8dc0>, 'predict_func': <function predict at 0x7fe08470c670>}
                         : Loaded pytorch train function: 
                         : Loaded pytorch optimizer: 
                         : Loaded pytorch loss function: 
                         : Loaded pytorch predict function: 
                         : Loaded model from file: PyTorchTrainedModelCNN.pt
PyTorch                  : [dataset] : Evaluation of PyTorch on testing sample (400 events)
                         : Elapsed time for evaluation of 400 events: 0.0249 sec       
Factory                  : ␛[1mEvaluate all methods␛[0m
Factory                  : Evaluate classifier: BDT
                         : 
BDT                      : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
                         : Dataset[dataset] :  variable plots are not produces ! The number of variables is 256 , it is larger than 200
Factory                  : Evaluate classifier: TMVA_DNN_CPU
                         : 
TMVA_DNN_CPU             : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
                         : Evaluate deep neural network on CPU using batches with size = 1000
                         : 
                         : Dataset[dataset] :  variable plots are not produces ! The number of variables is 256 , it is larger than 200
Factory                  : Evaluate classifier: TMVA_CNN_CPU
                         : 
TMVA_CNN_CPU             : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
                         : Evaluate deep neural network on CPU using batches with size = 1000
                         : 
                         : Dataset[dataset] :  variable plots are not produces ! The number of variables is 256 , it is larger than 200
Factory                  : Evaluate classifier: PyKeras
                         : 
PyKeras                  : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
                         : Dataset[dataset] :  variable plots are not produces ! The number of variables is 256 , it is larger than 200
Factory                  : Evaluate classifier: PyTorch
                         : 
PyTorch                  : [dataset] : Loop over test events and fill histograms with classifier response...
                         : 
                         : Dataset[dataset] :  variable plots are not produces ! The number of variables is 256 , it is larger than 200
                         : 
                         : Evaluation results ranked by best signal efficiency and purity (area)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet       MVA                       
                         : Name:         Method:          ROC-integ
                         : dataset       PyTorch        : 0.849
                         : dataset       BDT            : 0.730
                         : dataset       TMVA_DNN_CPU   : 0.704
                         : dataset       TMVA_CNN_CPU   : 0.699
                         : dataset       PyKeras        : 0.688
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
                         : Testing efficiency compared to training efficiency (overtraining check)
                         : -------------------------------------------------------------------------------------------------------------------
                         : DataSet              MVA              Signal efficiency: from test sample (from training sample) 
                         : Name:                Method:          @B=0.01             @B=0.10            @B=0.30   
                         : -------------------------------------------------------------------------------------------------------------------
                         : dataset              PyTorch        : 0.175 (0.205)       0.565 (0.615)      0.846 (0.849)
                         : dataset              BDT            : 0.025 (0.215)       0.335 (0.590)      0.638 (0.783)
                         : dataset              TMVA_DNN_CPU   : 0.040 (0.100)       0.310 (0.523)      0.585 (0.777)
                         : dataset              TMVA_CNN_CPU   : 0.065 (0.050)       0.285 (0.313)      0.565 (0.620)
                         : dataset              PyKeras        : 0.025 (0.055)       0.275 (0.353)      0.585 (0.652)
                         : -------------------------------------------------------------------------------------------------------------------
                         : 
Dataset:dataset          : Created tree 'TestTree' with 400 events
                         : 
Dataset:dataset          : Created tree 'TrainTree' with 1600 events
                         : 
Factory                  : ␛[1mThank you for using TMVA!␛[0m
                         : ␛[1mFor citation information, please visit: http://tmva.sf.net/citeTMVA.html␛[0m

 
/***
 
    # TMVA Classification Example Using a Convolutional Neural Network
 
 
**/
 
/// Helper function to create input images data
/// we create a signal and background 2D histograms from 2d gaussians
/// with a location (means in X and Y)  different for each event
/// The difference between signal and background is in the gaussian width.
/// The width for the background gaussian is slightly larger than the signal width by few % values
///
///
void MakeImagesTree(int n, int nh, int nw)
{
 
   // image size (nh x nw)
   const int ntot = nh * nw;
   const TString fileOutName = TString::Format("images_data_%dx%d.root", nh, nw);
   TFile f(fileOutName, "RECREATE");
 
   const int nRndmEvts = 10000; // number of events we use to fill each image
   double delta_sigma = 0.1;    // 5% difference in the sigma
   double pixelNoise = 5;
 
   double sX1 = 3;
   double sY1 = 3;
   double sX2 = sX1 + delta_sigma;
   double sY2 = sY1 - delta_sigma;
 
   TH2D h1("h1", "h1", nh, 0, 10, nw, 0, 10);
   TH2D h2("h2", "h2", nh, 0, 10, nw, 0, 10);
 
   TF2 f1("f1", "xygaus");
   TF2 f2("f2", "xygaus");
 
   TTree sgn("sig_tree", "signal_tree");
   TTree bkg("bkg_tree", "background_tree");
 
 
   std::vector<float> x1(ntot);
   std::vector<float> x2(ntot);
 
   // create signal and background trees with a single branch
   // an std::vector<float> of size nh x nw containing the image data
 
   std::vector<float> *px1 = &x1;
   std::vector<float> *px2 = &x2;
 
   bkg.Branch("vars", "std::vector<float>", &px1);
   sgn.Branch("vars", "std::vector<float>", &px2);
 
   // std::cout << "create tree " << std::endl;
 
   sgn.SetDirectory(&f);
   bkg.SetDirectory(&f);
 
   f1.SetParameters(1, 5, sX1, 5, sY1);
   f2.SetParameters(1, 5, sX2, 5, sY2);
   gRandom->SetSeed(0);
   std::cout << "Filling ROOT tree " << std::endl;
   for (int i = 0; i < n; ++i) {
      if (i % 1000 == 0)
         std::cout << "Generating image event ... " << i << std::endl;
      h1.Reset();
      h2.Reset();
      // generate random means in range [3,7] to be not too much on the border
      f1.SetParameter(1, gRandom->Uniform(3, 7));
      f1.SetParameter(3, gRandom->Uniform(3, 7));
      f2.SetParameter(1, gRandom->Uniform(3, 7));
      f2.SetParameter(3, gRandom->Uniform(3, 7));
 
      h1.FillRandom("f1", nRndmEvts);
      h2.FillRandom("f2", nRndmEvts);
 
      for (int k = 0; k < nh; ++k) {
         for (int l = 0; l < nw; ++l) {
            int m = k * nw + l;
            // add some noise in each bin
            x1[m] = h1.GetBinContent(k + 1, l + 1) + gRandom->Gaus(0, pixelNoise);
            x2[m] = h2.GetBinContent(k + 1, l + 1) + gRandom->Gaus(0, pixelNoise);
         }
      }
      sgn.Fill();
      bkg.Fill();
   }
   sgn.Write();
   bkg.Write();
 
   Info("MakeImagesTree", "Signal and background tree with images data written to the file %s", f.GetName());
   sgn.Print();
   bkg.Print();
   f.Close();
}
 
/// @brief Run the TMVA CNN Classification example
/// @param nevts : number of signal/background events. Use by default a low value (1000)
///                but increase to at least 5000 to get a good result
/// @param opt :   vector of bool with method used (default all on if available). The order is:
///                   - TMVA CNN
///                   - Keras CNN
///                   - TMVA DNN
///                   - TMVA BDT
///                   - PyTorch CNN
void TMVA_CNN_Classification(int nevts = 1000, std::vector<bool> opt = {1, 1, 1, 1, 1})
{
 
   int imgSize = 16 * 16;
   TString inputFileName = "images_data_16x16.root";
 
   bool fileExist = !gSystem->AccessPathName(inputFileName);
 
   // if file does not exists create it
   if (!fileExist) {
      MakeImagesTree(nevts, 16, 16);
   }
 
   bool useTMVACNN = (opt.size() > 0) ? opt[0] : false;
   bool useKerasCNN = (opt.size() > 1) ? opt[1] : false;
   bool useTMVADNN = (opt.size() > 2) ? opt[2] : false;
   bool useTMVABDT = (opt.size() > 3) ? opt[3] : false;
   bool usePyTorchCNN = (opt.size() > 4) ? opt[4] : false;
#ifndef R__HAS_TMVACPU
#ifndef R__HAS_TMVAGPU
   Warning("TMVA_CNN_Classification",
           "TMVA is not build with GPU or CPU multi-thread support. Cannot use TMVA Deep Learning for CNN");
   useTMVACNN = false;
#endif
#endif
 
   bool writeOutputFile = true;
 
#ifdef R__USE_IMT
   int num_threads = 4;  // use by default 4 threads if value is not set before
   // switch off MT in OpenBLAS to avoid conflict with tbb
   gSystem->Setenv("OMP_NUM_THREADS", "1");
 
   // do enable MT running
   if (num_threads >= 0) {
      ROOT::EnableImplicitMT(num_threads);
   }
#endif
 
   TMVA::Tools::Instance();
 
 
   std::cout << "Running with nthreads  = " << ROOT::GetThreadPoolSize() << std::endl;
 
#ifdef R__HAS_PYMVA
   gSystem->Setenv("KERAS_BACKEND", "tensorflow");
   // for using Keras
   TMVA::PyMethodBase::PyInitialize();
#else
   useKerasCNN = false;
   usePyTorchCNN = false;
#endif
 
   TFile *outputFile = nullptr;
   if (writeOutputFile)
      outputFile = TFile::Open("TMVA_CNN_ClassificationOutput.root", "RECREATE");
 
   /***
       ## Create TMVA Factory
 
    Create the Factory class. Later you can choose the methods
    whose performance you'd like to investigate.
 
    The factory is the major TMVA object you have to interact with. Here is the list of parameters you need to pass
 
    - The first argument is the base of the name of all the output
    weight files in the directory weight/ that will be created with the
    method parameters
 
    - The second argument is the output file for the training results
 
    - The third argument is a string option defining some general configuration for the TMVA session.
      For example all TMVA output can be suppressed by removing the "!" (not) in front of the "Silent" argument in the
   option string
 
    - note that we disable any pre-transformation of the input variables and we avoid computing correlations between
   input variables
   ***/
 
   TMVA::Factory factory(
      "TMVA_CNN_Classification", outputFile,
      "!V:ROC:!Silent:Color:AnalysisType=Classification:Transformations=None:!Correlations");
 
   /***
 
       ## Declare DataLoader(s)
 
       The next step is to declare the DataLoader class that deals with input variables
 
       Define the input variables that shall be used for the MVA training
       note that you may also use variable expressions, which can be parsed by TTree::Draw( "expression" )]
 
       In this case the input data consists of an image of 16x16 pixels. Each single pixel is a branch in a ROOT TTree
 
   **/
 
   TMVA::DataLoader loader("dataset");
 
   /***
 
       ## Setup Dataset(s)
 
       Define input data file and signal and background trees
 
   **/
 
   std::unique_ptr<TFile> inputFile{TFile::Open(inputFileName)};
   if (!inputFile) {
      Error("TMVA_CNN_Classification", "Error opening input file %s - exit", inputFileName.Data());
      return;
   }
 
   // --- Register the training and test trees
 
   auto signalTree = inputFile->Get<TTree>("sig_tree");
   auto backgroundTree = inputFile->Get<TTree>("bkg_tree");
 
   if (!signalTree) {
      Error("TMVA_CNN_Classification", "Could not find signal tree in file '%s'", inputFileName.Data());
      return;
   }
   if (!backgroundTree) {
      Error("TMVA_CNN_Classification", "Could not find background tree in file '%s'", inputFileName.Data());
      return;
   }
 
   int nEventsSig = signalTree->GetEntries();
   int nEventsBkg = backgroundTree->GetEntries();
 
   // global event weights per tree (see below for setting event-wise weights)
   Double_t signalWeight = 1.0;
   Double_t backgroundWeight = 1.0;
 
   // You can add an arbitrary number of signal or background trees
   loader.AddSignalTree(signalTree, signalWeight);
   loader.AddBackgroundTree(backgroundTree, backgroundWeight);
 
   /// add event variables (image)
   /// use new method (from ROOT 6.20 to add a variable array for all image data)
   loader.AddVariablesArray("vars", imgSize);
 
   // Set individual event weights (the variables must exist in the original TTree)
   //    for signal    : factory->SetSignalWeightExpression    ("weight1*weight2");
   //    for background: factory->SetBackgroundWeightExpression("weight1*weight2");
   // loader.SetBackgroundWeightExpression( "weight" );
 
   // Apply additional cuts on the signal and background samples (can be different)
   TCut mycuts = ""; // for example: TCut mycuts = "abs(var1)<0.5 && abs(var2-0.5)<1";
   TCut mycutb = ""; // for example: TCut mycutb = "abs(var1)<0.5";
 
   // Tell the factory how to use the training and testing events
   //
   // If no numbers of events are given, half of the events in the tree are used
   // for training, and the other half for testing:
   //    loader.PrepareTrainingAndTestTree( mycut, "SplitMode=random:!V" );
   // It is possible also to specify the number of training and testing events,
   // note we disable the computation of the correlation matrix of the input variables
 
   int nTrainSig = 0.8 * nEventsSig;
   int nTrainBkg = 0.8 * nEventsBkg;
 
   // build the string options for DataLoader::PrepareTrainingAndTestTree
   TString prepareOptions = TString::Format(
      "nTrain_Signal=%d:nTrain_Background=%d:SplitMode=Random:SplitSeed=100:NormMode=NumEvents:!V:!CalcCorrelations",
      nTrainSig, nTrainBkg);
 
   loader.PrepareTrainingAndTestTree(mycuts, mycutb, prepareOptions);
 
   /***
 
       DataSetInfo              : [dataset] : Added class "Signal"
       : Add Tree sig_tree of type Signal with 10000 events
       DataSetInfo              : [dataset] : Added class "Background"
       : Add Tree bkg_tree of type Background with 10000 events
 
 
 
   **/
 
   /****
        # Booking Methods
 
        Here we book the TMVA methods. We book a Boosted Decision Tree method (BDT)
 
   **/
 
   // Boosted Decision Trees
   if (useTMVABDT) {
      factory.BookMethod(&loader, TMVA::Types::kBDT, "BDT",
                         "!V:NTrees=200:MinNodeSize=2.5%:MaxDepth=2:BoostType=AdaBoost:AdaBoostBeta=0.5:"
                         "UseBaggedBoost:BaggedSampleFraction=0.5:SeparationType=GiniIndex:nCuts=20");
   }
   /**
 
      #### Booking Deep Neural Network
 
      Here we book the DNN of TMVA. See the example TMVA_Higgs_Classification.C for a detailed description of the
   options
 
   **/
 
   if (useTMVADNN) {
 
      TString layoutString(
         "Layout=DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,BNORM,DENSE|100|RELU,DENSE|1|LINEAR");
 
      // Training strategies
      // one can catenate several training strings with different parameters (e.g. learning rates or regularizations
      // parameters) The training string must be concatenates with the `|` delimiter
      TString trainingString1("LearningRate=1e-3,Momentum=0.9,Repetitions=1,"
                              "ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,"
                              "MaxEpochs=10,WeightDecay=1e-4,Regularization=None,"
                              "Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.");
 
      TString trainingStrategyString("TrainingStrategy=");
      trainingStrategyString += trainingString1; // + "|" + trainingString2 + ....
 
      // Build now the full DNN Option string
 
      TString dnnOptions("!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:"
                         "WeightInitialization=XAVIER");
      dnnOptions.Append(":");
      dnnOptions.Append(layoutString);
      dnnOptions.Append(":");
      dnnOptions.Append(trainingStrategyString);
 
      TString dnnMethodName = "TMVA_DNN_CPU";
// use GPU if available
#ifdef R__HAS_TMVAGPU
      dnnOptions += ":Architecture=GPU";
      dnnMethodName = "TMVA_DNN_GPU";
#elif defined(R__HAS_TMVACPU)
      dnnOptions += ":Architecture=CPU";
#endif
 
      factory.BookMethod(&loader, TMVA::Types::kDL, dnnMethodName, dnnOptions);
   }
 
   /***
    ### Book Convolutional Neural Network in TMVA
 
    For building a CNN one needs to define
 
    -  Input Layout :  number of channels (in this case = 1)  | image height | image width
    -  Batch Layout :  batch size | number of channels | image size = (height*width)
 
    Then one add Convolutional layers and MaxPool layers.
 
    -  For Convolutional layer the option string has to be:
       - CONV | number of units | filter height | filter width | stride height | stride width | padding height | paddig
   width | activation function
 
       - note in this case we are using a filer 3x3 and padding=1 and stride=1 so we get the output dimension of the
   conv layer equal to the input
 
      - note we use after the first convolutional layer a batch normalization layer. This seems to help significantly the
   convergence
 
     - For the MaxPool layer:
        - MAXPOOL  | pool height | pool width | stride height | stride width
 
    The RESHAPE layer is needed to flatten the output before the Dense layer
 
 
    Note that to run the CNN is required to have CPU  or GPU support
 
   ***/
 
   if (useTMVACNN) {
 
      TString inputLayoutString("InputLayout=1|16|16");
 
      // Batch Layout
      TString layoutString("Layout=CONV|10|3|3|1|1|1|1|RELU,BNORM,CONV|10|3|3|1|1|1|1|RELU,MAXPOOL|2|2|1|1,"
                           "RESHAPE|FLAT,DENSE|100|RELU,DENSE|1|LINEAR");
 
      // Training strategies.
      TString trainingString1("LearningRate=1e-3,Momentum=0.9,Repetitions=1,"
                              "ConvergenceSteps=5,BatchSize=100,TestRepetitions=1,"
                              "MaxEpochs=10,WeightDecay=1e-4,Regularization=None,"
                              "Optimizer=ADAM,DropConfig=0.0+0.0+0.0+0.0");
 
      TString trainingStrategyString("TrainingStrategy=");
      trainingStrategyString +=
         trainingString1; // + "|" + trainingString2 + "|" + trainingString3; for concatenating more training strings
 
      // Build full CNN Options.
      TString cnnOptions("!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:"
                         "WeightInitialization=XAVIER");
 
      cnnOptions.Append(":");
      cnnOptions.Append(inputLayoutString);
      cnnOptions.Append(":");
      cnnOptions.Append(layoutString);
      cnnOptions.Append(":");
      cnnOptions.Append(trainingStrategyString);
 
      //// New DL (CNN)
      TString cnnMethodName = "TMVA_CNN_CPU";
// use GPU if available
#ifdef R__HAS_TMVAGPU
      cnnOptions += ":Architecture=GPU";
      cnnMethodName = "TMVA_CNN_GPU";
#else
      cnnOptions += ":Architecture=CPU";
      cnnMethodName = "TMVA_CNN_CPU";
#endif
 
      factory.BookMethod(&loader, TMVA::Types::kDL, cnnMethodName, cnnOptions);
   }
 
   /**
      ### Book Convolutional Neural Network in Keras using a generated model
 
   **/
 
#ifdef R__HAS_PYMVA
   // The next section uses Python packages, execute it only if PyMVA is available
   TString tmva_python_exe{TMVA::Python_Executable()};
   TString python_exe = tmva_python_exe.IsNull() ? "python" : tmva_python_exe;
 
   if (useKerasCNN) {
 
      Info("TMVA_CNN_Classification", "Building convolutional keras model");
      // create python script which can be executed
      // create 2 conv2d layer + maxpool + dense
      TMacro m;
      m.AddLine("import tensorflow");
      m.AddLine("from tensorflow.keras.models import Sequential");
      m.AddLine("from tensorflow.keras.optimizers import Adam");
      m.AddLine(
         "from tensorflow.keras.layers import Input, Dense, Dropout, Flatten, Conv2D, MaxPooling2D, Reshape, BatchNormalization");
      m.AddLine("");
      m.AddLine("model = Sequential() ");
      m.AddLine("model.add(Reshape((16, 16, 1), input_shape = (256, )))");
      m.AddLine("model.add(Conv2D(10, kernel_size = (3, 3), kernel_initializer = 'glorot_normal',activation = "
                "'relu', padding = 'same'))");
      m.AddLine("model.add(BatchNormalization())");
      m.AddLine("model.add(Conv2D(10, kernel_size = (3, 3), kernel_initializer = 'glorot_normal',activation = "
                "'relu', padding = 'same'))");
      // m.AddLine("model.add(BatchNormalization())");
      m.AddLine("model.add(MaxPooling2D(pool_size = (2, 2), strides = (1,1))) ");
      m.AddLine("model.add(Flatten())");
      m.AddLine("model.add(Dense(256, activation = 'relu')) ");
      m.AddLine("model.add(Dense(2, activation = 'sigmoid')) ");
      m.AddLine("model.compile(loss = 'binary_crossentropy', optimizer = Adam(learning_rate = 0.001), weighted_metrics = ['accuracy'])");
      m.AddLine("model.save('model_cnn.h5')");
      m.AddLine("model.summary()");
 
      m.SaveSource("make_cnn_model.py");
      // execute
      gSystem->Exec(python_exe + " make_cnn_model.py");
 
      if (gSystem->AccessPathName("model_cnn.h5")) {
         Warning("TMVA_CNN_Classification", "Error creating Keras model file - skip using Keras");
      } else {
         // book PyKeras method only if Keras model could be created
         Info("TMVA_CNN_Classification", "Booking tf.Keras CNN model");
         factory.BookMethod(
            &loader, TMVA::Types::kPyKeras, "PyKeras",
            "H:!V:VarTransform=None:FilenameModel=model_cnn.h5:tf.keras:"
            "FilenameTrainedModel=trained_model_cnn.h5:NumEpochs=10:BatchSize=100:"
            "GpuOptions=allow_growth=True"); // needed for RTX NVidia card and to avoid TF allocates all GPU memory
      }
   }
 
   if (usePyTorchCNN) {
 
      Info("TMVA_CNN_Classification", "Using Convolutional PyTorch Model");
      TString pyTorchFileName = gROOT->GetTutorialDir() + TString("/tmva/PyTorch_Generate_CNN_Model.py");
      // check that pytorch can be imported and file defining the model and used later when booking the method is
      // existing
      if (gSystem->Exec(python_exe + " -c 'import torch'") || gSystem->AccessPathName(pyTorchFileName)) {
         Warning("TMVA_CNN_Classification", "PyTorch is not installed or model building file is not existing - skip using PyTorch");
      } else {
         // book PyTorch method only if PyTorch model could be created
         Info("TMVA_CNN_Classification", "Booking PyTorch CNN model");
         TString methodOpt = "H:!V:VarTransform=None:FilenameModel=PyTorchModelCNN.pt:"
                             "FilenameTrainedModel=PyTorchTrainedModelCNN.pt:NumEpochs=10:BatchSize=100";
         methodOpt += TString(":UserCode=") + pyTorchFileName;
         factory.BookMethod(&loader, TMVA::Types::kPyTorch, "PyTorch", methodOpt);
      }
   }
#endif
 
   ////  ## Train Methods
 
   factory.TrainAllMethods();
 
   /// ## Test and Evaluate Methods
 
   factory.TestAllMethods();
 
   factory.EvaluateAllMethods();
 
   /// ## Plot ROC Curve
 
   auto c1 = factory.GetROCCurve(&loader);
   c1->Draw();
 
   // close outputfile to save output file
   outputFile->Close();
}

Author

Lorenzo Moneta

Definition in file TMVA_CNN_Classification.C.