Execution Time13.96s

Test: tutorial-tmva-TMVAMulticlass (Passed)
Build: master-x86_64-mac1013-clang100 (macphsft16.dyndns.cern.ch) on 2019-11-14 00:49:58

Test Timing: Passed
Processors1

Show Command Line
Display graphs:

Test output
Processing /Users/sftnight/build/jenkins/night/LABEL/mac1013/SPEC/cxx14/V/master/root/tutorials/tmva/TMVAMulticlass.C...

==> Start TMVAMulticlass
create data set info dataset
--- TMVAMulticlass   : Accessing ./tmva_example_multiple_background.root
DataSetInfo              : [dataset] : Added class "Signal"
                         : Add Tree TreeS of type Signal with 200 events
DataSetInfo              : [dataset] : Added class "bg0"
                         : Add Tree TreeB0 of type bg0 with 200 events
DataSetInfo              : [dataset] : Added class "bg1"
                         : Add Tree TreeB1 of type bg1 with 200 events
DataSetInfo              : [dataset] : Added class "bg2"
                         : Add Tree TreeB2 of type bg2 with 200 events
                         : Dataset[dataset] : Class index : 0  name : Signal
                         : Dataset[dataset] : Class index : 1  name : bg0
                         : Dataset[dataset] : Class index : 2  name : bg1
                         : Dataset[dataset] : Class index : 3  name : bg2
Factory                  : Booking method: [NON-XML-CHAR-0x1B][1mBDTG[NON-XML-CHAR-0x1B][0m
                         : 
                         : the option NegWeightTreatment=InverseBoostNegWeights does not exist for BoostType=Grad
                         : --> change to new default NegWeightTreatment=Pray
                         : Building event vectors for type 2 Signal
                         : Dataset[dataset] :  create input formulas for tree TreeS
                         : Building event vectors for type 2 bg0
                         : Dataset[dataset] :  create input formulas for tree TreeB0
                         : Building event vectors for type 2 bg1
                         : Dataset[dataset] :  create input formulas for tree TreeB1
                         : Building event vectors for type 2 bg2
                         : Dataset[dataset] :  create input formulas for tree TreeB2
DataSetFactory           : [dataset] : Number of events in input trees
                         : 
                         : 
                         : 
                         : 
                         : Number of training and testing events
                         : ---------------------------------------------------------------------------
                         : Signal -- training events            : 100
                         : Signal -- testing events             : 100
                         : Signal -- training and testing events: 200
                         : bg0    -- training events            : 100
                         : bg0    -- testing events             : 100
                         : bg0    -- training and testing events: 200
                         : bg1    -- training events            : 100
                         : bg1    -- testing events             : 100
                         : bg1    -- training and testing events: 200
                         : bg2    -- training events            : 100
                         : bg2    -- testing events             : 100
                         : bg2    -- training and testing events: 200
                         : 
DataSetInfo              : Correlation matrix (Signal):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  +0.303  +0.518  +0.811
                         :    var2:  +0.303  +1.000  +0.747  +0.724
                         :    var3:  +0.518  +0.747  +1.000  +0.825
                         :    var4:  +0.811  +0.724  +0.825  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (bg0):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  +0.274  +0.597  +0.826
                         :    var2:  +0.274  +1.000  +0.682  +0.644
                         :    var3:  +0.597  +0.682  +1.000  +0.847
                         :    var4:  +0.826  +0.644  +0.847  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (bg1):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  +0.442  +0.565  +0.823
                         :    var2:  +0.442  +1.000  +0.791  +0.780
                         :    var3:  +0.565  +0.791  +1.000  +0.849
                         :    var4:  +0.823  +0.780  +0.849  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (bg2):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  -0.657  +0.094  +0.153
                         :    var2:  -0.657  +1.000  -0.029  -0.159
                         :    var3:  +0.094  -0.029  +1.000  +0.122
                         :    var4:  +0.153  -0.159  +0.122  +1.000
                         : ----------------------------------------
DataSetFactory           : [dataset] :  
                         : 
Factory                  : Booking method: [NON-XML-CHAR-0x1B][1mMLP[NON-XML-CHAR-0x1B][0m
                         : 
MLP                      : Building Network. 
                         : Initializing weights
Factory                  : [NON-XML-CHAR-0x1B][1mTrain all methods[NON-XML-CHAR-0x1B][0m
Factory                  : [dataset] : Create Transformation "I" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
Factory                  : [dataset] : Create Transformation "D" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
Factory                  : [dataset] : Create Transformation "P" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
Factory                  : [dataset] : Create Transformation "G" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
Factory                  : [dataset] : Create Transformation "D" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:    0.12296    0.93324   [    -3.1150     2.1743 ]
                         :     var2:    0.33916    0.97369   [    -3.0952     3.1113 ]
                         :     var3:    0.24960     1.0884   [    -2.3587     3.9796 ]
                         :     var4:   0.081689     1.1382   [    -3.7913     3.5074 ]
                         : -----------------------------------------------------------
                         : Preparing the Decorrelation transformation...
TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:    0.13058     1.0000   [    -3.0340     2.6107 ]
                         :     var2:    0.34245     1.0000   [    -2.6833     2.9150 ]
                         :     var3:    0.17597     1.0000   [    -2.5318     3.1735 ]
                         :     var4:  -0.090847     1.0000   [    -2.6761     2.4283 ]
                         : -----------------------------------------------------------
                         : Preparing the Principle Component (PCA) transformation...
TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:-2.4447e-10     1.6344   [    -6.2668     5.1503 ]
                         :     var2: 4.8894e-10    0.89261   [    -2.5767     2.5517 ]
                         :     var3:-9.4617e-10    0.71800   [    -2.3717     1.7098 ]
                         :     var4: 8.5100e-10    0.56167   [    -1.6125     1.9674 ]
                         : -----------------------------------------------------------
                         : Preparing the Gaussian transformation...
                         : Preparing the Decorrelation transformation...
TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:   0.093209     1.0000   [    -1.7806     5.9786 ]
                         :     var2:   0.072716     1.0000   [    -1.8056     5.9648 ]
                         :     var3:   0.090011     1.0000   [    -2.2288     5.7510 ]
                         :     var4:   0.049916     1.0000   [    -2.5521     5.7800 ]
                         : -----------------------------------------------------------
                         : Ranking input variables (method unspecific)...
Factory                  : Train method: BDTG for Multiclass classification
                         : 
                         : Training 1000 Decision Trees ... patience please
0%, time left: unknown
6%, time left: 2 sec
12%, time left: 2 sec
18%, time left: 2 sec
25%, time left: 1 sec
31%, time left: 1 sec
37%, time left: 1 sec
43%, time left: 1 sec
50%, time left: 1 sec
56%, time left: 1 sec
62%, time left: 1 sec
68%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
93%, time left: 0 sec
                         : Elapsed time for training with 400 events: 2.71 sec         
                         : Dataset[dataset] : Create results for training
                         : Dataset[dataset] : Multiclass evaluation of BDTG on training sample
0%, time left: unknown
6%, time left: 0 sec
12%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
31%, time left: 0 sec
37%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
56%, time left: 0 sec
62%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
94%, time left: 0 sec
                         : Dataset[dataset] : Elapsed time for evaluation of 400 events: 0.305 sec       
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
                         : Creating xml weight file: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_BDTG.weights.xml[NON-XML-CHAR-0x1B][0m
                         : Creating standalone class: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_BDTG.class.C[NON-XML-CHAR-0x1B][0m
                         : TMVAMulticlass.root:/dataset/Method_BDT/BDTG
Factory                  : Training finished
                         : 
Factory                  : Train method: MLP for Multiclass classification
                         : 
                         : Training Network
                         : 
0%, time left: unknown
6%, time left: 5 sec
12%, time left: 5 sec
18%, time left: 5 sec
25%, time left: 4 sec
31%, time left: 4 sec
37%, time left: 3 sec
43%, time left: 3 sec
50%, time left: 2 sec
56%, time left: 2 sec
62%, time left: 2 sec
68%, time left: 1 sec
75%, time left: 1 sec
81%, time left: 1 sec
87%, time left: 0 sec
93%, time left: 0 sec
                         : Elapsed time for training with 400 events: 5.65 sec         
                         : Dataset[dataset] : Create results for training
                         : Dataset[dataset] : Multiclass evaluation of MLP on training sample
0%, time left: unknown
6%, time left: 0 sec
12%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
31%, time left: 0 sec
37%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
56%, time left: 0 sec
62%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
94%, time left: 0 sec
                         : Dataset[dataset] : Elapsed time for evaluation of 400 events: 0.00531 sec       
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
                         : Creating xml weight file: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_MLP.weights.xml[NON-XML-CHAR-0x1B][0m
                         : Creating standalone class: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_MLP.class.C[NON-XML-CHAR-0x1B][0m
                         : Write special histos to file: TMVAMulticlass.root:/dataset/Method_MLP/MLP
Factory                  : Training finished
                         : 
                         : Ranking input variables (method specific)...
BDTG                     : Ranking result (top variable is best ranked)
                         : --------------------------------------
                         : Rank : Variable  : Variable Importance
                         : --------------------------------------
                         :    1 : var4      : 2.781e-01
                         :    2 : var1      : 2.594e-01
                         :    3 : var2      : 2.521e-01
                         :    4 : var3      : 2.104e-01
                         : --------------------------------------
MLP                      : Ranking result (top variable is best ranked)
                         : -----------------------------
                         : Rank : Variable  : Importance
                         : -----------------------------
                         :    1 : var4      : 2.973e+01
                         :    2 : var3      : 2.807e+01
                         :    3 : var2      : 2.668e+01
                         :    4 : var1      : 1.650e+01
                         : -----------------------------
Factory                  : === Destroy and recreate all methods via weight files for testing ===
                         : 
                         : Reading weight file: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_BDTG.weights.xml[NON-XML-CHAR-0x1B][0m
                         : Reading weight file: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_MLP.weights.xml[NON-XML-CHAR-0x1B][0m
MLP                      : Building Network. 
                         : Initializing weights
Factory                  : [NON-XML-CHAR-0x1B][1mTest all methods[NON-XML-CHAR-0x1B][0m
Factory                  : Test method: BDTG for Multiclass classification performance
                         : 
                         : Dataset[dataset] : Create results for testing
                         : Dataset[dataset] : Multiclass evaluation of BDTG on testing sample
0%, time left: unknown
6%, time left: 0 sec
12%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
31%, time left: 0 sec
37%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
56%, time left: 0 sec
62%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
94%, time left: 0 sec
                         : Dataset[dataset] : Elapsed time for evaluation of 400 events: 0.257 sec       
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
Factory                  : Test method: MLP for Multiclass classification performance
                         : 
                         : Dataset[dataset] : Create results for testing
                         : Dataset[dataset] : Multiclass evaluation of MLP on testing sample
0%, time left: unknown
6%, time left: 0 sec
12%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
31%, time left: 0 sec
37%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
56%, time left: 0 sec
62%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
94%, time left: 0 sec
                         : Dataset[dataset] : Elapsed time for evaluation of 400 events: 0.00586 sec       
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
Factory                  : [NON-XML-CHAR-0x1B][1mEvaluate all methods[NON-XML-CHAR-0x1B][0m
                         : Evaluate multiclass classification method: BDTG
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
TFHandler_BDTG           : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:   0.054327    0.99140   [    -3.1150     2.9998 ]
                         :     var2:    0.38917     1.0308   [    -3.6952     3.1113 ]
                         :     var3:    0.17354     1.0956   [    -3.3587     3.9796 ]
                         :     var4:  0.0025079     1.3058   [    -3.7913     4.1179 ]
                         : -----------------------------------------------------------
                         : Evaluate multiclass classification method: MLP
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
TFHandler_MLP            : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:   0.054327    0.99140   [    -3.1150     2.9998 ]
                         :     var2:    0.38917     1.0308   [    -3.6952     3.1113 ]
                         :     var3:    0.17354     1.0956   [    -3.3587     3.9796 ]
                         :     var4:  0.0025079     1.3058   [    -3.7913     4.1179 ]
                         : -----------------------------------------------------------
                         : 
                         : 1-vs-rest performance metrics per class
                         : -------------------------------------------------------------------------------------------------------
                         : 
                         : Considers the listed class as signal and the other classes
                         : as background, reporting the resulting binary performance.
                         : A score of 0.820 (0.850) means 0.820 was acheived on the
                         : test set and 0.850 on the training set.
                         : 
                         : Dataset        MVA Method     ROC AUC        Sig eff@B=0.01 Sig eff@B=0.10 Sig eff@B=0.30 
                         : Name:          / Class:       test  (train)  test  (train)  test  (train)  test  (train)  
                         : 
                         : dataset        BDTG           
                         : ------------------------------
                         :                Signal         0.973 (1.000)  0.640 (1.000)  0.930 (1.000)  1.000 (1.000)  
                         :                bg0            0.880 (0.987)  0.180 (0.730)  0.600 (0.970)  0.890 (1.000)  
                         :                bg1            0.904 (0.991)  0.330 (0.780)  0.700 (0.990)  0.910 (1.000)  
                         :                bg2            0.976 (0.998)  0.350 (0.980)  0.960 (1.000)  1.000 (1.000)  
                         : 
                         : dataset        MLP            
                         : ------------------------------
                         :                Signal         0.938 (0.988)  0.350 (0.780)  0.840 (0.970)  0.970 (1.000)  
                         :                bg0            0.879 (0.936)  0.170 (0.330)  0.720 (0.790)  0.880 (0.970)  
                         :                bg1            0.944 (0.966)  0.270 (0.440)  0.880 (0.890)  0.950 (1.000)  
                         :                bg2            0.952 (0.980)  0.170 (0.760)  0.850 (0.930)  0.980 (0.990)  
                         : 
                         : -------------------------------------------------------------------------------------------------------
                         : 
                         : 
                         : Confusion matrices for all methods
                         : -------------------------------------------------------------------------------------------------------
                         : 
                         : Does a binary comparison between the two classes given by a 
                         : particular row-column combination. In each case, the class 
                         : given by the row is considered signal while the class given 
                         : by the column index is considered background.
                         : 
                         : === Showing confusion matrix for method : BDTG           
                         : (Signal Efficiency for Background Efficiency 0.01%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              1.000 (0.180)  1.000 (0.850)  1.000 (0.640) 
                         :  bg0            0.990 (0.450)  -              0.420 (0.020)  0.750 (0.570) 
                         :  bg1            0.980 (0.690)  0.650 (0.330)  -              0.790 (0.620) 
                         :  bg2            1.000 (0.350)  0.810 (0.470)  0.980 (0.210)  -             
                         : 
                         : (Signal Efficiency for Background Efficiency 0.10%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              1.000 (0.900)  1.000 (0.990)  1.000 (0.940) 
                         :  bg0            1.000 (0.620)  -              0.930 (0.510)  0.990 (0.890) 
                         :  bg1            1.000 (0.880)  0.940 (0.450)  -              1.000 (0.880) 
                         :  bg2            1.000 (0.990)  1.000 (0.990)  1.000 (0.810)  -             
                         : 
                         : (Signal Efficiency for Background Efficiency 0.30%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              1.000 (0.970)  1.000 (1.000)  1.000 (1.000) 
                         :  bg0            1.000 (0.890)  -              0.970 (0.710)  1.000 (0.990) 
                         :  bg1            1.000 (0.940)  0.990 (0.780)  -              1.000 (0.950) 
                         :  bg2            1.000 (1.000)  1.000 (1.000)  1.000 (1.000)  -             
                         : 
                         : === Showing confusion matrix for method : MLP            
                         : (Signal Efficiency for Background Efficiency 0.01%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              0.440 (0.110)  0.980 (0.570)  0.660 (0.360) 
                         :  bg0            0.160 (0.170)  -              0.290 (0.040)  0.440 (0.140) 
                         :  bg1            0.990 (0.590)  0.180 (0.060)  -              0.110 (0.490) 
                         :  bg2            0.760 (0.100)  0.710 (0.010)  0.730 (0.480)  -             
                         : 
                         : (Signal Efficiency for Background Efficiency 0.10%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              0.950 (0.760)  1.000 (0.970)  0.970 (0.840) 
                         :  bg0            0.910 (0.440)  -              0.670 (0.720)  0.910 (0.880) 
                         :  bg1            1.000 (0.980)  0.780 (0.610)  -              0.990 (0.940) 
                         :  bg2            0.940 (0.810)  0.920 (0.950)  0.940 (0.890)  -             
                         : 
                         : (Signal Efficiency for Background Efficiency 0.30%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              0.990 (0.920)  1.000 (0.970)  1.000 (0.980) 
                         :  bg0            0.990 (0.880)  -              0.910 (0.840)  0.980 (0.960) 
                         :  bg1            1.000 (0.990)  0.910 (0.890)  -              1.000 (0.990) 
                         :  bg2            0.990 (0.940)  0.990 (1.000)  0.990 (0.980)  -             
                         : 
                         : -------------------------------------------------------------------------------------------------------
                         : 
Dataset:dataset          : Created tree 'TestTree' with 400 events
                         : 
Dataset:dataset          : Created tree 'TrainTree' with 400 events
                         : 
Factory                  : [NON-XML-CHAR-0x1B][1mThank you for using TMVA![NON-XML-CHAR-0x1B][0m
                         : [NON-XML-CHAR-0x1B][1mFor citation information, please visit: http://tmva.sf.net/citeTMVA.html[NON-XML-CHAR-0x1B][0m
==> Wrote root file: TMVAMulticlass.root
==> TMVAMulticlass is done!