Execution Time18.65s

Test: tutorial-tmva-TMVAMulticlass (Passed)
Build: master-x86_64-centos7-clang100-opt-no-rt-cxxmodules (olsnba08.cern.ch) on 2019-11-14 01:39:53
Repository revision: 32b17abcda23e44b64218a42d0ca69cb30cda7e0

Test Timing: Passed
Processors1

Show Command Line
Display graphs:

Test output
Processing /data/sftnight/workspace/root-benchmark-no-rt-cxxmodules/BUILDTYPE/Release/COMPILER/clang_gcc62/LABEL/performance-sandy-cc7/root/tutorials/tmva/TMVAMulticlass.C...

==> Start TMVAMulticlass
create data set info dataset
--- TMVAMulticlass   : Accessing ./tmva_example_multiple_background.root
DataSetInfo              : [dataset] : Added class "Signal"
                         : Add Tree TreeS of type Signal with 200 events
DataSetInfo              : [dataset] : Added class "bg0"
                         : Add Tree TreeB0 of type bg0 with 200 events
DataSetInfo              : [dataset] : Added class "bg1"
                         : Add Tree TreeB1 of type bg1 with 200 events
DataSetInfo              : [dataset] : Added class "bg2"
                         : Add Tree TreeB2 of type bg2 with 200 events
                         : Dataset[dataset] : Class index : 0  name : Signal
                         : Dataset[dataset] : Class index : 1  name : bg0
                         : Dataset[dataset] : Class index : 2  name : bg1
                         : Dataset[dataset] : Class index : 3  name : bg2
Factory                  : Booking method: [NON-XML-CHAR-0x1B][1mBDTG[NON-XML-CHAR-0x1B][0m
                         : 
                         : the option NegWeightTreatment=InverseBoostNegWeights does not exist for BoostType=Grad
                         : --> change to new default NegWeightTreatment=Pray
                         : Building event vectors for type 2 Signal
                         : Dataset[dataset] :  create input formulas for tree TreeS
                         : Building event vectors for type 2 bg0
                         : Dataset[dataset] :  create input formulas for tree TreeB0
                         : Building event vectors for type 2 bg1
                         : Dataset[dataset] :  create input formulas for tree TreeB1
                         : Building event vectors for type 2 bg2
                         : Dataset[dataset] :  create input formulas for tree TreeB2
DataSetFactory           : [dataset] : Number of events in input trees
                         : 
                         : 
                         : 
                         : 
                         : Number of training and testing events
                         : ---------------------------------------------------------------------------
                         : Signal -- training events            : 100
                         : Signal -- testing events             : 100
                         : Signal -- training and testing events: 200
                         : bg0    -- training events            : 100
                         : bg0    -- testing events             : 100
                         : bg0    -- training and testing events: 200
                         : bg1    -- training events            : 100
                         : bg1    -- testing events             : 100
                         : bg1    -- training and testing events: 200
                         : bg2    -- training events            : 100
                         : bg2    -- testing events             : 100
                         : bg2    -- training and testing events: 200
                         : 
DataSetInfo              : Correlation matrix (Signal):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  +0.485  +0.637  +0.878
                         :    var2:  +0.485  +1.000  +0.752  +0.759
                         :    var3:  +0.637  +0.752  +1.000  +0.840
                         :    var4:  +0.878  +0.759  +0.840  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (bg0):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  +0.377  +0.577  +0.847
                         :    var2:  +0.377  +1.000  +0.745  +0.722
                         :    var3:  +0.577  +0.745  +1.000  +0.811
                         :    var4:  +0.847  +0.722  +0.811  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (bg1):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  +0.385  +0.594  +0.838
                         :    var2:  +0.385  +1.000  +0.754  +0.739
                         :    var3:  +0.594  +0.754  +1.000  +0.833
                         :    var4:  +0.838  +0.739  +0.833  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (bg2):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  -0.637  +0.070  +0.294
                         :    var2:  -0.637  +1.000  -0.060  -0.287
                         :    var3:  +0.070  -0.060  +1.000  +0.067
                         :    var4:  +0.294  -0.287  +0.067  +1.000
                         : ----------------------------------------
DataSetFactory           : [dataset] :  
                         : 
Factory                  : Booking method: [NON-XML-CHAR-0x1B][1mMLP[NON-XML-CHAR-0x1B][0m
                         : 
MLP                      : Building Network. 
                         : Initializing weights
Factory                  : [NON-XML-CHAR-0x1B][1mTrain all methods[NON-XML-CHAR-0x1B][0m
Factory                  : [dataset] : Create Transformation "I" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
Factory                  : [dataset] : Create Transformation "D" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
Factory                  : [dataset] : Create Transformation "P" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
Factory                  : [dataset] : Create Transformation "G" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
Factory                  : [dataset] : Create Transformation "D" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:    0.10369    0.98551   [    -3.1150     2.9998 ]
                         :     var2:    0.37389     1.0384   [    -3.4854     3.1113 ]
                         :     var3:    0.22320     1.1154   [    -3.0033     3.9796 ]
                         :     var4:   0.051417     1.2668   [    -3.2294     4.1179 ]
                         : -----------------------------------------------------------
                         : Preparing the Decorrelation transformation...
TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:    0.10863     1.0000   [    -2.8503     2.7119 ]
                         :     var2:    0.37298     1.0000   [    -2.6596     2.7270 ]
                         :     var3:    0.14120     1.0000   [    -2.5526     3.1841 ]
                         :     var4:   -0.12201     1.0000   [    -2.7592     2.4423 ]
                         : -----------------------------------------------------------
                         : Preparing the Principle Component (PCA) transformation...
TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1: 1.4040e-09     1.8033   [    -5.7995     6.0945 ]
                         :     var2: 3.0617e-10    0.90659   [    -2.6495     2.7301 ]
                         :     var3: 3.7136e-10    0.71839   [    -2.2171     1.6283 ]
                         :     var4: 1.4875e-09    0.55556   [    -1.6975     1.7716 ]
                         : -----------------------------------------------------------
                         : Preparing the Gaussian transformation...
                         : Preparing the Decorrelation transformation...
TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:   0.087552     1.0000   [    -1.9476     6.3059 ]
                         :     var2:   0.097998     1.0000   [    -2.0181     5.7448 ]
                         :     var3:   0.078771     1.0000   [    -2.1973     5.9363 ]
                         :     var4:   0.047092     1.0000   [    -2.5243     5.4080 ]
                         : -----------------------------------------------------------
                         : Ranking input variables (method unspecific)...
Factory                  : Train method: BDTG for Multiclass classification
                         : 
                         : Training 1000 Decision Trees ... patience please
0%, time left: unknown
6%, time left: 2 sec
12%, time left: 2 sec
18%, time left: 2 sec
25%, time left: 2 sec
31%, time left: 1 sec
37%, time left: 1 sec
43%, time left: 1 sec
50%, time left: 1 sec
56%, time left: 1 sec
62%, time left: 1 sec
68%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
93%, time left: 0 sec
                         : Elapsed time for training with 400 events: 2.9 sec         
                         : Dataset[dataset] : Create results for training
                         : Dataset[dataset] : Multiclass evaluation of BDTG on training sample
0%, time left: unknown
6%, time left: 0 sec
12%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
31%, time left: 0 sec
37%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
56%, time left: 0 sec
62%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
94%, time left: 0 sec
                         : Dataset[dataset] : Elapsed time for evaluation of 400 events: 0.36 sec       
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
                         : Creating xml weight file: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_BDTG.weights.xml[NON-XML-CHAR-0x1B][0m
                         : Creating standalone class: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_BDTG.class.C[NON-XML-CHAR-0x1B][0m
                         : TMVAMulticlass.root:/dataset/Method_BDT/BDTG
Factory                  : Training finished
                         : 
Factory                  : Train method: MLP for Multiclass classification
                         : 
                         : Training Network
                         : 
0%, time left: unknown
6%, time left: 6 sec
12%, time left: 6 sec
18%, time left: 4 sec
25%, time left: 4 sec
31%, time left: 4 sec
37%, time left: 3 sec
43%, time left: 3 sec
50%, time left: 3 sec
56%, time left: 2 sec
62%, time left: 2 sec
68%, time left: 2 sec
75%, time left: 1 sec
81%, time left: 1 sec
87%, time left: 0 sec
93%, time left: 0 sec
                         : Elapsed time for training with 400 events: 6.9 sec         
                         : Dataset[dataset] : Create results for training
                         : Dataset[dataset] : Multiclass evaluation of MLP on training sample
0%, time left: unknown
6%, time left: 0 sec
12%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
31%, time left: 0 sec
37%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
56%, time left: 0 sec
62%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
94%, time left: 0 sec
                         : Dataset[dataset] : Elapsed time for evaluation of 400 events: 0.0174 sec       
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
                         : Creating xml weight file: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_MLP.weights.xml[NON-XML-CHAR-0x1B][0m
                         : Creating standalone class: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_MLP.class.C[NON-XML-CHAR-0x1B][0m
                         : Write special histos to file: TMVAMulticlass.root:/dataset/Method_MLP/MLP
Factory                  : Training finished
                         : 
                         : Ranking input variables (method specific)...
BDTG                     : Ranking result (top variable is best ranked)
                         : --------------------------------------
                         : Rank : Variable  : Variable Importance
                         : --------------------------------------
                         :    1 : var4      : 2.813e-01
                         :    2 : var1      : 2.630e-01
                         :    3 : var2      : 2.584e-01
                         :    4 : var3      : 1.973e-01
                         : --------------------------------------
MLP                      : Ranking result (top variable is best ranked)
                         : -----------------------------
                         : Rank : Variable  : Importance
                         : -----------------------------
                         :    1 : var4      : 3.372e+01
                         :    2 : var1      : 2.443e+01
                         :    3 : var2      : 2.044e+01
                         :    4 : var3      : 1.401e+01
                         : -----------------------------
Factory                  : === Destroy and recreate all methods via weight files for testing ===
                         : 
                         : Reading weight file: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_BDTG.weights.xml[NON-XML-CHAR-0x1B][0m
                         : Reading weight file: [NON-XML-CHAR-0x1B][0;36mdataset/weights/TMVAMulticlass_MLP.weights.xml[NON-XML-CHAR-0x1B][0m
MLP                      : Building Network. 
                         : Initializing weights
Factory                  : [NON-XML-CHAR-0x1B][1mTest all methods[NON-XML-CHAR-0x1B][0m
Factory                  : Test method: BDTG for Multiclass classification performance
                         : 
                         : Dataset[dataset] : Create results for testing
                         : Dataset[dataset] : Multiclass evaluation of BDTG on testing sample
0%, time left: unknown
6%, time left: 0 sec
12%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
31%, time left: 0 sec
37%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
56%, time left: 0 sec
62%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
94%, time left: 0 sec
                         : Dataset[dataset] : Elapsed time for evaluation of 400 events: 0.321 sec       
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
Factory                  : Test method: MLP for Multiclass classification performance
                         : 
                         : Dataset[dataset] : Create results for testing
                         : Dataset[dataset] : Multiclass evaluation of MLP on testing sample
                         : Dataset[dataset] : Elapsed time for evaluation of 400 events: 0.00573 sec       
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
0%, time left: unknown
6%, time left: 0 sec
12%, time left: 0 sec
19%, time left: 0 sec
25%, time left: 0 sec
31%, time left: 0 sec
37%, time left: 0 sec
44%, time left: 0 sec
50%, time left: 0 sec
56%, time left: 0 sec
62%, time left: 0 sec
69%, time left: 0 sec
75%, time left: 0 sec
81%, time left: 0 sec
87%, time left: 0 sec
94%, time left: 0 sec
Factory                  : [NON-XML-CHAR-0x1B][1mEvaluate all methods[NON-XML-CHAR-0x1B][0m
                         : Evaluate multiclass classification method: BDTG
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
TFHandler_BDTG           : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:   0.073601    0.94047   [    -2.7150     2.5998 ]
                         :     var2:    0.35444    0.96614   [    -3.6952     2.5113 ]
                         :     var3:    0.19994     1.0693   [    -3.3587     3.3281 ]
                         :     var4:   0.032780     1.1827   [    -3.7913     3.5074 ]
                         : -----------------------------------------------------------
                         : Evaluate multiclass classification method: MLP
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
TFHandler_MLP            : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:   0.073601    0.94047   [    -2.7150     2.5998 ]
                         :     var2:    0.35444    0.96614   [    -3.6952     2.5113 ]
                         :     var3:    0.19994     1.0693   [    -3.3587     3.3281 ]
                         :     var4:   0.032780     1.1827   [    -3.7913     3.5074 ]
                         : -----------------------------------------------------------
                         : 
                         : 1-vs-rest performance metrics per class
                         : -------------------------------------------------------------------------------------------------------
                         : 
                         : Considers the listed class as signal and the other classes
                         : as background, reporting the resulting binary performance.
                         : A score of 0.820 (0.850) means 0.820 was acheived on the
                         : test set and 0.850 on the training set.
                         : 
                         : Dataset        MVA Method     ROC AUC        Sig eff@B=0.01 Sig eff@B=0.10 Sig eff@B=0.30 
                         : Name:          / Class:       test  (train)  test  (train)  test  (train)  test  (train)  
                         : 
                         : dataset        BDTG           
                         : ------------------------------
                         :                Signal         0.951 (1.000)  0.350 (1.000)  0.870 (1.000)  0.970 (1.000)  
                         :                bg0            0.869 (0.982)  0.200 (0.670)  0.610 (0.940)  0.880 (1.000)  
                         :                bg1            0.940 (0.983)  0.380 (0.500)  0.840 (0.950)  0.970 (1.000)  
                         :                bg2            0.971 (0.998)  0.660 (0.950)  0.940 (1.000)  0.970 (1.000)  
                         : 
                         : dataset        MLP            
                         : ------------------------------
                         :                Signal         0.943 (0.998)  0.220 (0.980)  0.820 (0.990)  0.960 (1.000)  
                         :                bg0            0.889 (0.976)  0.180 (0.740)  0.700 (0.920)  0.910 (0.970)  
                         :                bg1            0.940 (0.978)  0.280 (0.510)  0.810 (0.970)  0.970 (1.000)  
                         :                bg2            0.946 (0.991)  0.430 (0.860)  0.840 (0.970)  0.970 (1.000)  
                         : 
                         : -------------------------------------------------------------------------------------------------------
                         : 
                         : 
                         : Confusion matrices for all methods
                         : -------------------------------------------------------------------------------------------------------
                         : 
                         : Does a binary comparison between the two classes given by a 
                         : particular row-column combination. In each case, the class 
                         : given by the row is considered signal while the class given 
                         : by the column index is considered background.
                         : 
                         : === Showing confusion matrix for method : BDTG           
                         : (Signal Efficiency for Background Efficiency 0.01%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              1.000 (0.130)  1.000 (0.350)  1.000 (0.490) 
                         :  bg0            0.660 (0.020)  -              0.330 (0.200)  0.810 (0.530) 
                         :  bg1            0.990 (0.850)  0.400 (0.170)  -              0.740 (0.530) 
                         :  bg2            1.000 (0.350)  0.860 (0.680)  0.870 (0.700)  -             
                         : 
                         : (Signal Efficiency for Background Efficiency 0.10%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              1.000 (0.780)  1.000 (0.930)  1.000 (0.880) 
                         :  bg0            1.000 (0.550)  -              0.920 (0.520)  0.990 (0.780) 
                         :  bg1            1.000 (0.970)  0.920 (0.630)  -              0.950 (0.880) 
                         :  bg2            1.000 (0.860)  1.000 (0.940)  1.000 (0.960)  -             
                         : 
                         : (Signal Efficiency for Background Efficiency 0.30%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              1.000 (0.920)  1.000 (0.990)  1.000 (0.990) 
                         :  bg0            1.000 (0.950)  -              0.940 (0.790)  1.000 (0.940) 
                         :  bg1            1.000 (0.980)  0.980 (0.880)  -              1.000 (0.980) 
                         :  bg2            1.000 (0.990)  1.000 (0.970)  1.000 (0.970)  -             
                         : 
                         : === Showing confusion matrix for method : MLP            
                         : (Signal Efficiency for Background Efficiency 0.01%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              0.980 (0.060)  1.000 (0.220)  0.940 (0.360) 
                         :  bg0            0.830 (0.010)  -              0.520 (0.180)  0.860 (0.380) 
                         :  bg1            1.000 (0.760)  0.210 (0.070)  -              0.630 (0.220) 
                         :  bg2            0.860 (0.170)  0.480 (0.430)  0.690 (0.520)  -             
                         : 
                         : (Signal Efficiency for Background Efficiency 0.10%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              0.990 (0.840)  1.000 (0.990)  0.990 (0.780) 
                         :  bg0            0.970 (0.660)  -              0.890 (0.670)  0.950 (0.840) 
                         :  bg1            1.000 (0.990)  0.890 (0.680)  -              0.970 (0.790) 
                         :  bg2            0.970 (0.840)  1.000 (0.880)  0.960 (0.840)  -             
                         : 
                         : (Signal Efficiency for Background Efficiency 0.30%)
                         : ---------------------------------------------------
                         :                 Signal         bg0            bg1            bg2           
                         :                  test (train)   test (train)   test (train)   test (train) 
                         :  Signal         -              1.000 (0.930)  1.000 (1.000)  1.000 (0.940) 
                         :  bg0            1.000 (0.940)  -              0.950 (0.840)  1.000 (0.950) 
                         :  bg1            1.000 (0.990)  0.970 (0.890)  -              1.000 (0.990) 
                         :  bg2            1.000 (0.930)  1.000 (0.980)  1.000 (0.980)  -             
                         : 
                         : -------------------------------------------------------------------------------------------------------
                         : 
Dataset:dataset          : Created tree 'TestTree' with 400 events
                         : 
Dataset:dataset          : Created tree 'TrainTree' with 400 events
                         : 
Factory                  : [NON-XML-CHAR-0x1B][1mThank you for using TMVA![NON-XML-CHAR-0x1B][0m
                         : [NON-XML-CHAR-0x1B][1mFor citation information, please visit: http://tmva.sf.net/citeTMVA.html[NON-XML-CHAR-0x1B][0m
==> Wrote root file: TMVAMulticlass.root
==> TMVAMulticlass is done!