1
$\begingroup$

I am testing H2O's deep learning in comparison with random forest in R. When I used (1) IRIS dataset [150 observations/samples by 4 predictors/features; the last column shows the class labels (iris types)] for training with default parameters, both shows the "correct" confusion matrix, in the sense that each class (Iris-setosa, Iris-versicolor, Iris-virginica) has the correct number of data points of 50. However, when I used (2) MNIST training dataset [60,000 observations (rows) by 784 predictors (columns); the first column shows the class labels (digits)], the deep learning shows "incorrect" confusion matrix, in the sense that each class (0, 1, ... 9) has incorrect number of data points. Each class should have (5923, 6742,..., 5949) data points. The random forest shows the correct confusion matrix. Dose anyone know if this is an expected result, or am I missing something here? The code fragment is shown below.

Any suggestions are appreciated.

(1) IRIS dataset

> irisdata
   C1  C2  C3  C4          C5
1 5.1 3.5 1.4 0.2 Iris-setosa
2 4.9 3.0 1.4 0.2 Iris-setosa
3 4.7 3.2 1.3 0.2 Iris-setosa
4 4.6 3.1 1.5 0.2 Iris-setosa
5 5.0 3.6 1.4 0.2 Iris-setosa
6 5.4 3.9 1.7 0.4 Iris-setosa
....

> # Deep Learning
> model.dl <- h2o.deeplearning(
+  x = 1:4, y = 5,
+  data = irisdata
+)
>print(model.dl)
...
Confusion matrix:
Reported on iris.hex 
                 Predicted
Actual            Iris-setosa Iris-versicolor Iris-virginica   Error
  Iris-setosa              49               1              0 0.02000
  Iris-versicolor           0              50              0 0.00000
  Iris-virginica            0              10             40 0.20000
  Totals                   49              61             40 0.07333

# Random Forest
> model.rf <- h2o.randomForest(
+  x = 1:4, y = 5,
+  data = irisdata
+)
> print(model.rf)
...
Confusion matrix:
Reported on iris.hex 
                 Predicted
Actual            Iris-setosa Iris-versicolor Iris-virginica   Error
  Iris-setosa              49               1              0 0.02000
  Iris-versicolor           0              50              0 0.00000
  Iris-virginica            0              10             40 0.20000
  Totals                   49              61             40 0.07333

(2) MNIST dataset

> # Deep Learning
> model.dl <- h2o.deeplearning(
+   x = 2:785,
+   y = 1,
+   data = train.mnist
+ )
> model.dl
...
Confusion matrix:
Reported on Last.value.1 
        Predicted
Actual      0    1   2    3   4   5    6    7   8    9   Error
  0      1005    0   0    0   0   0    0    0   0    0 0.00000
  1         0 1118   1    0   0   0    0    0   0    2 0.00268
  2         0    0 954    0   0   0    0    0   0    0 0.00000
  3         0    0   1 1039   0   1    0    0   0    1 0.00288
  4         0    0   0    0 981   0    1    0   0    1 0.00203
  5         1    0   0    0   0 971    0    1   0    0 0.00206
  6         0    0   0    0   0   0 1004    0   0    0 0.00000
  7         0    1   0    0   0   0    0 1026   0    1 0.00195
  8         0    0   0    1   0   0    0    0 970    0 0.00103
  9         0    0   0    0   0   0    0    0   0 1063 0.00000
  Totals 1006 1119 956 1040 981 972 1005 1027 970 1068 0.00128

> # Random Forest
> model.rf <- h2o.randomForest(
+   x = 2:785,
+   y = 1,
+   data = train.mnist
+ )
> model.rf
...
Classification: TRUE
Confusion matrix:
Reported on training data.        Predicted
Actual      0    1    2    3    4    5    6    7    8    9   Error
  0      5826    1    9    7    3    7   26    4   32    8 0.01638
  1         1 6633   33   14   17    5    5   17   14    3 0.01617
  2        24   12 5699   39   30    5   25   53   58   13 0.04347
  3        11    6   76 5748    5  108   12   51   77   37 0.06247
  4        15   11   12    2 5602    3   41   15   19  122 0.04108
  5        28    6   11   95   13 5107   50    9   70   32 0.05792
  6        28    9    8    3    8   43 5793    1   25    0 0.02112
  7         7   16   61   10   34    2    1 6013   23   98 0.04022
  8        15   22   55   52   23   58   41    6 5506   73 0.05896
  9        24    7   18   73   90   34    7   55   52 5589 0.06051
  Totals 5979 6723 5982 6043 5825 5372 6001 6224 5876 5975 0.04140
$\endgroup$
2
$\begingroup$

I figured it out. By default it subsample the training data. Set the following options to use all of the training and validation data: train_samples_per_iteration = -1, score_training_samples=0, score_validation_samples=0.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.