1
$\begingroup$

I'm training on a dataset with 3600 columns. 100948 training rows & 25238 validation rows.

These are the R commands I'm using:

train_hex = h2o.importFile(localH2O, paste(getwd(), "TrainingData.gz", sep="/"))
train_hex_split = h2o.splitFrame(train_hex, ratios = 0.8)
h2o.deeplearning(x = 242:(ncol(train_hex) - 18), y = "Predict-A",
    training_frame = train_hex_split[[1]], validation_frame = train_hex_split[[2]],
    activation = "RectifierWithDropout",
    hidden = c(1600, 1600, 1600),
    hidden_dropout_ratios = c(0.1, 0.1, 0.1),
    epochs = 100,
    initial_weight_distribution = "UniformAdaptive", 
    input_dropout_ratio = 0.1,
    l1 = 0.00001,
    l2 = 0,
    loss = "Automatic",
    shuffle_training_data = TRUE)

After about 24h of training this is the result I'm seeing.

training graph

(blue: training; orange: validation)

I'm pretty new to the field and I'm struggling to understand what's going on here? I like the training progress on the blue line. But I don't even understand how the validation can start of lower than the training!?

I'd appreciate any help.

$\endgroup$
2
$\begingroup$

Here are the most likely explanations for what you are observing, in order of likelihood:

Train/Test split

splitFrame doesn't actually shuffle, but "cuts" the frame into two consecutive pieces (first 80% and last 20%). If your dataset is sorted in any way, train/test samples will be biased.

Proof:

df <- as.h2o(iris)
head(df)
sp <- h2o.splitFrame(df, ratio=0.8)
head(sp[[1]]) ## same as head(df)

Better split, based on random split assignments per row:

rnd <- h2o.runif(df, seed=1234)
head(df[rnd<0.8,])
head(df[rnd>=0.8,])

Sampling for scoring

The parameter 'score_training_samples' is set to 10,000 by default (to avoid taking too long for large datasets), so only 10k rows of the training data (out of the ~100k) are used for computing the training MSE. On the other hand, 'score_validation_samples' is set to 0 by default, so the full validation set (~25k rows) is used for computing the validation MSE.

This means that the statistical accuracy of the computed training MSE is not as good as the validation MSE, and the training sample of 10k (random) rows might include a few too many "outliers" or "hard-to-predict" observations, you can try different random splits of the data.

Distribution

Of course, all of this really depends on the distribution/data. I assume you have a normally distributed response, and the squared error (quadratic) loss is appropriate. Otherwise, the MSE is a less meaningful metric, and the deviance is preferred. If you have outliers, you can try the 'gaussian' distribution with 'Absolute' or 'Huber' loss. Otherwise, you can try other distributions such as 'poisson' or 'gamma'.

Hope this helps, please feel free to ask for more questions (also see https://groups.google.com/forum/#!forum/h2ostream).

$\endgroup$
  • $\begingroup$ Welcome to CV. Note that your username, identicon, & a link to your user page are automatically added to every post you make, so there is no need to sign your posts. In fact, we prefer you don't. $\endgroup$ – gung - Reinstate Monica Oct 6 '15 at 4:48
  • 1
    $\begingroup$ Note that this answer is now outdated. The R h2o.splitFrame() behavior changed in the rel-tibshirani release of H2O (Oct./Nov. 2015 timeframe). The changed behavior randomly assigns each row to a result frame using a row-by-row coin flip. See github.com/h2oai/h2o-3/commit/… $\endgroup$ – TomKraljevic Apr 23 '17 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.