Inverse progression for training & validation data during training with H2O I'm training on a dataset with 3600 columns. 100948 training rows & 25238 validation rows.
These are the R commands I'm using:
train_hex = h2o.importFile(localH2O, paste(getwd(), "TrainingData.gz", sep="/"))
train_hex_split = h2o.splitFrame(train_hex, ratios = 0.8)
h2o.deeplearning(x = 242:(ncol(train_hex) - 18), y = "Predict-A",
    training_frame = train_hex_split[[1]], validation_frame = train_hex_split[[2]],
    activation = "RectifierWithDropout",
    hidden = c(1600, 1600, 1600),
    hidden_dropout_ratios = c(0.1, 0.1, 0.1),
    epochs = 100,
    initial_weight_distribution = "UniformAdaptive", 
    input_dropout_ratio = 0.1,
    l1 = 0.00001,
    l2 = 0,
    loss = "Automatic",
    shuffle_training_data = TRUE)

After about 24h of training this is the result I'm seeing.

(blue: training; orange: validation)
I'm pretty new to the field and I'm struggling to understand what's going on here? I like the training progress on the blue line. But I don't even understand how the validation can start of lower than the training!?
I'd appreciate any help.
 A: Here are the most likely explanations for what you are observing, in order of likelihood:
Train/Test split
splitFrame doesn't actually shuffle, but "cuts" the frame into two consecutive pieces (first 80% and last 20%).  If your dataset is sorted in any way, train/test samples will be biased.
Proof:
df <- as.h2o(iris)
head(df)
sp <- h2o.splitFrame(df, ratio=0.8)
head(sp[[1]]) ## same as head(df)

Better split, based on random split assignments per row:
rnd <- h2o.runif(df, seed=1234)
head(df[rnd<0.8,])
head(df[rnd>=0.8,])

Sampling for scoring
The parameter 'score_training_samples' is set to 10,000 by default (to avoid taking too long for large datasets), so only 10k rows of the training data (out of the ~100k) are used for computing the training MSE.  On the other hand, 'score_validation_samples' is set to 0 by default, so the full validation set (~25k rows) is used for computing the validation MSE.
This means that the statistical accuracy of the computed training MSE is not as good as the validation MSE, and the training sample of 10k (random) rows might include a few too many "outliers" or "hard-to-predict" observations, you can try different random splits of the data.
Distribution
Of course, all of this really depends on the distribution/data.  I assume you have a normally distributed response, and the squared error (quadratic) loss is appropriate.  Otherwise, the MSE is a less meaningful metric, and the deviance is preferred.  If you have outliers, you can try the 'gaussian' distribution with 'Absolute' or 'Huber' loss. Otherwise, you can try other distributions such as 'poisson' or 'gamma'.
Hope this helps, please feel free to ask for more questions (also see https://groups.google.com/forum/#!forum/h2ostream).
