2
$\begingroup$

I'm looking into understanding the Deeplearning anomaly detection algorithm provided by h2o. I tried to recreate an example below. Perhaps some of these questions are basic, but I'm trying to better understand this technique as I'm not too familiar with the approach. There seems to be a wealth of information around 'deep learning' but it often is overly technical or theoretical , making it hard for me to connect the dots if you will....

That said... how can I properly 'estimate' an ideal number of hidden layers in an h2o Autoencoders model? How can I tune this model properly?

  1. My Understanding is generally layers is most important parameter to tune. Are there good rules of thumb on how to choose this value? A given value such as 5? a % of features? I know it probably 'depends', but at same time I don't want to spend hours and hours grid-searching when it is possible that a 'rule-of-thumb' value(s) are more often than not close enough.
  2. How can I understand the print-outs of an h2o Autoencoders model output? The values returned seem to be:

    • layer
    • units
    • weight_rms
    • mean_bias
    • etc...

How should I be looking at this to ensure that my un-supervised model is appropriate?

  1. This is probably answered by the above questions but...looking at models with a variety of different hidden layers you get some pretty different looking reconstruction MSE plots. How do I know what one is ideal?

This likely isn't the best dataset to show this problem since it's relatively small, but here is an example:

library(mlbench)
library(h2o)
library(tidyverse)
data('Sonar')


h2o.init()
df=Sonar
h2odf = as.h2o(df)


features = setdiff(names(df), 'Class')

hyperparms = list(hidden=c(1,5,10,15,20,25,35))

grid=h2o.grid(algorithm = 'deeplearning', x = features, training_frame = h2odf, epochs=100, autoencoder=TRUE, hyper_params =hyperparms , activation='Tanh', grid_id='understanding_dl')

check_models=c(as.data.frame(grid@summary_table)$model_ids)



initialize_df =h2o::h2o.createFrame(rows=nrow(df), cols = 7)
counter = 0
for (each_model in check_models){
    print(h2o.getModel(each_model))
    counter = counter+1
    initialize_df[,counter]=h2o.anomaly(h2o.getModel(each_model), data=h2odf)
} 
#interesting to see correlation of values
cor(initialize_df)

recon_mse=as.data.frame(initialize_df)

names(recon_mse)=c(hyperparms$hidden)
recon_mse_gathered = recon_mse%>% mutate(record=seq(1,nrow(df)))

df2 <- melt(recon_mse_gathered ,  id.vars = 'record', variable.name = 'hidden')
 # density of recon MSE w/ each hidden layer
df2 %>% ggplot(aes(x=value, color=hidden))+geom_density()+labs(x='Recon MSE')
$\endgroup$

1 Answer 1

1
$\begingroup$

You can think of an autoencoder as a form of dimension-reduction, similar to PCA. But, unlike in PCA, all dimensions are of equal importance. The number of neurons in your hidden layer are the number of dimensions you are reducing to.

The interesting metric for an autoencoder is MSE (h2o.mse(m) where m is your model), and the lower the better. Generally, the more hidden neurons you use, the lower the MSE will be, but it will be diminishing returns, and the curve can be a bit noisy, too.

(That is my answer to your Q2: the other fields are "informational", but only MSE is of interest.)

For purposes of anomaly detection... if you have just a few features, but little correlation between them, I'd be tempted to use the same number of hidden neurons as features. If you have thousands of features, but most of them are correlated, or carrying little information (e.g. each column is zero or NA for 90+% of your samples), then I'd be thinking to try something like the square root of the number of columns.

Another approach is to use PCA (h2o.prcomp()) on your data, and see how many dimensions are needed to capture, say, 99% of the variance, and use that as the number of hidden neurons.

If the data is noisy, consider using a bit of dropout, or l1/l2 regularization, and increasing the number of neurons to compensate. Remember that you can have more hidden neurons than input columns (which is another way it differs from PCA)! You can also experiment with more than one hidden layer.


By the way, you were asking for rules of thumb, ways to avoid hyperparameter grid searches. But, in a real project, I would do a grid search, doubling and halving my rule-of-thumbed hidden neuron count. E.g. if 1000 input columns, most of which are sparse, I'd maybe try 20, 40 and 80.

I'd be looking for an MSE (and a scoring history chart) that is basically the same even after doubling the number of neurons, so I'd feel comfortable going with my original guess.

If using l1/l2 regularization, I'd also put them in the grid search, at plus and minus an order of magnitude. E.g. trying 1e-4, 1e-5, 1e-6 for each.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.