31
$\begingroup$

Is "deep learning" just another term for multilevel/hierarchical modeling?

I'm much more familiar with the latter than the former, but from what I can tell, the primary difference is not in their definition, but how they are used and evaluated within their application domain.

It looks like the number of nodes in a typical "deep learning" application is larger and uses a generic hierarchical form, whereas applications of multilevel modeling typically uses a hierarchical relationships that mimic the generative process being modeled. Using a generic hierarchy in an applied statistics (hierarchical modeling) domain would be regarded as an "incorrect" model of the phenomena, whereas modeling a domain-specific hierarchy might be regarded as subverting the objective of making a generic deep learning learning machine.

Are these two things really the same machinery under two different names, used in two different ways?

$\endgroup$
38
+50
$\begingroup$

Similarity

Fundamentally both types of algorithms were developed to answer one general question in machine learning applications:

Given predictors (factors) $x_1, x_2, \ldots, x_p$ - how to incorporate the interactions between this factors in order to increase the performance?

One way is to simply introduce new predictors: $x_{p+1} = x_1x_2, x_{p+2} = x_1x_3, \ldots$ But this proves to be bad idea due to huge number of parameters and very specific type of interactions.

Both Multilevel modelling and Deep Learning algorithms answer this question by introducing much smarter model of interactions. And from this point of view they are very similar.

Difference

Now let me try to give my understanding on what is the great conceptual difference between them. In order to give some explanation, let's see the assumptions that we make in each of the models:

Multilevel modelling:$^1$ layers that reflect the data structure can be represented as a Bayesian Hierarchical Network. This network is fixed and usually comes from domain applications.

Deep Learning:$^2$ the data were generated by the interactions of many factors. The structure of interactions is not known, but can be represented as a layered factorisation: higher-level interactions are obtained by transforming lower-level representations.

The fundamental difference comes from the phrase "the structure of interactions is not known" in Deep Learning. We can assume some priors on the type of interaction, but yet the algorithm defines all the interactions during the learning procedure. On the other hand, we have to define the structure of interactions for Multilevel modelling (we learn only vary the parameters of the model afterwards).

Examples

For example, let's assume we are given three factors $x_1, x_2, x_3$ and we define $\{x_1\}$ and $\{x_2, x_3\}$ as different layers.

In the Multilevel modelling regression, for example, we will get the interactions $x_1 x_2$ and $x_1 x_3$, but we will never get the interaction $x_2 x_3$. Of course, partly the results will be affected by the correlation of the errors, but this is not that important for the example.

In Deep learning, for example in multilayered Restricted Boltzmann machines (RBM) with two hidden layers and linear activation function, we will have all the possible polinomial interactions with the degree less or equal than three.

Common advantages and disadvantages

Multilevel modelling

(-) need to define the structure of interactions

(+) results are usually easier to interpret

(+) can apply statistics methods (evaluate confidence intervals, check hypotheses)

Deep learning

(-) requires huge amount of data to train (and time for training as well)

(-) results are usually impossible to interpret (provided as a black box)

(+) no expert knowledge required

(+) once well-trained, usually outperforms most other general methods (not application specific)

Hope it will help!

$\endgroup$
  • $\begingroup$ Why does a deep neural network require a huge amount of data to train? I have not heard of this before. $\endgroup$ – Jase Feb 6 '14 at 12:02
  • $\begingroup$ @Jase Neural networks usually have lots of parameters, therefore in many cases if you use a small dataset, you will probably just overfit. Of course, everything depends on the task, but most of the most impressive NN results nowadays use extremely huge datasets. $\endgroup$ – Dmitry Laptev Feb 9 '14 at 13:15
  • $\begingroup$ Agreed that the impressive results are on huge datasets, but I'm not sure that we can't use dropout and other tricks to get it to generalise well from small datasets. $\endgroup$ – Jase Feb 9 '14 at 13:27
  • $\begingroup$ @Jase Sure, there are different heuristics you can use. But up to my experience with image processing, almost all of them introduce some regularization, which is basically equivalent to introducing some prior and therefore bias. Which is not always what you want. $\endgroup$ – Dmitry Laptev Feb 9 '14 at 13:32
  • $\begingroup$ Presumably you do not need to define the structure of the interactions if you use nonparametrics e.g. a hierarchical Dirichlet prior. $\endgroup$ – Astrid Jan 2 '16 at 1:15
2
$\begingroup$

While this question/answer has been out there for a bit, I hought it might be helpful to clarify a few points in the answer. First, the phrase raised as a major distinction between hierarchical methods and deep neural networks 'This network is fixed.' is incorrect. Hierarchical methods are no more 'fixed' than the alternative, neural networks. See, for example, the paper Deep Learning with Hierarchical Convolutional Factor Analysis, Chen et. al. . I think you will also find that the requirement to define interactions is also no longer a distinguishing point. A couple of points that are not listed as a plus with the hierarchical modeling is, from my experience, the significantly reduced problem of overfitting and the ability to handle both very large and very small training sets. A nitpick point is that when Bayesian hierarchical methods are used, confidence intervals and hypothesis testing are generally not statistical methods that would be applied.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.