I just started to study about stats and models stuff. Currently, my understanding is that we use MLE to estimate the best parameter(s) for a model. However, when I try to understand how the neural networks work, it seems like they commonly use another approach to estimate the parameters instead. Why don't we use MLE or is it possible to use MLE at all?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
MLE estimates of artificial neural network weights (ANN) certainly are possible; indeed, it's entirely typical. For classification problems, a standard objective function is cross-entropy, which is the same as the negative log-likelihood of a binomial model. For regression problems, residual square error is used, which parallels the MLE of OLS regression. See: How to construct a cross-entropy loss for general regression targets?
But there are some problems with assuming that the nice properties of MLEs derived in classical statistics, such as uniqueness, also hold for MLEs of neural networks.
There is a general problem with ANN estimation: there are many symmetric solutions to even single-layer ANNs. Reversing the signs of the weights for the hidden layer, and reversing the signs of the hidden layer activation parameters both have equal likelihood. Additionally, you can permute any of the hidden nodes and these permutations also have the same likelihood. This is consequential insofar as you must acknowledge that you are giving up identifiability. However, if identifiability is not important, then you can simply accept that these alternative solutions are just reflections and/or permutations of each other.
This is in contrast to classical usages of MLE in statistics, such as a OLS regression: the OLS problem is convex, and strictly convex when the design matrix is full rank. Strong convexity implies that there is a single, unique minimizer.
It's true that these solutions have the same quality (same loss, same accuracy), but a number of students who arrive at neural networks from an understanding of regression are surprised to learn that NNs are non-convex and do not have unique optimal parameter estimates.
ANNs will tend to overfit the data when using an unconstrained solution. The weights will tend to race away from the origin to implausibly large values which do not generalize well or predict new data with much accuracy. Imposing weight decay or other regularization methods has the effect of shrinking weight estimates toward zero. This doesn't necessarily resolve the indeterminacy issue from (1), but it can improve the generalization of the network.
The loss function is nonconvex and optimization can find locally optimal solutions which are not globally optimal. Or perhaps these solutions are saddle points, where some optimization methods stall. The results in this paper find that modern estimation methods sidestep this issue.
In a classical statistical setting, penalized fit methods such as elastic net, $L^1$ or $L^2$ regularization can make convex a rank-deficient (i.e. non-convex) problem. This fact does not extend to the neural network setting, due to the permutation issue in (1). Even if you restrict the norm of your parameters, permuting the weights or symmetrically reversing signs won't change the norm of the parameter vector; nor will it change the likelihood. Therefore the loss will remain the same for the permuted or reflected models and the model is still non-identified.
answered Apr 11, 2015 at 19:25
-
5$\begingroup$ I beg to differ with what you say. The different local minima arising from symmetries are all of the same quality, so you don't have to worry about that at all. What you probably want to say is that ANNs do not have convex loss functions, which makes optimisation more involved and does not guarantee finding a global optimum. However, there has been quite some evidence recently that ANNs actually do not have that much of local minima issues, but rather saddle point issues. See e.g. arxiv.org/abs/1412.6544. $\endgroup$– bayerjCommented Apr 11, 2015 at 19:38
$\begingroup$
$\endgroup$
In classification problems, maximizing the likelihood is the most common way to train a neural network (both supervised and unsupervised models).
In practice, we usually minimize the negative log-likelihood (equivalent MLE). The only constraint to use the negative log-likelihood is to have an output layer that can be interpreted as a probability distribution. A softmax output layer is commonly used to do so. Note that in the neural-networks community, the negative log-likelihood is sometimes referred as the cross-entropy. Regularization terms can of course be added (and sometimes can be interpreted as prior distributions over the parameters, in that case we are looking for the maximum a posteriori (MAP)).
