What is the 'fundamental' idea of machine learning for estimating parameters?

Question

The 'fundamental' idea of statistics for estimating parameters is maximum likelihood. I am wondering what is the corresponding idea in machine learning.

Qn 1. Would it be fair to say that the 'fundamental' idea in machine learning for estimating parameters is: 'Loss Functions'

[Note: It is my impression that machine learning algorithms often optimize a loss function and hence the above question.]

Qn 2: Is there any literature that attempts to bridge the gap between statistics and machine learning?

[Note: Perhaps, by way of relating loss functions to maximum likelihood. (e.g., OLS is equivalent to maximum likelihood for normally distributed errors etc)]

Answer 1

If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss. Since you don't know the loss you will incur on future data, you minimize an approximation, ie empirical loss.

For instance, if you have a prediction task and are evaluated by the number of misclassifications, you could train parameters so that resulting model produces the smallest number of misclassifications on the training data. "Number of misclassifications" (ie, 0-1 loss) is a hard loss function to work with because it's not differentiable, so you approximate it with a smooth "surrogate". For instance, log loss is an upper bound on 0-1 loss, so you could minimize that instead, and this will turn out to be the same as maximizing conditional likelihood of the data. With parametric model this approach becomes equivalent to logistic regression.

In a structured modeling task, and log-loss approximation of 0-1 loss, you get something different from maximum conditional likelihood, you will instead maximize product of (conditional) marginal likelihoods.

To get better approximation of loss, people noticed that training model to minimize loss and using that loss as an estimate of future loss is an overly optimistic estimate. So for more accurate (true future loss) minimization they add a bias correction term to empirical loss and minimize that, this is known as structured risk minimization.

In practice, figuring out the right bias correction term may be too hard, so you add an expression "in the spirit" of the bias correction term, for instance, sum of squares of parameters. In the end, almost all parametric machine learning supervised classification approaches end up training the model to minimize the following

$\sum_{i} L(\textrm{m}(x_i,w),y_i) + P(w)$

where $\textrm{m}$ is your model parametrized by vector $w$, $i$ is taken over all datapoints $\{x_i,y_i\}$, $L$ is some computationally nice approximation of your true loss and $P(w)$ is some bias-correction/regularization term

For instance if your $x \in \{-1,1\}^d$, $y \in \{-1,1\}$, a typical approach would be to let $\textrm{m}(x)=\textrm{sign}(w \cdot x)$, $L(\textrm{m}(x),y)=-\log(y \times (x \cdot w))$, $P(w)=q \times (w \cdot w)$, and choose $q$ by cross validation

Answer 2

I will give an itemized answer. Can provide more citations on demand, although this is not really controversial.

• Statistics is not all about maximizing (log)-likelihood. That's anathema to principled bayesians who just update their posteriors or propagate their beliefs through an appropriate model.
• A lot of statistics is about loss minimization. And so is a lot of Machine Learning. Empirical loss minimization has a different meaning in ML. For a clear, narrative view, check out Vapnik's "The nature of statistical learning"
• Machine Learning is not all about loss minimization. First, because there are a lot of bayesians in ML; second, because a number of applications in ML have to do with temporal learning and approximate DP. Sure, there is an objective function, but it has a very different meaning than in "statistical" learning.

I don't think there is a gap between the fields, just many different approaches, all overlapping to some degree. I don't feel the need to make them into systematic disciplines with well-defined differences and similarities, and given the speed at which they evolve, I think it's a doomed enterprise anyway.

Answer 3

I can't post a comment (the appropriate place for this comment) as I don't have enough reputation, but the answer accepted as the best answer by the question owner misses the point.

"If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss."

The likelihood is a loss function. Maximising likelihood is the same as minimising a loss function: the deviance, which is just -2 times the log-likelihood function. Similarly finding a least squares solution is about minimising the loss function describing the residual sum of squares.

Both ML and stats use algorithms to optimise the fit of some function (in the broadest terms) to data. Optimisation necessarily involves minimising some loss function.

Answer 4

There is a trivial answer -- there is no parameter estimation in machine learning! We don't assume that our models are equivalent to some hidden background models; we treat both reality and the model as black boxes and we try to shake the model box (train in official terminology) so that its output will be similar to that of the reality box.

The concept of not only likelihood but the whole model selection based on the training data is replaced by optimizing the accuracy (whatever defined; in principle the goodness in desired use) on the unseen data; this allows to optimize both precision and recall in a coupled manner. This leads to the concept of an ability to generalize, which is achieved in different ways depending on the learner type.

The answer to the question two depends highly on definitions; still I think that the nonparametric statistics is something that connects the two.

Answer 5

I don't think there is a fundamental idea around parameter estimation in Machine Learning. The ML crowd will happily maximize the likelihood or the posterior, as long as the algorithms are efficient and predict "accurately". The focus is on computation, and results from statistics are widely used.

If you're looking for fundamental ideas in general, then in computational learning theory, PAC is central; in statistical learning theory, structural risk miniminization; and there are other areas (for example, see the Prediction Science post by John Langford).

On bridging statistics/ML, the divide seems exagerrated. I liked gappy's answer to the "Two Cultures" question.

Answer 6

You can rewrite a likelihood-maximization problem as a loss-minimization problem by defining the loss as the negative log likelihood. If the likelihood is a product of independent probabilities or probability densities, the loss will be a sum of independent terms, which can be computed efficiently. Furthermore, if the stochastic variables are normally distributed, the corresponding loss-minimization problem will be a least squares problem.

If it is possible to create a loss-minimization problem by rewriting a likelihood-maximization, this should be to prefer to creating a loss-minimization problem from scratch, since it will give rise to a loss-minimization problem that is (hopefully) more theoretically founded and less ad hoc. For example, weights, such as in weighted least squares, which you usually have to guesstimate values for, will simply emerge from the process of rewriting the original likelihood-maximization problem and already have (hopefully) optimal values.