19
$\begingroup$

The 'fundamental' idea of statistics for estimating parameters is maximum likelihood. I am wondering what is the corresponding idea in machine learning.

Qn 1. Would it be fair to say that the 'fundamental' idea in machine learning for estimating parameters is: 'Loss Functions'

[Note: It is my impression that machine learning algorithms often optimize a loss function and hence the above question.]

Qn 2: Is there any literature that attempts to bridge the gap between statistics and machine learning?

[Note: Perhaps, by way of relating loss functions to maximum likelihood. (e.g., OLS is equivalent to maximum likelihood for normally distributed errors etc)]

|cite|improve this question
$\endgroup$
  • 3
    $\begingroup$ I don't see the interest of these questions about trying to bridge a fictive gap. what is the aim of all that ? in addition there are a lot of others idea that are fundamental in statistic... and loss function is at least 100 years old. can you reduce statistic like that ? maybe your question is about fondamental concept in datamining/statistic/machine learning however you call it ... Then the question already exists and is too wide stats.stackexchange.com/questions/372/…. $\endgroup$ – robin girard Jul 28 '10 at 12:19
  • $\begingroup$ Well, I do not know much about machine learning or its connections to statistics. In any case, look at this question: stats.stackexchange.com/questions/6/… which suggests that at the very least that the approaches to answer the same questions are different. Is it that 'unnatural' to wonder if there is some sort of link between them? Yes, I agree that there lot of ideas in statistics. That is why I have fundamental in quotes and restricted the scope to estimating parameters of interest. $\endgroup$ – user28 Jul 28 '10 at 13:13
  • $\begingroup$ @Srikant link between what ? note that I really like to search link between to well defined objects, I find it really natural. $\endgroup$ – robin girard Jul 28 '10 at 13:20
  • 6
    $\begingroup$ As, arguably, a machine learner, I'm here to tell you we maximise the heck out of likelihoods. All the time. Loads of machine learning papers start with "hey look at my likelihood, look how it factorises, watch me maximise". I'd suggest that it's dangerous to claim a fundamental basis of either discipline in terms of inference techniques. It's more about which conference you go to! $\endgroup$ – Mike Dewar Aug 2 '10 at 21:47
  • 6
    $\begingroup$ I don't think Bayesians would agree with maximum likelihood being the fundamental idea of statistics. $\endgroup$ – Marc Claesen Aug 23 '15 at 9:16
17
$\begingroup$

If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss. Since you don't know the loss you will incur on future data, you minimize an approximation, ie empirical loss.

For instance, if you have a prediction task and are evaluated by the number of misclassifications, you could train parameters so that resulting model produces the smallest number of misclassifications on the training data. "Number of misclassifications" (ie, 0-1 loss) is a hard loss function to work with because it's not differentiable, so you approximate it with a smooth "surrogate". For instance, log loss is an upper bound on 0-1 loss, so you could minimize that instead, and this will turn out to be the same as maximizing conditional likelihood of the data. With parametric model this approach becomes equivalent to logistic regression.

In a structured modeling task, and log-loss approximation of 0-1 loss, you get something different from maximum conditional likelihood, you will instead maximize product of (conditional) marginal likelihoods.

To get better approximation of loss, people noticed that training model to minimize loss and using that loss as an estimate of future loss is an overly optimistic estimate. So for more accurate (true future loss) minimization they add a bias correction term to empirical loss and minimize that, this is known as structured risk minimization.

In practice, figuring out the right bias correction term may be too hard, so you add an expression "in the spirit" of the bias correction term, for instance, sum of squares of parameters. In the end, almost all parametric machine learning supervised classification approaches end up training the model to minimize the following

$\sum_{i} L(\textrm{m}(x_i,w),y_i) + P(w)$

where $\textrm{m}$ is your model parametrized by vector $w$, $i$ is taken over all datapoints $\{x_i,y_i\}$, $L$ is some computationally nice approximation of your true loss and $P(w)$ is some bias-correction/regularization term

For instance if your $x \in \{-1,1\}^d$, $y \in \{-1,1\}$, a typical approach would be to let $\textrm{m}(x)=\textrm{sign}(w \cdot x)$, $L(\textrm{m}(x),y)=-\log(y \times (x \cdot w))$, $P(w)=q \times (w \cdot w)$, and choose $q$ by cross validation

|cite|improve this answer
$\endgroup$
  • 3
    $\begingroup$ I'd love to see this loss minimizing in clustering, kNN or random ferns... $\endgroup$ – user88 Jul 29 '10 at 8:09
  • $\begingroup$ Well, for a loss function characterization of k-means nearest neighbor, see the relevant subsection (2.5) of this paper: hpl.hp.com/conferences/icml2003/papers/21.pdf $\endgroup$ – John L. Taylor Jul 30 '10 at 23:41
  • $\begingroup$ @John Still, this is mixing aims with reasons. To great extent you can explain each algorithm in terms of minimizing something and call this something "loss". kNN wasn't invented in such a way: Guys, I have thought of loss like this, let's optimize it and see what will happen!; rather Guys, let's say that decision is more less continuous over the feature space, then if we would have a good similarity measure... and so on. $\endgroup$ – user88 Aug 4 '10 at 9:38
  • 2
    $\begingroup$ "If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss" I disagree with your premise - strongly and in its entirety. Maybe it was true-ish of statistics in 1920, but it certainly is not today. $\endgroup$ – JMS Apr 13 '11 at 19:25
19
$\begingroup$

I will give an itemized answer. Can provide more citations on demand, although this is not really controversial.

  • Statistics is not all about maximizing (log)-likelihood. That's anathema to principled bayesians who just update their posteriors or propagate their beliefs through an appropriate model.
  • A lot of statistics is about loss minimization. And so is a lot of Machine Learning. Empirical loss minimization has a different meaning in ML. For a clear, narrative view, check out Vapnik's "The nature of statistical learning"
  • Machine Learning is not all about loss minimization. First, because there are a lot of bayesians in ML; second, because a number of applications in ML have to do with temporal learning and approximate DP. Sure, there is an objective function, but it has a very different meaning than in "statistical" learning.

I don't think there is a gap between the fields, just many different approaches, all overlapping to some degree. I don't feel the need to make them into systematic disciplines with well-defined differences and similarities, and given the speed at which they evolve, I think it's a doomed enterprise anyway.

|cite|improve this answer
$\endgroup$
8
$\begingroup$

I can't post a comment (the appropriate place for this comment) as I don't have enough reputation, but the answer accepted as the best answer by the question owner misses the point.

"If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss."

The likelihood is a loss function. Maximising likelihood is the same as minimising a loss function: the deviance, which is just -2 times the log-likelihood function. Similarly finding a least squares solution is about minimising the loss function describing the residual sum of squares.

Both ML and stats use algorithms to optimise the fit of some function (in the broadest terms) to data. Optimisation necessarily involves minimising some loss function.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Good point, still the main differences are somewhere else; first, statistics is about fitting a model to the data one has, ML is about fitting a model to data one will have; second, statistics ASSUME that a process one observes is fully driven by some embarassingly trivial "hidden" model that they want to excavate, while ML TRIES to make some complex enough to be problem-independent model behave like reality. $\endgroup$ – user88 Aug 4 '10 at 9:28
  • $\begingroup$ @mbq. That's a rather harsh caricature of statistics. I've worked in five university statistics departments and I don't think I've met anybody who would think of statistics like that. $\endgroup$ – Rob Hyndman Aug 4 '10 at 9:59
  • 1
    $\begingroup$ @Rob Caricature? I think this is what makes statistics beautiful! You assume all those gaussians and linearities and it just works -- and there is a reason for it which is called Taylor expansion. World is hell of a complex, but in linear approx. (which is often ninety-something% of complexity) embarrassingly trivial. ML (and nonparametric statistics) comes in in these few per cent of situations where some more subtle approach is needed. This is just no free lunch -- if you want theorems, you need assumptions; if you don't want assumptions, you need approximate methods. $\endgroup$ – user88 Aug 4 '10 at 10:54
  • $\begingroup$ @mbq. Fair enough. I must have misinterpreted your comment. $\endgroup$ – Rob Hyndman Aug 4 '10 at 11:21
4
$\begingroup$

There is a trivial answer -- there is no parameter estimation in machine learning! We don't assume that our models are equivalent to some hidden background models; we treat both reality and the model as black boxes and we try to shake the model box (train in official terminology) so that its output will be similar to that of the reality box.

The concept of not only likelihood but the whole model selection based on the training data is replaced by optimizing the accuracy (whatever defined; in principle the goodness in desired use) on the unseen data; this allows to optimize both precision and recall in a coupled manner. This leads to the concept of an ability to generalize, which is achieved in different ways depending on the learner type.

The answer to the question two depends highly on definitions; still I think that the nonparametric statistics is something that connects the two.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I'm not sure that this is entirely correct. In what sense do machine learning methods work without parameter estimation (within a parametric or distribution-free set of models)? $\endgroup$ – John L. Taylor Jul 28 '10 at 15:48
  • 1
    $\begingroup$ You are estimating/calculating something (the exact term may be different). For example, consider a neural network. Are you not calculating the weights for the net when you are trying to predict something? In addition, when you say that you train to match output to reality, you seem to be implicitly talking about some sort of loss function. $\endgroup$ – user28 Jul 28 '10 at 15:59
  • $\begingroup$ @John, @Srikant Learners have parameters, but those are not the parameters in a statistical sense. Consider linear regression y=ax (without free term for simp.). a is a parameter that statistical methods will fit, feed by the assumption that y=ax. Machine learning will just try produce ax when asked for x within the range of train (this makes sense, since it is not assuming y=ax); it may fit hundreds of parameters to do this. $\endgroup$ – user88 Jul 28 '10 at 16:07
  • 3
    $\begingroup$ [citation needed]. In other words, intriguing answer, although it doesn't jive (at least) with a lot of ML literature. $\endgroup$ – gappy Aug 2 '10 at 5:03
  • 1
    $\begingroup$ Classical one is Breiman's "Statistical Modeling: The Two Cultures". $\endgroup$ – user88 Aug 2 '10 at 6:42
2
$\begingroup$

I don't think there is a fundamental idea around parameter estimation in Machine Learning. The ML crowd will happily maximize the likelihood or the posterior, as long as the algorithms are efficient and predict "accurately". The focus is on computation, and results from statistics are widely used.

If you're looking for fundamental ideas in general, then in computational learning theory, PAC is central; in statistical learning theory, structural risk miniminization; and there are other areas (for example, see the Prediction Science post by John Langford).

On bridging statistics/ML, the divide seems exagerrated. I liked gappy's answer to the "Two Cultures" question.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Statistical crowd is clicking randomly in SPSS until desired p-value appears... $\endgroup$ – user88 Jul 29 '10 at 8:17
1
$\begingroup$

You can rewrite a likelihood-maximization problem as a loss-minimization problem by defining the loss as the negative log likelihood. If the likelihood is a product of independent probabilities or probability densities, the loss will be a sum of independent terms, which can be computed efficiently. Furthermore, if the stochastic variables are normally distributed, the corresponding loss-minimization problem will be a least squares problem.

If it is possible to create a loss-minimization problem by rewriting a likelihood-maximization, this should be to prefer to creating a loss-minimization problem from scratch, since it will give rise to a loss-minimization problem that is (hopefully) more theoretically founded and less ad hoc. For example, weights, such as in weighted least squares, which you usually have to guesstimate values for, will simply emerge from the process of rewriting the original likelihood-maximization problem and already have (hopefully) optimal values.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy