14
$\begingroup$

I finished up a machine learning (ML) course a while back. Everything was as an optimization problem. No matter what predictive challenge you face in ML, you're generally minimizing some objective (i.e., cost) function. In so doing, you come up with "optimal" parameters that satisfy the equation you're working with (e.g., gradient descent and linear regression where MSE is the objective function you minimize).

Is it the case with all machine learning models that, when you find the optimal parameters that minimize your objective function, you are also, almost by definition, finding the same statistically significant coefficients you would otherwise discover if you were to attack the problem from a stats perspective where the focus is on tests of statistical significance? Let's define "stats perspective" as running a model in R and adding or deleting new variables based on their statistical significance or degree to which they change AIC .

$\endgroup$
2
  • 9
    $\begingroup$ Part of this is computational cost; the significance tests you describe don't scale well. In a model with (not kidding) a billion parameters trained on tens of gigabytes of data, are you really going to pluck out each variable and re-fit the model? $\endgroup$ Apr 11 at 23:56
  • 7
    $\begingroup$ You might be interested in reading the classic paper Breiman (2001). Statistical Modeling: The Two Cultures. $\endgroup$
    – jkpate
    Apr 12 at 10:33
15
$\begingroup$

Is it the case with all machine learning models that, when you find the optimal parameters that minimize your objective function, you are also, almost by definition, finding the same statistically significant coefficients you would otherwise discover if you were to attack the problem from a stats perspective where the focus is on tests of statistical significance?

In some cases, yes. If you're using logistic regression as a classifier, then optimizing the cost is the same as optimizing the log likelihood. Not every model is like this. Deep neural networks do not map nicely onto a statistical counterpart (though there is some active research on their statistical properties I imagine).

As to variable selection via AIC or similar, that would be a form of feature selection. I think that is something to ask about on its own.

As to your titular question, inference is not our main concern; its predictive capability. Besides, most ML problems (most, not all) work with data sets so large that significance becomes a straw man. The sheer size of the data would allow for high precision estimates, and since no effect is truly 0 you would find that all effects are significant.

$\endgroup$
9
  • $\begingroup$ First, thank you for the well-considered response. Second, am I right in interpreting your answer, as well as one provided by a commenter above, by thinking that the sheer volume of data and number of features makes it impracticable to perform tests of statistical significance? Hence, we rely on a strategy of optimization for large data sets? $\endgroup$
    – Ryan
    Apr 12 at 2:21
  • 7
    $\begingroup$ I mean, there are many examples where it might seem practical; not all ML problems are solved with billion parameter models. The problem for testing is that a) its hard to derive what the null should be in a lot of cases, and b) it doesn't really matter because inference is not on the menu. Take penalized models for example. Its hard to know what the null would be for a penalized model, especially if it is for something like an SVM. Its hard, it isn't what we want, so we don't bother with it. $\endgroup$ Apr 12 at 2:42
  • 1
    $\begingroup$ "Deep neural networks do not map nicely onto a statistical counterpart" In logistic regression we view the label $Y_i$ corresponding to the feature vector $x_i$ as a Bernoulli random variable, and we make the assumption that $P(Y_i = 1) = \sigma(f(x_i))$ where $\sigma$ is the sigmoid function and $f$ is a linear function. This is an arbitrary choice for the form of $f$, though. Couldn't we equally well assume that $f$ is a neural network? Then we're doing deep learning, and the optimization problem we solve is to maximize the log likelihood, just as in logistic regression. $\endgroup$
    – littleO
    Apr 13 at 6:40
  • 1
    $\begingroup$ Thank you for the explanation. I just meant I do not get the grammar of that sentence. $\endgroup$ Apr 13 at 16:11
  • 2
    $\begingroup$ @RichardHardy It is a hyperbaton $\endgroup$ Apr 13 at 16:30
11
$\begingroup$

tl;dr: Because it's 1) not possible and 2) not necessary.

Long answer: Your question is posed from a statistical perspective, where testing the parameters is a standard thing to do. This makes it sound as if the way Machine Learning does it is in need of justification. But we can equally well pose the question in the opposite direction: "Why are parameters being tested in statistics?"

The answer differs slightly, depending whether you're into inferential or predictive modelling. If you are doing inference, you want to know (understand!) which parameters are relevant to the output. If you're doing prediction, you want to avoid overfitting, and eliminating superfluous parameters helps with that. However, tests make assumptions regarding the probability distribution behind the parameters, and these assumptions may be more or less hard to justify. AIC (and BIC, too) also make assumptions. Your test and parameter elimination strategies work only as good as your assumptions are satisfied.

Machine Learning is not an inferential toolbox. It was, from the beginning, meant to generate automated systems ("machines") which can make decisions ("predictions") without human intervention. Interpretability was never an issue, and this allows the models to be of arbitrary complexity (up to, but not necessarily, billions of parameters, as mentioned in a comment). So, the information which parameters are relevant ("significant") would be only relevant to ensure generalisation. However, machine learning makes much weaker assumptions about the data and, as a rule, we know (almost) nothing about the underlying probability distributions. This makes reasonable statistical tests next to impossible. Instead, generalisation in machine learning is ensured by cross validation and testing on a separate (test) data set. But, in order for that to work half-way reliably, you need much more data then for statistical modelling.

$\endgroup$
4
  • $\begingroup$ Brilliant! Yes, the question did come from a stats perspective. I assumed something was missing from ML, but after all of these great replies, it seems ML is not missing a thing! $\endgroup$
    – Ryan
    Apr 12 at 14:53
  • $\begingroup$ "Interpretability was never an issue". Interesting. In Natural Language Processing (NLP) at least, it is a big topic recently. Since nobody wants to use a predictive system which cannot be interpreted by humans (classic example is automated medical diagnosis). $\endgroup$
    – justhalf
    Apr 14 at 16:37
  • $\begingroup$ @justhalf This can get arbitrarily philosophical. I recommend Richard Feynman's "Why is aunt Minnie in hospital", youtube.com/watch?v=36GT2zI8lVA. And of course there have alway's been people who have asked themselves why a machine has given a specific answer. But, the development of machine learning algorithms wasn't driven by the idea that analysing the internal workings of a machine would teach you something about the outer world. $\endgroup$
    – Igor F.
    Apr 14 at 17:20
  • $\begingroup$ Thanks for the video. Re: your last statement, agree, I don't think that's the idea either. Rather, it's more about how the internal workings of a machine would teach you something about the machine, so that you can justify its decision, and add missing parts if it's not there yet (example: various attention mechanisms in NLP was started from this kind of intuition: "oh, the model isn't able to predict correctly because there is no ability to pay attention to specific words, let's add that capability"). $\endgroup$
    – justhalf
    Apr 15 at 1:59
10
$\begingroup$

There are some different mindsets between classical statistics and machine learning, where modern machine learning can have huge amount of data.

And as you said, the whole problem is an "optimization problem", where people put little or no assumptions to the model. And the golden standard is the performance on a large testing set. If we go back and review all the nice properties (confidence interval, p value, etc.) they are coming from strong assumptions about the data and model.

With less data, people are willing to do more work in math (say Gauss–Markov theorem for OLS) to make the results nice and solid. But modern machine learning people are more practical and do not care too much about the assumption and statistically significant, but focusing on the model will "work" on large testing data.

Note deep neural network can have billions of parameters and the number of parameters can even much bigger than number of data points. (which may not really acceptable from classical statistics perspective) So, AIC and BIC also not working there.

$\endgroup$
2
  • $\begingroup$ Thanks so much for your reply! Scale is definitely an issue I completely forgot about. It's one of the reasons why ML is preferred to traditional stats methods! $\endgroup$
    – Ryan
    Apr 12 at 14:45
  • 3
    $\begingroup$ @Ryan Biostatisticians do inferential statistics with genes all the time. People in tech do inference at scale all the time. So big data does not imply an automatic preference for ML over statistical approaches. $\endgroup$
    – dimitriy
    Apr 12 at 23:47
9
$\begingroup$

Let's define "stats perspective" as running a model in R and adding or deleting new variables based on their statistical significance or degree to which they change AIC.

First of all, "adding or removing variables based on their significance" is not a good practice. As the name suggests, significance testing is about testing a hypothesis, it is not a tool for optimizing anything, and by using it for the variable selection you assume some kind of optimization problem.

Second, ask yourself what exactly does a "statistically significant parameter" means. We can do many different statistical tests, but the most common ones, the out-of-the-box statical tests for regression models test the hypothesis that the parameter differs from zero. If a parameter is equal to zero in the regression model, it has no effect on the results. Let's start with the fact that not every machine learning model has parameters (e.g. $k$-NN, or decision tree don't). But even if it has parameters, why would you care if they are zero or not? If the model makes correct predictions, it can have as many zero parameters as you want, worst case they would make it computationally slower than in the case of a lighter model. More than this, sometimes we intentionally drag the parameters towards zero, like with $L_1$ regularization, as means of feature selection. In such a case we want as many of the parameters to be zeros, as long as they do not worsen the results.

$\endgroup$
7
$\begingroup$

Your descriptions of Statistical Model Building and Machine Learning are imcomplete if not false. Adding and deleting data based on their AIC ist stepwise regression and that is widely discouraged because it leads to overfitting to the data at hand. So that is not what we should do in statistics.

In machine learning you left out the important aspect of splitting you data into training, tuning and test data. Let's take LASSO regression as an example. It usually does not lead to $p$-values but it has a tuning paramater $\lambda$ that decides, how easily the model accepts or rejects additional data. This tuning parameter is usually defined via cross validation so that the model should have no more or no less parameters than are optimal for prediction purposes.

A statistics models gets it's justification from some complicated-to-explain $p$-value that should lie below some completely arbitrarily chosen limit. A machine learning model gets it's justification from cross validation. So machine learning does not have something less, just something different that does not appear in your depiction of ML.

$\endgroup$
1
  • $\begingroup$ Point well taken. My question assumed best practices common to stats and ML, like train/test splitting, were being followed. Thanks for pointing out the role of tuning parameters. It's impossible to have a discussion about goodness of fit without them! $\endgroup$
    – Ryan
    Apr 12 at 14:43
4
$\begingroup$

Is it the case with all machine learning models that, when you find the optimal parameters that minimize your objective function, you are also, almost by definition, finding the same statistically significant coefficients you would otherwise discover if you were to attack the problem from a stats perspective where the focus is on tests of statistical significance?

No. Assuming the general aim in machine learning would be to minimize prediction error on future observations. In that, we often accept a little bias in our estimator, as long as its variance is reduced more strongly (through e.g., smoothing, penalization, ensembling).

In statistical significance testing (inference), at least within the frequentist approach, the aim is to obtain unbiased parameter estimates (i.e., on average, we obtain the 'true' parameter values).

Unbiasedness tends to come at the cost of (much) increased variance. Though not always: An OLS linear regression likely provides very good power and predictive accuracy if assumptions are met, the number of predictors is not too large and sample size not too small. Then the aims of prediction and inference coincide to yield the same optimal solution.

Both viewpoints are statistical. There is no real distinction between statistics and machine learning. One should never do ML without taking the sampling aspects of the problem into account (sample size and representativeness, number of possible predictors, distribution of predictors and response, etc.).

What you refer to as a 'stats perspective' is statistically a (very) unsound approach; it is misusing a confirmatory hypothesis-testing approach in an exploratory manner.

$\endgroup$
3
$\begingroup$

Many of the comments/responses by others in the thread give good points. I'll add 1 practical and 1 theoretical.

Practical: Many ML models are concerned more with prediction than with statistical inference (e.g. hypothesis tests of model parameters). See. e.g. Leo Breiman's paper on the two cultures for good elucidation on this distinction.

Theoretical: In many ML models the parameters you would be doing inference on aren't identified nor estimable so doing hypothesis testing wouldn't make sense. There are exceptions to this of course (e.g. using a GLM to predict probability of loan default) and many ML models do not have a clear likelihood,

e.g. what is the likelihood for a splitting rule on the root node of a decision tree?

In these cases you can't do hypothesis testing unless you do some deep thinking which might take longer than you have to solve the problem.

$\endgroup$
2
$\begingroup$

Many ML models are not based on a probability distribution with which you can determine statistical significance, and many are not set up to determine statistical significance. In such cases these models ordinarily reach some type of convergence criterion (e.g., the MSE delta between iterations goes below a threshold).

However, you can employ empirical hypothesis testing (p-value testing) for any model by (a) first determining the observed $MSE_{obs}$ via a single run from $k$-fold CV, and (b) then run $B=1000$ iterations (1000 runs) for $k$-fold CV but this time with permuted class labels of test objects (in test folds), each time getting $MSE^{(b)}$. Before each of the $B=1000$ iterations using $k$-fold CV with permuted class labels of test objects, it helps to "re-partition" all objects and assign them to different folds. This merely re-orders all objects before assigning to the folds -- and is a way to have different objects in each fold during each iteration.

When done, an "empirical p-value" will be equal to

$P=\frac{\#\{MSE^{(b)}<MSE_{obs} \} }{B}$

where the p-value is equal to the number of times the MSE from test objects with permuted labels is less than the MSE from test objects of the non-permuted labels, divided by the number of iterations. Looking at the logic of the above equation, if your model is not very predictive of outcome, either during classification analysis or function approximation (regression) the MSE will be high. If your model is junk and the observed $MSE_{obs}$ is very high, like 0.4, then the $MSE^{(b)}$ from prediction when class labels are permuted could be lower during many iterations. If $MSE^{(b)}$ is lower than $MSE_{obs}$ 400 times out of 1000 iterations, then the p-value is 0.4. However, if your model is very predictive, and the number of times $MSE^{(b)}$ is less than $MSE_{obs}$ is 1/1000, then the p-value is 0.001, which is highly significant.

A main problem with empirical p-value testing, however, is that it is sometimes difficult to enforce the null hypothesis for whatever you are doing. But empirical p-value testing is used exclusively in statistical genetics a lot. The only modification done if many genes are involved is that the p-value criterion ($\alpha=0.05$) for significance undergoes a Bonferroni adjustment for the multiple testing problem to become

$P^* = \frac{0.05}{\#tests}$

where $\# tests$ is equal to the number of genes evaluated. A bar graph is then made plotting $-\log(P)$ for each gene with a reference line for significance called "Bonf" which is equal to $-\log(P^*)$. [there are other adjustments for multiple testing such as Benjamini-Hochberg, Storey q-values, Westfall-Young, etc., so it's a matter of stringency for determining the false discovery rate ($FDR$)].

In summary, there are ways to easily navigate around the problems you raise about lack of statistical testing in ML, since empirical p-value testing allows you turn anything into a hypothesis test for significance.

$\endgroup$
1
  • $\begingroup$ Thanks for the thoughtful reply! I may just have to try this! $\endgroup$
    – Ryan
    Apr 12 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.