Why do parameters go untested in Machine Learning?

Question

I finished up a machine learning (ML) course a while back. Everything was as an optimization problem. No matter what predictive challenge you face in ML, you're generally minimizing some objective (i.e., cost) function. In so doing, you come up with "optimal" parameters that satisfy the equation you're working with (e.g., gradient descent and linear regression where MSE is the objective function you minimize).

Is it the case with all machine learning models that, when you find the optimal parameters that minimize your objective function, you are also, almost by definition, finding the same statistically significant coefficients you would otherwise discover if you were to attack the problem from a stats perspective where the focus is on tests of statistical significance? Let's define "stats perspective" as running a model in R and adding or deleting new variables based on their statistical significance or degree to which they change AIC .

Answer 1

Is it the case with all machine learning models that, when you find the optimal parameters that minimize your objective function, you are also, almost by definition, finding the same statistically significant coefficients you would otherwise discover if you were to attack the problem from a stats perspective where the focus is on tests of statistical significance?

In some cases, yes. If you're using logistic regression as a classifier, then optimizing the cost is the same as optimizing the log likelihood. Not every model is like this. Deep neural networks do not map nicely onto a statistical counterpart (though there is some active research on their statistical properties I imagine).

As to variable selection via AIC or similar, that would be a form of feature selection. I think that is something to ask about on its own.

As to your titular question, inference is not our main concern; its predictive capability. Besides, most ML problems (most, not all) work with data sets so large that significance becomes a straw man. The sheer size of the data would allow for high precision estimates, and since no effect is truly 0 you would find that all effects are significant.

Answer 2

tl;dr: Because it's 1) not possible and 2) not necessary.

Long answer: Your question is posed from a statistical perspective, where testing the parameters is a standard thing to do. This makes it sound as if the way Machine Learning does it is in need of justification. But we can equally well pose the question in the opposite direction: "Why are parameters being tested in statistics?"

The answer differs slightly, depending whether you're into inferential or predictive modelling. If you are doing inference, you want to know (understand!) which parameters are relevant to the output. If you're doing prediction, you want to avoid overfitting, and eliminating superfluous parameters helps with that. However, tests make assumptions regarding the probability distribution behind the parameters, and these assumptions may be more or less hard to justify. AIC (and BIC, too) also make assumptions. Your test and parameter elimination strategies work only as good as your assumptions are satisfied.

Machine Learning is not an inferential toolbox. It was, from the beginning, meant to generate automated systems ("machines") which can make decisions ("predictions") without human intervention. Interpretability was never an issue, and this allows the models to be of arbitrary complexity (up to, but not necessarily, billions of parameters, as mentioned in a comment). So, the information which parameters are relevant ("significant") would be only relevant to ensure generalisation. However, machine learning makes much weaker assumptions about the data and, as a rule, we know (almost) nothing about the underlying probability distributions. This makes reasonable statistical tests next to impossible. Instead, generalisation in machine learning is ensured by cross validation and testing on a separate (test) data set. But, in order for that to work half-way reliably, you need much more data then for statistical modelling.

Answer 3

There are some different mindsets between classical statistics and machine learning, where modern machine learning can have huge amount of data.

And as you said, the whole problem is an "optimization problem", where people put little or no assumptions to the model. And the golden standard is the performance on a large testing set. If we go back and review all the nice properties (confidence interval, p value, etc.) they are coming from strong assumptions about the data and model.

With less data, people are willing to do more work in math (say Gauss–Markov theorem for OLS) to make the results nice and solid. But modern machine learning people are more practical and do not care too much about the assumption and statistically significant, but focusing on the model will "work" on large testing data.

Note deep neural network can have billions of parameters and the number of parameters can even much bigger than number of data points. (which may not really acceptable from classical statistics perspective) So, AIC and BIC also not working there.

Answer 4

Let's define "stats perspective" as running a model in R and adding or deleting new variables based on their statistical significance or degree to which they change AIC.

First of all, "adding or removing variables based on their significance" is not a good practice. As the name suggests, significance testing is about testing a hypothesis, it is not a tool for optimizing anything, and by using it for the variable selection you assume some kind of optimization problem.

Second, ask yourself what exactly does a "statistically significant parameter" means. We can do many different statistical tests, but the most common ones, the out-of-the-box statical tests for regression models test the hypothesis that the parameter differs from zero. If a parameter is equal to zero in the regression model, it has no effect on the results. Let's start with the fact that not every machine learning model has parameters (e.g. $k$-NN, or decision tree don't). But even if it has parameters, why would you care if they are zero or not? If the model makes correct predictions, it can have as many zero parameters as you want, worst case they would make it computationally slower than in the case of a lighter model. More than this, sometimes we intentionally drag the parameters towards zero, like with $L_1$ regularization, as means of feature selection. In such a case we want as many of the parameters to be zeros, as long as they do not worsen the results.

Answer 5

Your descriptions of Statistical Model Building and Machine Learning are imcomplete if not false. Adding and deleting data based on their AIC ist stepwise regression and that is widely discouraged because it leads to overfitting to the data at hand. So that is not what we should do in statistics.

In machine learning you left out the important aspect of splitting you data into training, tuning and test data. Let's take LASSO regression as an example. It usually does not lead to $p$-values but it has a tuning paramater $\lambda$ that decides, how easily the model accepts or rejects additional data. This tuning parameter is usually defined via cross validation so that the model should have no more or no less parameters than are optimal for prediction purposes.

A statistics models gets it's justification from some complicated-to-explain $p$-value that should lie below some completely arbitrarily chosen limit. A machine learning model gets it's justification from cross validation. So machine learning does not have something less, just something different that does not appear in your depiction of ML.

Answer 6

Is it the case with all machine learning models that, when you find the optimal parameters that minimize your objective function, you are also, almost by definition, finding the same statistically significant coefficients you would otherwise discover if you were to attack the problem from a stats perspective where the focus is on tests of statistical significance?

No. Assuming the general aim in machine learning would be to minimize prediction error on future observations. In that, we often accept a little bias in our estimator, as long as its variance is reduced more strongly (through e.g., smoothing, penalization, ensembling).

In statistical significance testing (inference), at least within the frequentist approach, the aim is to obtain unbiased parameter estimates (i.e., on average, we obtain the 'true' parameter values).

Unbiasedness tends to come at the cost of (much) increased variance. Though not always: An OLS linear regression likely provides very good power and predictive accuracy if assumptions are met, the number of predictors is not too large and sample size not too small. Then the aims of prediction and inference coincide to yield the same optimal solution.

Both viewpoints are statistical. There is no real distinction between statistics and machine learning. One should never do ML without taking the sampling aspects of the problem into account (sample size and representativeness, number of possible predictors, distribution of predictors and response, etc.).

What you refer to as a 'stats perspective' is statistically a (very) unsound approach; it is misusing a confirmatory hypothesis-testing approach in an exploratory manner.

Answer 7

Many ML models are not based on a probability distribution with which you can determine statistical significance, and many are not set up to determine statistical significance. In such cases these models ordinarily reach some type of convergence criterion (e.g., the MSE delta between iterations goes below a threshold).

However, you can employ empirical hypothesis testing (p-value testing) for any model by (a) first determining the observed $MSE_{obs}$ via a single run from $k$-fold CV, and (b) then run $B=1000$ iterations (1000 runs) for $k$-fold CV but this time with permuted class labels of test objects (in test folds), each time getting $MSE^{(b)}$. Before each of the $B=1000$ iterations using $k$-fold CV with permuted class labels of test objects, it helps to "re-partition" all objects and assign them to different folds. This merely re-orders all objects before assigning to the folds -- and is a way to have different objects in each fold during each iteration.

When done, an "empirical p-value" will be equal to

$P=\frac{\#\{MSE^{(b)}<MSE_{obs} \} }{B}$

where the p-value is equal to the number of times the MSE from test objects with permuted labels is less than the MSE from test objects of the non-permuted labels, divided by the number of iterations. Looking at the logic of the above equation, if your model is not very predictive of outcome, either during classification analysis or function approximation (regression) the MSE will be high. If your model is junk and the observed $MSE_{obs}$ is very high, like 0.4, then the $MSE^{(b)}$ from prediction when class labels are permuted could be lower during many iterations. If $MSE^{(b)}$ is lower than $MSE_{obs}$ 400 times out of 1000 iterations, then the p-value is 0.4. However, if your model is very predictive, and the number of times $MSE^{(b)}$ is less than $MSE_{obs}$ is 1/1000, then the p-value is 0.001, which is highly significant.

A main problem with empirical p-value testing, however, is that it is sometimes difficult to enforce the null hypothesis for whatever you are doing. But empirical p-value testing is used exclusively in statistical genetics a lot. The only modification done if many genes are involved is that the p-value criterion ($\alpha=0.05$) for significance undergoes a Bonferroni adjustment for the multiple testing problem to become

$P^* = \frac{0.05}{\#tests}$

where $\# tests$ is equal to the number of genes evaluated. A bar graph is then made plotting $-\log(P)$ for each gene with a reference line for significance called "Bonf" which is equal to $-\log(P^*)$. [there are other adjustments for multiple testing such as Benjamini-Hochberg, Storey q-values, Westfall-Young, etc., so it's a matter of stringency for determining the false discovery rate ($FDR$)].

In summary, there are ways to easily navigate around the problems you raise about lack of statistical testing in ML, since empirical p-value testing allows you turn anything into a hypothesis test for significance.

Answer 8

Many of the comments/responses by others in the thread give good points. I'll add 1 practical and 1 theoretical.

Practical: Many ML models are concerned more with prediction than with statistical inference (e.g. hypothesis tests of model parameters). See. e.g. Leo Breiman's paper on the two cultures for good elucidation on this distinction.

Theoretical: In many ML models the parameters you would be doing inference on aren't identified nor estimable so doing hypothesis testing wouldn't make sense. There are exceptions to this of course (e.g. using a GLM to predict probability of loan default) and many ML models do not have a clear likelihood,

e.g. what is the likelihood for a splitting rule on the root node of a decision tree?

In these cases you can't do hypothesis testing unless you do some deep thinking which might take longer than you have to solve the problem.