11

No, you cannot safely assume that. The reason is that conditional independence does not imply independence and vice versa (wiki). Moreover the forward selection style approach you follow suffers from a fundamental problem: model selection criteria like that usually rely on p-values/t-statistics/... To be based on the "correct" underlying model. This ...


8

You seem to be assuming that the models work in additive fashion, so adding a feature to the model just "adds" some stuff related to this feature alone and does not influence the rest of the model, same with removing the feature. That is not the case. If machine learning models worked like this, then to build a model with $k$ features you would need only to ...


2

Perhaps these will be helpful: Hyndman & Kostenko "Forecasting without significance tests?" (2008) Hyndman "Why I don't like statistical tests".


2

There are quite some similarities between Aris Spanos' framework and David Hendry's econometric methodology; no wonder as Spanos was a student of Hendry. Here is my brief summary of what Hendry had to say when confronted by Edward Leamer and Dale Poirier on the problem of pretesting and post-selection inference (Hendry et al., 1990). Summary Hendry does not ...


2

You should try to include any predictor associated with outcome in a logistic regression, but you should consider a different approach to selecting predictors. Selecting predictors for multiple regression models based on their individual associations with outcome is not a very good idea in general. Even in standard linear regression, if you omit any ...


1

I think the easiest way to do this with mgcv would be to not force linearity for the parametric effects, but to admit potential for a small amount of non-linearity and then you can use the double penalty on all terms with select = TRUE. gam(count ~ s(X,Y) + s(A, k = 3) + s(B, k = 3) + s(C, k = 3) + s(D, k = 3) + s(E, k = 3) + s(F, k = 3) + s(G, ...


1

You have to find out what they mean by "held-out". Usually, the "held-out" set is a validation set, a separate dataset that is used to estimate the test set error. For example, it is useful to estimate the test set error to do model selection on different hyperparameters and/or early stopping*. That is very common because ML algorithms ...


1

How is it possible to 'estimate' [...] the parameters when no. of parameters (P) > no. of data points (K). [...] What your optimizer does it iteratively "improves" the parameters in such way that minimizes the loss. You usually start with randomly initialized parameters, then the optimizer makes a suggestion on how they could be changed so that ...


1

As was already pointed out by the answer of cebeleites, inner and outer CV loop have different purposes: the inner CV loop is used to get the best model, the outer CV loop can serve different purposes. It can help you to estimate in a more unbiased way the generalisation error of your top performing model. Additionally it gives you insights into the "...


Only top voted, non community-wiki answers of a minimum length are eligible