Which of these regression analysis to use? I come primarily from a machine learning background, which I believe is what you'd say as a predictive form of analysis. However, I am now trying to publish some results in a paper about how different variables influence an outcome variable. For this, I believe inferential statistics is the way to go.
My problem setting
I have 107 rows of data with ~15 integer-valued continuous features (counts data), and a floating-point continuous column that I need to predict. I am trying to prove that these features are predictive of the continuous column (I'm not sure if using the word "predictive" is apt here because my objective is to persuade people that there is an underlying, real-world phenomenon that relates these features to the outcome, so like I mentioned, it's probably more inferential).
My doubt
I am using SAS JMP to do my analyses. When I do a "Fit Model" on my data, I am presented with a bunch of different options for modelling (which JMP terms as "Personalities"):

*

*Standard Least Squares

*Stepwise

*Generalized Regression

*Mixed model

*Loglinear variance

*Manova

*Generalized Linear Model

I have a rough idea of what 1, 2, 3, and 7 are about (although I don't know the difference between 3 and 7).
Now, when I choose a specific model, say, Generalized Regression, I am further presented with options for "Estimation Method":

*

*Lasso

*Best Subset

*Backward elimination

*Forward elimination

*Standard least squares

*... and a few more

Then, there is also the choices for "Validation method":

*

*AICc

*BIC

*KFold

*Leave-One-Out

The reason I am overwhelmed by the number of choices is not because of the sheer number of combinations possible, but also because of the following observation:
When I run Generalized Regression + Standard Least Squares + AICc validation, I find that none of the features have a p-value < 0.05 (or even close). However, the moment I switch to Generalized Regression + Best subset + AICc validation, I suddenly have 3 features whose p-values are < 0.05. This makes me confused as I would have expected a best subset to be not that different from a standard least squares (at least for a small problem having 15 variables and 107 rows).
While I got excited by the low p-value from the Best Subset method, I want to make sure that I am using a standard method of analysis and not blindly going with the one that gives me the lowest p-values.
Am I in the right direction? What would be the "standard, go-to" method of analysis in a situation like mine? I'd be happy to provide more details about my problem/results if that would help. Thank you!
 A: Starting at the end. AIC and BIC are heuristics that are applied to prevent overfitting. These heuristics are often used if testing for the effect of adding a feature to a model is not easy. In the case of linear regression one can test the effect of adding a feature using an F-tests. In case of forward regression an F-test is applied. The progressive models are called hierarchical because the larger models contain the smaller parent model, and an extra predictor. KFold and Leave-One-Out are the machine learning equivalents of these tests in a way.
Forward selection uses a gready strategy, as in just add the next best predictor. Best subset does an exhaustive search over all combinations. These methods can render different results. If the predictors all have predictive power and are not correlated, then one would expect a similar result for both methods.
Where it gets tricky is if the predictors are correlated; i.e. there is multi-collinearity. One of the effects of correlated predictors is an inflation of the confidence interval of the predictors. Adding all predictors might inflate the confidence intervals so much that none of the predictors seem significant. This mechanism could explain your observations. Make a correlation matrix to corroborate this hypothesis.
You could test the individual predictors as a post-hoc analysis in a small regression omdel, showing that each predictor has some power. But that the configuration of the best subset has the best predictive power. Note that if one performs lots of tests, then some Bonferroni correction is usually warranted. In case of post-hoc analysis this is less relevant, the post-hoc analysis is more of a side argument.
Note that if any predictor is added by either forward or best subset, that means that there is and predictive effect that can not be attributed to coincidence at the alpha level.
Also note that there is a difference between a significant effect and a relevant effect. This issue is more relevant for larger datasets; small effects can result in statistically significant results, without actually predicting very much. So, are the predictions good enough? Or do you just want to prove some predictive power?
