2
$\begingroup$

I'm unclear on if its important to perform cross-validation when determining if a dependent variable has a significant effect on my independent variable in multilinear regression.

Specifically, I'm fitting some linear models to determine if my independent variable predicts my dependent variable, controlling for age and sex of my participants. I run this model using stats models in Python like so:

mod = smf.ols(formula='y ~ x + Age + Sex', data =
subject_data) 
res = mod.fit() 
print(res.summary())

The model summary provides F stats, t stats, etc. Mostly, I'm interested in t-test for my x variable to determine if it significantly predicts my y variable.

However, it seems statsmodels doesn't perform cross-validation. I'm aware on how to perform cross-validation using other python packages (like sklearn), however, these packages don't provide a framework for the t-test to determine if my predictor is significant. So my specific questions are:

  1. Is cross-validation necessary when determining the significance of a predictor variable in multilinear regression?
  2. How do I perform cross-validation using statsmodel framework?
  3. How do I determine the significance of my predictor variable using the sklearn framework?

Any help would be greatly appreciated,

$\endgroup$
0

1 Answer 1

3
$\begingroup$

When approaching this question, I think it's helpful to clarify that the word significant is often a loaded term. Significant can mean:

  1. Statistically significant that comes with p-values and confidence intervals.
  2. Significant meaning the variable x is able to predict your response y well

These have different implications and answer different questions, so it is important to understand what question you are trying to answer. I think you are interested in the second meaning, but I will explain both below.

I mention this because 1) and 2) are not the same. A variable can be very statistically significant (small p-value), but not be very predictive of the outcome y if the model coefficient is very small.

If you are interested in assessing the statistical significance of x's relationship with y (this tries to answer the question does x have an association of any strength with y), you should not use cross validation (CV). Statistical significance is partially a function of sample size, and using CV will mean that you have a smaller sample size due to data splitting, making it more difficult to detect a significant association if x is truly associated with y. Takeaway: If you are trying to calculate p-values/confidence intervals, CV is not necessary and shouldn't be used.

If you are interested in assessing whether x is predictive of y, statistical significance and p-values are probably not the best measure of whether x is a good variable to use. Although sometimes people use a p-value cutoff to choose which variables are strongly predictive (say keep all variables with $p < .05$), this is flawed due to the uneven relationship between p-values and prediction accuracy mentioned above. To gauge whether x is a good predictor of y, there are a few different options:

  1. Use CV to assess the performance of a model with x and another without x. If the model with x performs better, x probably is predictive of y.
  2. Some people use feature selection methods to find the variables that together best predict the response y. You could use a feature selection method and see if the algorithm retains x as a predictive feature. These approaches usually use CV to choose their predictive features. Lasso is one such approach and is already implemented in sklearn
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.