1
$\begingroup$

I am working with a dataset that has a big amount of observations (2000). The purpose of my work is to find which dependent variables (x1, x2, x3...) are linked to my independent variable (y). I have discovered the adjusted R2 method and it seems pretty accurate for what I am looking for since it makes a ponderation of the influence of new variables in the prediction.

The equation for the Adjusted R2 method is shown in 1:

R2_Adj = 1 - [(1-R2)(n-1)/n-k-1] [1]

where:

  • N is the number of points in your data sample.
  • K is the number of independent regressor.

Apparently it should work. However, when I have a big amount of data: N = 2000 the difference between 1 or 2 regressors is sometimes not big enough to spot a good regressor and instead, sometimes the model accepts a 'bad' regressor as 'good'.

Do you know how can I fix this? Does it exist any other method similar to this one that allows to use this big amount of Data samples?

$\endgroup$
5
  • $\begingroup$ how do you know it is not accepting good regressors and accepting bad regressors? $\endgroup$
    – rep_ho
    Commented Nov 2, 2021 at 12:56
  • $\begingroup$ I created my own dataset with the independent variable and 'good' and 'bad' regressors for it and this was the result $\endgroup$ Commented Nov 2, 2021 at 13:00
  • $\begingroup$ just by chance this will sometimes happen. Or did you find a systematic problem? $\endgroup$
    – rep_ho
    Commented Nov 2, 2021 at 13:02
  • $\begingroup$ In my model It happens whenever I add a new regressor to the proper one. The adjusted regression value is higher for the 2 variables model (which contains a bug) than for the one variable model (which only contains the proper variable). But I can try to create a new dataset and check, thank you! $\endgroup$ Commented Nov 2, 2021 at 13:10
  • $\begingroup$ I don't know much about adjusted R2, but if your goal is to find variables related to the outcome, than you can use p-values or other similar measures, no? Also, a question is if you want to put variables into one model or test them separately, which has different interpretations $\endgroup$
    – rep_ho
    Commented Nov 2, 2021 at 13:16

1 Answer 1

0
$\begingroup$

This depends on what you care about.

For forecast accuracy just use cross validation with the mse of test sets to select features. For inference then I would let any theory you have for the domain get you started and then what you do from there is, quite frankly, up for debate but a full bayesian treatment injecting as much domain knowledge into your priors would be my recommendation.

Variables that are 'linked' begs the question of correlation vs causation. I will just assume correlation.

For simplicity, you could just toss adjusted r squared and minimize the AIC or AICc or BIC. They have some nice qualities that r-squared does not. Minimizing AIC results in a model that approximates the best model from cross validation and BIC approximates 'True' model process if it exists as a subset of your variables (this is almost never the case). All have been shown to outperform R squared and Adjusted R squared on average for model selection in simulated studies from awhile back.

Alternatively, you could use a LASSO regression to do variable selection and regularization. If you use scikitlearn you can just use the LassoCV class and it will handle everything for you. Variable coefficients will be 'shrunk' towards 0 and many may be 0 and therefore dropped from your equation. What's left would be variables that were more 'useful'.

And finally, you could just feed it to a random forest and use the feature importance to grab the most 'relevant'. Definitely a looked down upon approach from a statistics standpoint but it has some nice-ities. Primarily it will handle feature interactions better than standard regressions so it is pretty useful as plug-and-play approach.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.