2
$\begingroup$

I have data about 40 stores described by 50+ continuous variables in terms of customer behaviour (types of purchases, demographic attributes, etc). I want to build a simple regression model to explain the profitability of stores based on what we know about the rest. So I:

  1. Removed variables with many missing values
  2. Removed co-variates above a certain correlation threshold (for example rho > .7)

I still have a lot of variables (18) that might have a decent explanatory power. I read that stepwise regression is a terrible idea, but I'm not sure what better method I could apply in this context, considering the very small number of observations that makes cross-validation approaches hard.

Any sound alternative to both linear regression and stepwise selection methods?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Using theory to guide your model building and reporting (including negative findings) is the sound alternative you are looking for because stepwise model building is indeed a terrible idea. $\endgroup$
    – Alexis
    Commented Nov 7, 2022 at 17:12
  • $\begingroup$ "Removed co-variates above a certain correlation threshold (for example rho > .7)" Correlation with what? Why? $\endgroup$
    – Christian Hennig
    Commented Nov 7, 2022 at 17:12
  • $\begingroup$ In a situation like this, I think that forward selection isn't that terrible, maybe using the AIC rather than significance testing. I'm not saying it's the best you can do (Lasso may well be better), but if I were you I'd well may run it at least for curiosity reasons. And with $n=40$ you actually can do leave-one-out CV to compare models (like those resulting from different dimension reduction approaches). $\endgroup$
    – Christian Hennig
    Commented Nov 7, 2022 at 17:18
  • $\begingroup$ if you want a fancy alternative to stepwise regression, you could try folded concave penalized regression, like that implemented in Spike-Slab Lasso's R package: cran.r-project.org/web/packages/SSLASSO/index.html $\endgroup$
    – John Madden
    Commented Nov 7, 2022 at 17:23
  • $\begingroup$ What's the goal of this analysis, ie, what do you mean by "explain profitability"? (And you could also use regularization depending on why you are building a model.) $\endgroup$
    – dipetkov
    Commented Nov 7, 2022 at 19:10

1 Answer 1

Reset to default
1
$\begingroup$

A thing to think about is whether you can use subject matter knowledge to define a small number of meaningful indexes summarising your original variables, and then build a model on them.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Something like PCA but simply combining the variables? Interesting suggestion $\endgroup$
    – Strabonio
    Commented Nov 7, 2022 at 20:02
  • 1
    $\begingroup$ @Strabonio In fact my proposal can be seen as "something like PCA", but PCA does such a thing in an automatic data driven manner, which will give you something harder to make sense of in an application like this. Also PCA does not take into account the $y$ and so PCs may not work well for predicting $y$, and if you use subject matter knowledge for defining indexes, you may well arrive at something better (although this depends on the knowledge you have and cannot be taken for granted). $\endgroup$
    – Christian Hennig
    Commented Nov 8, 2022 at 9:53
  • 1
    $\begingroup$ @Strabonio Note that not taking into account the $y$ when doing dimension reduction has the advantage that the regression later carried out on the resulting variables will not suffer from model selection bias. $\endgroup$
    – Christian Hennig
    Commented Nov 8, 2022 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.