5
$\begingroup$

I'm working on a promotional response analysis. I have a really small real world dataset with 25 observations and 15 variables. The variables have a high degree of multicollinearity and some have outliers. Also, I cannot use machine learning methods because I need interpretable coefficients.

Here is what I have tried so far:

  1. I used GLM, but all variables seemed insignificant. I know for a fact that the dependent variable is responsive to many independent variables.
  2. I used robust regression, but basically the program fails because the software showed an error that stated, "The number of observations must be at least twice the number of coefficients."

I would really appreciate if you could suggest some methods/techniques or strategies to solve this problem. Thanks and regards.

Improve this question
$\endgroup$
2
  • 4
    $\begingroup$ It sounds like the program is providing solid advice :-). Have you established--by exploring the data--whether there is any chance at all these data could yield reliable results of any kind? They would need to exhibit very strong, consistent relationships indeed. $\endgroup$
    – whuber ♦
    Feb 25, 2014 at 23:23
  • $\begingroup$ Have a look at this question that might be helpful for the multicollinearity problem. $\endgroup$
    – Stat
    Feb 26, 2014 at 7:33

3 Answers 3

Reset to default
9
$\begingroup$

My advice is "don't try to do this".

25 observations with 15 variables is very overfit, even if it satisfies all the assumptions of linear regression. Collinearity will mess up your standard errors and make the output highly sensitive to small changes in input. Outliers may well be influential points (although they may not be).

If you want, you can run several simple regressions, with just one IV in each.

But if you need to run some model with all these IVs, you need a lot more data. Perhaps 10 times as much, maybe more. Then you might try regression trees. But don't use it on such a small data set, as anything it yields will also be overfit (although many tree methods will simply return no model with this sort of data).

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ thank you very much for your response. I do have an option to run separate regressions. I was actually trying to replicate what a consultant's "black box" did for us, and I was unable to proceed any further. I like you blog statisticalanalysisconsulting.com/… I try to adopt your suggestion as much as I can. Thanks $\endgroup$
    – forecaster
    Feb 26, 2014 at 1:25
4
$\begingroup$

To begin in terms of an analytical framework...I would say you have 15 independent variables to choose from. You don't have 15 independent variables you have to include in your model. Given that, I have a couple of ideas. Hopefully one of them will be helpful. Before trying any of the following ideas, I would scrutinize the outliers. I would not hesitate to throw out a few of those (or use a dummy variable for the respective time period).

The first idea is to try what I would call manual stepwise regression. You first do a simple linear regression with the one independent variable that has the highest absolute correlation with your dependent variable. Next, you calculate the residual of this regression. And you look for the independent variable among the remaining ones that has the highest correlation with the residual of your regression. Next, you rerun your regression with those two independent variables. You can add a 3rd or 4th variable, repeating this process until you can tell that none of the remaining variables are correlated enough to the residual of your last regression. Typically, after selecting 3 or 4 independent variables this way, you are done. The model typically breaks apart when you add any more than that. This process is typically robust, and usually does not result in "overfit" models because it selects few variables.

My second idea is to try principal component analysis (PCA) which is suited to dealing with multicollinear variables. For it to work, you may have to reduce your number of variables anyway. After doing the stepwise regression exploration, it will become clear what variables are superfluous. The challenge of PCA is that it is rather difficult. Unless you have the appropriate statistical software, it is rather inaccessible. It also often creates a bit of a black box. The principal components are essentially indices of independent variable combinations. Sometimes those combinations of variables may have an explanatory narrative (the S&P 500 is a pretty good index capturing the performance of 500 different stocks). Unfortunately, most principal components do not have as clear an interpretation as the S&P 500.

In any case, I hope those ideas help.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you very much. I will try your approach and report back if it yields goods results. $\endgroup$
    – forecaster
    Feb 26, 2014 at 1:29
  • $\begingroup$ My pleasure. I think in this situation I would try the manual Stepwise regression first. It has many benefits. It is much simpler. You can just use Excel. It is very transparent and easy to explain to Management. You extract most of the information within your body of variables by selecting the best combination of 3 or 4 variables. PCA, if you do give it a try can turn often into an overfit black box. $\endgroup$
    – Sympa
    Feb 26, 2014 at 1:35
  • $\begingroup$ Making sense of principal component analysis, eigenvectors & eigenvalues may help a bit with the black box feeling when analyzing principal components. R is a freely available software environment and data analytic program that can run PCA all sorts of ways, but there's a bit of a learning curve if your experiential background is a point-and-click GUI, e.g., SPSS. $\endgroup$
    – Nick Stauner
    Feb 26, 2014 at 1:41
  • $\begingroup$ -1, the stepwise idea is very poor advice. The fact that you end up selecting only a few variables is unrelated to whether it will yield an overfit model. This strategy is very likely to overfit. The PCA strategy is a good idea, +1. $\endgroup$
    – gung - Reinstate Monica
    Feb 26, 2014 at 3:25
  • $\begingroup$ gung, based on extensive firsthand experience backing my own answer, I disagree with each specific sentences within your own comment. On, PCA with the caveat I mentioned, we are not so far apart. Otherwise, we'll just have to agree to disagree on your other positions. $\endgroup$
    – Sympa
    Feb 26, 2014 at 16:46
0
$\begingroup$

if this is a survey analysis, then you might want to try something called "structural equation modeling". it's an entire field in qualitative analysis, but you can get it working quickly with software like Stata.

the fact that you say your data is collinear is exactly the problem with which SEM deals with.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ W/ 15 variables & only 25 data, there are only going to be very few possible models that would be identifiable. SEM isn't really going to be a solution here. $\endgroup$
    – gung - Reinstate Monica
    Feb 26, 2014 at 3:03
  • 3
    $\begingroup$ SEM on 25 observations of 15 variables, huh...I can hardly imagine fitting a model like that. I really wouldn't call SEM a qualitative analysis, nor would I say it's meant to handle multicollinearity by itself, especially in small samples (see Grewal, Cote, & Baumgartner, 2004). SEM is too often fit to excessively small samples, producing bad estimates (Westland, 2010). $\endgroup$
    – Nick Stauner
    Feb 26, 2014 at 3:04
  • $\begingroup$ with 25 observations it's barely possible to have more than one bona fide parameter in the regression. with SEM one could run a factor analysis and maybe come up factors, then regress on the factors. each of his 15 variables might be capturing some information about the behavior, and high collinearity points to this. it would be better instead of throwing variable out with stepwise regression, to try somehow use all the data, especially since the sample is so small. $\endgroup$
    – Aksakal
    Feb 26, 2014 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.