Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm working on a promotional response analysis. I have a really small real world dataset with 25 observations and 15 variables. The variables have a high degree of multicollinearity and some have outliers. Also, I cannot use machine learning methods because I need interpretable coefficients.

Here is what I have tried so far:

  1. I used GLM, but all variables seemed insignificant. I know for a fact that the dependent variable is responsive to many independent variables.
  2. I used robust regression, but basically the program fails because the software showed an error that stated, "The number of observations must be at least twice the number of coefficients."

I would really appreciate if you could suggest some methods/techniques or strategies to solve this problem. Thanks and regards.

share|improve this question
It sounds like the program is providing solid advice :-). Have you established--by exploring the data--whether there is any chance at all these data could yield reliable results of any kind? They would need to exhibit very strong, consistent relationships indeed. – whuber Feb 25 '14 at 23:23
Have a look at this question that might be helpful for the multicollinearity problem. – Stat Feb 26 '14 at 7:33
up vote 8 down vote accepted

My advice is "don't try to do this".

25 observations with 15 variables is very overfit, even if it satisfies all the assumptions of linear regression. Collinearity will mess up your standard errors and make the output highly sensitive to small changes in input. Outliers may well be influential points (although they may not be).

If you want, you can run several simple regressions, with just one IV in each.

But if you need to run some model with all these IVs, you need a lot more data. Perhaps 10 times as much, maybe more. Then you might try regression trees. But don't use it on such a small data set, as anything it yields will also be overfit (although many tree methods will simply return no model with this sort of data).

share|improve this answer
thank you very much for your response. I do have an option to run separate regressions. I was actually trying to replicate what a consultant's "black box" did for us, and I was unable to proceed any further. I like you blog… I try to adopt your suggestion as much as I can. Thanks – forecaster Feb 26 '14 at 1:25

To begin in terms of an analytical framework...I would say you have 15 independent variables to choose from. You don't have 15 independent variables you have to include in your model. Given that, I have a couple of ideas. Hopefully one of them will be helpful. Before trying any of the following ideas, I would scrutinize the outliers. I would not hesitate to throw out a few of those (or use a dummy variable for the respective time period).

The first idea is to try what I would call manual stepwise regression. You first do a simple linear regression with the one independent variable that has the highest absolute correlation with your dependent variable. Next, you calculate the residual of this regression. And you look for the independent variable among the remaining ones that has the highest correlation with the residual of your regression. Next, you rerun your regression with those two independent variables. You can add a 3rd or 4th variable, repeating this process until you can tell that none of the remaining variables are correlated enough to the residual of your last regression. Typically, after selecting 3 or 4 independent variables this way, you are done. The model typically breaks apart when you add any more than that. This process is typically robust, and usually does not result in "overfit" models because it selects few variables.

My second idea is to try principal component analysis (PCA) which is suited to dealing with multicollinear variables. For it to work, you may have to reduce your number of variables anyway. After doing the stepwise regression exploration, it will become clear what variables are superfluous. The challenge of PCA is that it is rather difficult. Unless you have the appropriate statistical software, it is rather inaccessible. It also often creates a bit of a black box. The principal components are essentially indices of independent variable combinations. Sometimes those combinations of variables may have an explanatory narrative (the S&P 500 is a pretty good index capturing the performance of 500 different stocks). Unfortunately, most principal components do not have as clear an interpretation as the S&P 500.

In any case, I hope those ideas help.

share|improve this answer
Thank you very much. I will try your approach and report back if it yields goods results. – forecaster Feb 26 '14 at 1:29
My pleasure. I think in this situation I would try the manual Stepwise regression first. It has many benefits. It is much simpler. You can just use Excel. It is very transparent and easy to explain to Management. You extract most of the information within your body of variables by selecting the best combination of 3 or 4 variables. PCA, if you do give it a try can turn often into an overfit black box. – Sympa Feb 26 '14 at 1:35
Making sense of principal component analysis, eigenvectors & eigenvalues may help a bit with the black box feeling when analyzing principal components. R is a freely available software environment and data analytic program that can run PCA all sorts of ways, but there's a bit of a learning curve if your experiential background is a point-and-click GUI, e.g., SPSS. – Nick Stauner Feb 26 '14 at 1:41
-1, the stepwise idea is very poor advice. The fact that you end up selecting only a few variables is unrelated to whether it will yield an overfit model. This strategy is very likely to overfit. The PCA strategy is a good idea, +1. – gung Feb 26 '14 at 3:25
gung, based on extensive firsthand experience backing my own answer, I disagree with each specific sentences within your own comment. On, PCA with the caveat I mentioned, we are not so far apart. Otherwise, we'll just have to agree to disagree on your other positions. – Sympa Feb 26 '14 at 16:46

if this is a survey analysis, then you might want to try something called "structural equation modeling". it's an entire field in qualitative analysis, but you can get it working quickly with software like Stata.

the fact that you say your data is collinear is exactly the problem with which SEM deals with.

share|improve this answer
W/ 15 variables & only 25 data, there are only going to be very few possible models that would be identifiable. SEM isn't really going to be a solution here. – gung Feb 26 '14 at 3:03
SEM on 25 observations of 15 variables, huh...I can hardly imagine fitting a model like that. I really wouldn't call SEM a qualitative analysis, nor would I say it's meant to handle multicollinearity by itself, especially in small samples (see Grewal, Cote, & Baumgartner, 2004). SEM is too often fit to excessively small samples, producing bad estimates (Westland, 2010). – Nick Stauner Feb 26 '14 at 3:04
with 25 observations it's barely possible to have more than one bona fide parameter in the regression. with SEM one could run a factor analysis and maybe come up factors, then regress on the factors. each of his 15 variables might be capturing some information about the behavior, and high collinearity points to this. it would be better instead of throwing variable out with stepwise regression, to try somehow use all the data, especially since the sample is so small. – Aksakal Feb 26 '14 at 15:03

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.