Regression for really small data with high degree of multicollinearity and outliers I'm working on a promotional response analysis. I have a really small real world dataset with 25 observations and 15 variables. The variables have a high degree of multicollinearity and some have outliers. Also, I cannot use machine learning methods because I need interpretable coefficients.
Here is what I have tried so far: 


*

*I used GLM, but all variables seemed insignificant. I know for a fact that the dependent variable is responsive to many independent variables.

*I used robust regression, but basically the program fails because the software showed an error that stated, "The number of observations must be at least twice the number of coefficients."


I would really appreciate if you could suggest some methods/techniques or strategies to solve this problem. Thanks and regards.
 A: My advice is "don't try to do this". 
25 observations with 15 variables is very overfit, even if it satisfies all the assumptions of linear regression.  Collinearity will mess up your standard errors and make the output highly sensitive to small changes in input. Outliers may well be influential points (although they may not be).
If you want, you can run several simple regressions, with just one IV in each. 
But if you need to run some model with all these IVs, you need a lot more data. Perhaps 10 times as much, maybe more.  Then you might try regression trees. But don't use it on such a small data set, as anything it yields will also be overfit (although many tree methods will simply return no model with this sort of data). 
A: To begin in terms of an analytical framework...I would say you have 15 independent variables to choose from. You don't have 15 independent variables you have to include in your model. Given that, I have a couple of ideas. Hopefully one of them will be helpful.  Before trying any of the following ideas, I would scrutinize the outliers. I would not hesitate to throw out a few of those (or use a dummy variable for the respective time period). 
The first idea is to try what I would call manual stepwise regression.  You first do a simple linear regression with the one independent variable that has the highest absolute correlation with your dependent variable. Next, you calculate the residual of this regression. And you look for the independent variable among the remaining ones that has the highest correlation with the residual of your regression.  Next, you rerun your regression with those two independent variables. You can add a 3rd or 4th variable, repeating this process until you can tell that none of the remaining variables are correlated enough to the residual of your last regression. Typically, after selecting 3 or 4 independent variables this way, you are done.  The model typically breaks apart when you add any more than that. This process is typically robust, and usually does not result in "overfit" models because it selects few variables.  
My second idea is to try principal component analysis (PCA) which is suited to dealing with multicollinear variables.  For it to work, you may have to reduce your number of variables anyway.  After doing the stepwise regression exploration, it will become clear what variables are superfluous.  The challenge of PCA is that it is rather difficult.  Unless you have the appropriate statistical software, it is rather inaccessible.  It also often creates a bit of a black box.  The principal components are essentially indices of independent variable combinations.  Sometimes those combinations of variables may have an explanatory narrative (the S&P 500 is a pretty good index capturing the performance of 500 different stocks).  Unfortunately, most principal components do not have as clear an interpretation as the S&P 500.
In any case, I hope those ideas help.
A: if this is a survey analysis, then you might want to try something called "structural equation modeling". it's an entire field in qualitative analysis, but you can get it working quickly with software like Stata.
the fact that you say your data is collinear is exactly the problem with which SEM deals with. 
