# How to use OLS or NLS on very small sample size?

right now I am facing the problem of handling a very small sample size which consists of 8 observations with 24 variables.

I would like to do first just simple OLS with one dependent and 23 independent variables and use these coefficients to estimate NLS estimates. These estimates will be used for further studies.

The problem obviously is that I got more parameters than observations. I tried to bootstrap my observations and just do simple OLS with a handful of parameter with bootstrap repetitions between 50 and 50,000 to improve my results, however, in fact, the significance of some coefficients even decreased.

Is it even possible to derive somewhat significant coefficients here? What else can I do? I am new here, so I dont know yet how to include a table with exemplary data. Sorry.

Thanks for any help, TheJoez

• The bootstrap will probably not help you. You may want to look at regularisation techniques like the Lasso, Generalized Elastic Nets and so forth. Commented Jan 10, 2013 at 10:16
• Is there a reason why bootstrap does not work here? Commented Jan 10, 2013 at 11:35
• If you bootstrap your eight observations, you will end up with a bootstrapped dataset with again eight observations and 24 variables, and you still have the problem of $p>N$. Of course you can do (some kind of) OLS fitting and assess the variability of estimates with the bootstrap, but this will be determined more by how your software does fitting in this context, perhaps even by the order of your 24 variables. Commented Jan 10, 2013 at 11:50
• Oh sure, $p>N$ remains. Thanks for opening my eyes. ;-) Commented Jan 10, 2013 at 12:55
• Bootstrapping often does not work as expected with small datasets, anyway: it is valid only asymptotically (for larger amounts of data). But more to the point, either model is worthless, because it will produce an absolutely perfect fit and its parameters will not be identifiable (assuming the independent variables don't have hugely reduced rank).
– whuber
Commented Jan 10, 2013 at 13:49

Your problem is not new and a whole chapter ("High-Dimensional Problems") of this book is dedicated to such cases where the number of variables $p$ is much bigger than the number of observations $N$. Numerous ways are possible.