right now I am facing the problem of handling a very small sample size which consists of 8 observations with 24 variables.

I would like to do first just simple OLS with one dependent and 23 independent variables and use these coefficients to estimate NLS estimates. These estimates will be used for further studies.

The problem obviously is that I got more parameters than observations. I tried to bootstrap my observations and just do simple OLS with a handful of parameter with bootstrap repetitions between 50 and 50,000 to improve my results, however, in fact, the significance of some coefficients even decreased.

Is it even possible to derive somewhat significant coefficients here? What else can I do? I am new here, so I dont know yet how to include a table with exemplary data. Sorry.

Thanks for any help, TheJoez

  • 1
    $\begingroup$ The bootstrap will probably not help you. You may want to look at regularisation techniques like the Lasso, Generalized Elastic Nets and so forth. $\endgroup$ – Stephan Kolassa Jan 10 '13 at 10:16
  • 1
    $\begingroup$ Is there a reason why bootstrap does not work here? $\endgroup$ – TheJoez Jan 10 '13 at 11:35
  • $\begingroup$ If you bootstrap your eight observations, you will end up with a bootstrapped dataset with again eight observations and 24 variables, and you still have the problem of $p>N$. Of course you can do (some kind of) OLS fitting and assess the variability of estimates with the bootstrap, but this will be determined more by how your software does fitting in this context, perhaps even by the order of your 24 variables. $\endgroup$ – Stephan Kolassa Jan 10 '13 at 11:50
  • $\begingroup$ Oh sure, $p>N$ remains. Thanks for opening my eyes. ;-) $\endgroup$ – TheJoez Jan 10 '13 at 12:55
  • 1
    $\begingroup$ Bootstrapping often does not work as expected with small datasets, anyway: it is valid only asymptotically (for larger amounts of data). But more to the point, either model is worthless, because it will produce an absolutely perfect fit and its parameters will not be identifiable (assuming the independent variables don't have hugely reduced rank). $\endgroup$ – whuber Jan 10 '13 at 13:49

Your problem is not new and a whole chapter ("High-Dimensional Problems") of this book is dedicated to such cases where the number of variables $p$ is much bigger than the number of observations $N$. Numerous ways are possible.

In my opinion, the two simplest methods to regularize the problem are the Lasso and the Ridge Regression which consist in adding a penalty to the standard least-square term.

  • $\begingroup$ +1, especially for recommending Elements of Statistical Learning. And even so, it will be hard to learn anything from only eight observations... $\endgroup$ – Stephan Kolassa Jan 10 '13 at 10:21
  • 1
    $\begingroup$ Thanks for the help. I'll have a look into that. Sure, 8 observations are very poor. The alternative, i.e. 7 days of work, would be 10 observations. So I would like to see what I can do with the existing 8 ones. $\endgroup$ – TheJoez Jan 10 '13 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.