I have a small dataset of 37 observations with students' performance on both cognitive tests (5) and professional tests (6). My goal is to predict professional tests (DV) with cognitive tests(IV). To summarize, the characteristics of my data are:

  1. A very small dataset
  2. DV close to being normally distributed (W-S test) while IV don't have any clear form
  3. Both dependent and independent variables are limited to the range of [0,1]
  4. Multicollinearity between DV is very low

I would like to ask your advice regarding the following:

1 What is the most suitable regression model in my case? - I tried GLM and thought of Tobit model, but it doesn't assume the limitation over independent variables. Also are there models taking into account the bias in the distribution of DV (i.e., median of each Yi>>0.5)?

2 What is the policy regarding outliers when it comes to analysis of small N?- I tried to use Cook's distance and boxplot but it results with expensively big number of outliers.

3 What is the best way to train a model having small sample size? - I'm particularly interested in obtaining model coefficients and, therefore, would like to have the model coefficients that maximize the prediction power. I have tried to make use of K-fold validation, but it's unclear how to average model coefficients without introducing overfitting. What is the reasonable test set and what is the appropriate setting in my case?

Your response and help will be mostly appreciated

X plot Y plot

  • $\begingroup$ I wouldn't train a model with only 37 observations. Maybe data scientists would, but with such a small sample I don't see how you can get valid results. $\endgroup$ – robin.datadrivers Feb 11 '15 at 14:20
  • $\begingroup$ @Robin: aren't there any methods to build bearable model for such a small sample? I always thought traditional regressions were developed in the era of small data :) $\endgroup$ – Michael Feb 12 '15 at 8:54
  • $\begingroup$ My area of expertise is not in predictive analytics or machine learning - that's what I think of when you talk about training. You can certainly run an OLS regression on 37 samples, and keeping in mind your standard errors may be large. Using more complex methods, however, typically require a larger sample size b/c they use maximum likelihood or another similar method that need larger data to properly converge. $\endgroup$ – robin.datadrivers Feb 12 '15 at 13:49
  • $\begingroup$ Are you planning on running 5 different models, one for each dependent variable? The challenge there is due to multiple testing - you may have inflated p-values, and a typical Bonferroni correction could hurt you because of your small sample size. You could get around it by doing a MANOVA first, but I don't know off the top of my head sample size guidelines there. Alternatively do you suspect that the 5 variables reflect an underlying latent ability level? In that case you should be thinking of a SEM or IRT model, which I believe require more data. Unless you are in an exploratory phase. $\endgroup$ – robin.datadrivers Feb 12 '15 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.