Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a small data set 30 features/predictors and 30 observations. My target variable is Oil production and my predictors are well & reservoir properties (depth, trajectory, temperature, pressure etc) I want to identify the manner in which these features impact the target variable via identifying patterns that may exist in my dataset. The relationships may be non-linear and many features are correlated to each other.

Can you please suggest me some great ways to approach this problem?

Any help will be greatly appreciated! Thanks.

share|improve this question
You've already asked this question here:…. You created an entirely new account just so you could ask it again?? – Steve S Aug 4 '14 at 3:03
Here are some issues: 1.) Your dataset is too small 2.) Your number of features is too large considering your sample size 3.) This issue is highly idiosyncratic--it has less to do with Statistics and more to do with the specifics of your field (i.e. geology/oil production) 4.)You have no idea about the functional form of the underlying relationship. 5.) Along with multicollinearity you should probably be worried about endogeneity (but I may be wrong since I don't know anything about geology and oil production...). – Steve S Aug 4 '14 at 3:12
Offcourse I did not create new account for this question. I had problems with accessing and resetting my password. – Batool Aug 4 '14 at 3:44
I seriously don't think this a idiosyncratic issue, though like any other data set in data sciences it does come from a specific field. I am well aware of the issues, which is precisely why I posted this question for some suggestions to tackle these issues. – Batool Aug 4 '14 at 3:54
As an aside, if you post a link to a csv file with your data (you could even post the data on GitHub) you might get more responses since it will give us something to work with... – Steve S Aug 4 '14 at 4:45
up vote 4 down vote accepted

Here are some strategies that may help you:

  1. Try glmnet, a user-friendly R package that implements elastic-net-penalized regression. In tuning the hyperparameters, you might be best off using leave-one-out cross-validation because your small sample size is so small. I suggest glmnet because it can handle $p\gg n$ cases where you have more predictors than observations. You could try this procedure using as many orders of interaction as your RAM can handle.

  2. Variable reduction strategies like Principle Components Analysis (choosing the components that explain, say 85-95% of the variance) may help, as well. Then again, every single time I have tried variable reduction strategies in $p\gg n$ cases, it hasn't reduced the parameter space that much, and once I found out about regularized regression I've depended even less on this. When using variable reduction, maybe focus on subsets of the variables that show high multicollinearity. If you find yourself in the happy situation whether you've reduced your predictor set to something quite small. Still, you may have separation or quasi-separation in your data, so maybe still go with glmnet for regularization, or a Bayesian regularization method as implemented in R package bayesglm.

  3. Use subject matter experts to create a set of alternative models containing more reasonable subsets of the predictors, and which actually use scientific theory to arrive at the functional relationships between the smaller subset of predictors and the response. Use Bayesian model averaging to combine these models together into a single predictive model. If you don't end up using all 30 features, who cares so long as you have improved prediction substantially beyond a coin flip? Sometimes, the machine learning approach isn't the right one. When you have a very small data set, and you're trying to reduce the dimensionality as much as possible, well, that's pretty much the anti-machine-learning-big-data-type use case.

  4. Collect more data.

share|improve this answer
Even good advice can be bad when it's applied to the wrong situation. Tell me: What do you think the odds are that following 1-3 exactly with this dataset will produce Statistically valid results? – Steve S Aug 4 '14 at 5:20
The odds are low, but the person asked for advice given what they've got, and this is what I believe to be the best advice. It also includes your advice (collect more data), although you went into that strategy in further detail. – Brash Equilibrium Aug 4 '14 at 5:33
I was holding out on up-voting your answer since I figured it'd end up with more votes anyway... But now you have me feeling all guilty and stuff... – Steve S Aug 4 '14 at 5:40
@BrashEquilibrium! That was very informative! Many thanks! Yes PCA is what I had begun with. Logistic regression after PCA gave poor results, but perhaps I should also try regression with LASSO. The whole purpose of posting this question was to check out if there is anything else that I should try (apart from several things I have already tried), before proceeding with further data collection. Thanks again. – Batool Aug 5 '14 at 1:31
Yeah, try regularized regression. But we're serious, you've got very little data, so proceed with extreme caution. – Brash Equilibrium Aug 5 '14 at 4:40

Get more data.

This is probably not the answer you're looking for but there's nothing else to tell you. Frankly, you currently don't even have enough data for a test set. [Are there really no resources online that have similar datasets you can (freely) use??]

Sure, someone may come along and suggest you try some fancy bootstrapping technique but, frankly, I really think you're best served by scouring the internet for some more data you can use (even if you don't end up using this data in your final analysis, additional data could still indirectly help you a lot by suggesting a functional form for your analysis).

Good luck!

share|improve this answer
It is not necessary to have one test set. If you use LOOCV you can use ALL of your data as the test set. Sure it is still not ideal have only 30 observations, but it is still not completely hopeless building a predictive model in this case. – Brash Equilibrium Aug 4 '14 at 5:04
@BrashEquilibrium: The "test set" part wasn't the point of my answer. Nevertheless, point taken... – Steve S Aug 4 '14 at 5:12
Maybe not "hopeless"... but also not too far from it... – Steve S Aug 4 '14 at 5:13
LOL. It just seems like this is freaking geology and there has got to be some good expert model or set of expert models out there that could inform the structure of some simple models that could then be averaged based on their Bayes factors or CV scores. – Brash Equilibrium Aug 4 '14 at 5:15
@BrashEquilibrium: "this is freaking geology" -- I just thought about that line again and it made me laugh. – Steve S Aug 5 '14 at 2:24 Forget the old Gauss and his models on small datasets. This one is possible the best solution. It will provide you with a continuous empirical distribution model (passing all Goodness-of-Fit tests ex. Komolgorov-Smirnov) even on small sets of data. I haven't talked to Kovanic (he must be old man now) for years but I have worked with his older SW for years and it is more like miracle than statistical method.

share|improve this answer
Thank you for bringing this interesting and little-known approach to our attention. – whuber Aug 4 '14 at 12:07
You are welcome. Unfortunately this guy is more NERD than marketing guru so unfortunately it is not that famous as (I believe) it should be. I have seen several use cases that proved not only the concept but the whole product. Im not aware of current SW package I used the ancient one for W3.11 in nineties ;) – Skiper Skiprovic Aug 4 '14 at 14:59
I confess that this sounds a little too good to be true. But I'll have to read about it further to be sure! – Brash Equilibrium Aug 4 '14 at 22:01
@SkiperSkiprovic, quite interesting! Let me do some more reading on it. Thanks! – Batool Aug 5 '14 at 1:23
Just curious: Has any of this work been peer-reviewed? – Steve S Aug 5 '14 at 2:38

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.