4
$\begingroup$

I have a small numeric dataset with 20 observations and 30 variables. I want to approximate Y as a function of the rest 29 Xs(x1,x2,x3...x29). I've tested:

  1. neural network (NN) with 1 hidden layer and 7 nodes

  2. NN with 0 hidden layers (equivalent to regression without interactions)

The reason for testing NN is because it's highly likely interaction to exist among the 29 variables which the NN will capture automatically.

When cross validated, the 1st option showed lower MAPE and MPE error % vs the 2nd. Hence I concluded it's a better fit. Is it safe to use NN with so little data?

Edit: I am planning to create new data points by using the approximation from the fitted model. I can control and change all Xs. The new data points will be fed back into the observations iteratively (21st observation->refit model->22nd observation-> refit model...->100th observation...). I am facing a cold start problem which I have to somehow overcome.

|cite|improve this question
$\endgroup$
  • 4
    $\begingroup$ Having more variables than observations is a problem. You’re going to make it worse by using the neural network. Did you see what happens when you train on 15 and test on 5? I have a hunch that both models will give unacceptable performance, with the neural net being worse. $\endgroup$ – Dave Sep 22 '19 at 14:47
  • $\begingroup$ Thanks for your response. The prediction error for neural network is ~7% and regression around 15%. It's expected that the 29 variables have interaction which NN is capable of capturing vs the regression that doesn't have interaction. $\endgroup$ – nba2020 Sep 22 '19 at 15:08
  • 1
    $\begingroup$ You got those 7% and 15% values testing on data on which you did not train? $\endgroup$ – Dave Sep 22 '19 at 15:12
  • $\begingroup$ Yes that's the average error on data not trained. I perform 80/20 split and lopped over the training 10 times (sampling 80/20 randomly each time). I calculated the error for each 80/20 time and then i averaged the errors of all 10 loops. $\endgroup$ – nba2020 Sep 22 '19 at 15:25
  • $\begingroup$ Theoretically justifiable interactions or not, you don't have the sample size to learn them, let alone using non-linear models. $\endgroup$ – Frans Rodenburg Sep 23 '19 at 12:49
17
$\begingroup$

Neural networks, in vast majority of cases, need lots of data. If you have 20 observations, neural network is clearly a bad choice. With that small sample size, network would easily memorize the data and overfit. Even cross-validation with that small sample size is disputable, because you'd be validating the results on just few samples at a time.

With that small sample you should aim at simple, robust models like (regularized) linear regression. Check also other questions tagged as .

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks Tim. Considering the relatively simple NN architecture, could it be the case that the simple NN captures interactions which option 2 overlooks? From the nature of the data I do expect interactions to occur among the 29 variables. $\endgroup$ – nba2020 Sep 22 '19 at 18:53
  • 3
    $\begingroup$ @nba2020 with 30 variables and 20 observations you are guaranteed to overfit unless you do something wrong, so I doubt it. $\endgroup$ – Tim♦ Sep 22 '19 at 20:01
  • 1
    $\begingroup$ With only 20 observations, you will need to exclude many of the variables even for linear regression, especially if you want interactions. I'm afraid you'll need to look at collecting more data. $\endgroup$ – James Sep 23 '19 at 12:36
5
$\begingroup$

In your first case, you will have 30 * 7 + 1 parameters to explain 30 * 20 data points. With such a complex model you are bound to overfit and memorize your training data to a degree.

With such a small sample size, your validation results can also be unreliable and merely due to chance. I would maybe try leave-one-out cross-validation to at least get some distribution of the validation score. That makes the comparison a bit more reasonable.

I would go with regression and maybe even do some feature elimination to make the model a bit simpler.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks what do you mean with all vs one cross validation ? $\endgroup$ – nba2020 Sep 22 '19 at 19:18
  • 1
    $\begingroup$ I meant "leave one out" method, where you perform N-fold cross-validation, where N is your number of observations. You train on N-1 samples and validate on the N-th. Here is a good resource cs.cmu.edu/~schneide/tut5/node42.html $\endgroup$ – aghd Sep 22 '19 at 19:27
2
$\begingroup$

The sample size is so low and the variables-to-observations ratio is so high that the modeling framework has to be made even more "modest", beyond linear regression. It is quite likely that some form of regularization will improve the performance of the estimated model out of sample. Try lasso, ridge regression or least angle regression. A good resource on these methods is the 3-rd chapter of

Hastie, T., Tibshirani, R., & Friedman, J. H. (2008). The elements of statistical learning: Data mining, inference, and prediction. New York: Springer.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ While regularization indeed yields finite coefficients and a unique solution even when $n<p$, I can't agree to this being a simpler model. Not only are you estimating the (constrained) coefficients, but you also have to choose the hyper parameter for regularization. $\endgroup$ – Frans Rodenburg Sep 23 '19 at 12:48
  • $\begingroup$ @Frans Rodenburg, I did not say "simpler model", I said: more "modest" model. Yes, in all three methods you have to chose 1 regularization parameter, which can be done via leave-one-out cross-validation. This is a much cleaner problem than thinking which of the 30 variables to include and which to omit.... When the proposed approaches are executed, they are likely to deliver lower out-of-sample prediction error than (regular) multiple linear regression... The three proposed approaches are "modest" in the sense that they shrink the naive OLS coefficients to 0. $\endgroup$ – stans - Reinstate Monica Sep 23 '19 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.