0
$\begingroup$

I have approximately 500 datasets and want to predict yield in one season (about 1000 dataset). I only have one kind of feature (20 years monthly rainfall) so there are 240 features I use. I got RMSE of 900 and average of data is about 4000-6000. Is it possible to prevent overfitting using small dataset? Thank you in advance

$\endgroup$
  • $\begingroup$ By 500 data sets do you mean 500 rows if no then can you tell us what these 500 data sets are? It is possible to create model using smaller representative sample without over fitting. $\endgroup$ – Nishad Aug 22 '16 at 5:25
  • $\begingroup$ @Nishad Yes, it mean 500 rows. 500 rows mean there are 500 fields of crop. Yes, because I do optimization using Simullated Annealing but it still result over fitting, tough. $\endgroup$ – Qhueen Aug 22 '16 at 6:00
  • $\begingroup$ You need to do cross validation, refer to below links - Cross validation in xgboost (R) rpackages.ianhowson.com/cran/xgboost/man/xgb.cv.html CV as a concept robjhyndman.com/hyndsight/crossvalidation & en.wikipedia.org/wiki/Cross-validation_(statistics) $\endgroup$ – Nishad Aug 22 '16 at 6:52
  • $\begingroup$ I did cross validation (10 folds) and tune parameters using grid search but the error is still high @Nishad $\endgroup$ – Qhueen Aug 22 '16 at 7:40
0
$\begingroup$

If you have only monthly rainfall data to forecast crop yield, then you have only one feature to use for that crop, so you indeed have 500 datasets of each crop and 240 records for each dataset with only one feature - monthly rainfall. I'm assuming that each crop is a different type of plant, and since the impact of rainfall on its growth would be different, you should really model these as 500 different regression models.

So a dataset with 240 records is not too small, you may still be able to train it well without overfitting. Cross-validation is a good approach to prevent overfitting. A good way to estimate how many records are good enough to fit your model, you could draw learning curves when training your model.

$\endgroup$
  • $\begingroup$ Thank you! I get the idea to solve the problem by drawing curve as you mentioned @Sandeep S. Sandhu $\endgroup$ – Qhueen Sep 13 '16 at 6:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.