As part of my Master Thesis, I'm designing a relatively simple Biomes Distribution Model. The aim of the model is to predict biomes (macro vegetation types) spatial distribution on Earth, using some (mainly climatic) data as input. I have 24 different variables I can use as input for the model. First I ran principal component analysis http://desktop.arcgis.com/en/arcmap/latest/tools/spatial-analyst-toolbox/how-principal-components-works.htm to sort predictor variables in order of importance. Then I performed a forward selection, starting with the simplest model having just one predictor, up to the most complex model having all 24 input variables. Model is based on Maximum Likelihood Classification:

h ttp://desktop.arcgis.com/en/arcmap/latest/tools/spatial-analyst-toolbox/how-maximum-likelihood-classification-works.htm

(sorry, don't have enough reputation to post another link...).

I used groundtruth data (which I have available for the whole Earth) as training data. For every model, from the simplest to the most complex, I ran the classification and calculated Cohen's Kappa statistics as a measure of goodness of fit. I got this training accuracy curve: training accuracy

As I had the suspicion that using all 24 variables model was overfitting, I wrote the code to perform a 3-fold cross validation (as explained on Wikipedia's Cross-validation article) and re-ran calculations to get the test (or validation) accuracy curve. I was expecting that this new curve would be decreasing after a global maximum for the effects of overfitting. Instead, test accuracy curve came out (almost) identical to the training curve. How to interpret this result? Does it mean that there was no overfitting in the first place?

  • $\begingroup$ It doesn't address your main question, but you have misapplied PCA by using it to rank variables like that. PCA is unsupervised, meaning that while the top components preserve the most variance in the data, they may have nothing at all to do with classification. The PCA rank is not a measure of predictive importance. $\endgroup$ – Nuclear Wang Oct 17 '17 at 19:52
  • $\begingroup$ Thanks for pointing this out. What can I use in place of PCA if I wanted a measure of predictive importance? $\endgroup$ – Damjam Kajim Oct 17 '17 at 20:15
  • $\begingroup$ There's a very serious issue here that needs to be addressed, and it is not even mentioned in the current accepted answer. That issue is whether the training and test data are truly independent. For example, if you know that the data are totally stationary at given a fixed time period but the predictor/response relationship is expected to change somewhat in time, then if your training/test data are from the same time period you could get exactly this result and the model could fail spectacularly when run out-of-sample with respect to the time period. In other words: we need more information. $\endgroup$ – Josh Oct 17 '17 at 21:00
  • $\begingroup$ @Josh , the input data are averages of a 50 years period. As a matter of fact, second part of the thesis would be to apply the model on future-projected climatic data to estimate biomes shifts due to climate change...but yes, I adopt the assumption that the predictor/response relationship won't be changing radically. Strong and harsh assumption, but unfortunately there is no way to test it. Thanks for pointing this out. $\endgroup$ – Damjam Kajim Oct 17 '17 at 21:53
  • $\begingroup$ @DamjamKajim, sounds like good news. As long as it's something you've taken into consideration you should be in good shape. On another note, you might want to consider letting a few answers come in before accepting one. Once a question has an accepted answer it's less likely to get additional answers, and sometimes you can learn a lot by hearing things expressed in different ways. $\endgroup$ – Josh Oct 18 '17 at 15:32

Yes, if the test set performance does not dip below the training set performance under any circumstances, then you don't have over-fitting..

But it was still the right move to measure performance on unseen data. That should always be done. Training set performance is much more optional, most of the time you wouldn't even look at it.

I'm not entirely sure what you mean with ground truth data of the earth that composes your training set but that it seems you later divided into training set and test-set. It matters how you do this division. You cannot expect to train only on data regarding Europe and test on Asia for example. Your training data would not be representative of the test cases. (But as your test performance is good, you seem to be fine here.)

PS. PCA wouldn't sort your variables in order of importance. Importance would be defined in terms of how indicative they are of the class label and PCA does not look at the class label.

| cite | improve this answer | |
  • $\begingroup$ Thanks for your quick answer. Yes of course I randomly divided into training set and test set. Thanks falso for your note about PCA. Do you have an indication of what techniques I could efficiently use to correctly rank variables in order of importance? $\endgroup$ – Damjam Kajim Oct 17 '17 at 20:17
  • $\begingroup$ Look for supervised feature selection. There are multiple questions on the subject here. Never use the labels from the test set to order you variables, only those from the training set. $\endgroup$ – David Ernst Oct 17 '17 at 20:56
  • $\begingroup$ If the question was worth answering, it's probably worth an upvote. $\endgroup$ – Josh Oct 17 '17 at 21:14
  • $\begingroup$ I did upvote, though the vote didn't show up cause I've got less than 15 rep. :/ $\endgroup$ – Damjam Kajim Oct 17 '17 at 21:57
  • $\begingroup$ I should have been more clear; that comment was directed at @DavidErnst. After all, he's the one who provided the answer. ;) $\endgroup$ – Josh Oct 18 '17 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.