3
$\begingroup$

As part of my Master Thesis, I'm designing a relatively simple Biomes Distribution Model. The aim of the model is to predict biomes (macro vegetation types) spatial distribution on Earth, using some (mainly climatic) data as input. I have 24 different variables I can use as input for the model. First I ran principal component analysis http://desktop.arcgis.com/en/arcmap/latest/tools/spatial-analyst-toolbox/how-principal-components-works.htm to sort predictor variables in order of importance. Then I performed a forward selection, starting with the simplest model having just one predictor, up to the most complex model having all 24 input variables. Model is based on Maximum Likelihood Classification:

h ttp://desktop.arcgis.com/en/arcmap/latest/tools/spatial-analyst-toolbox/how-maximum-likelihood-classification-works.htm

(sorry, don't have enough reputation to post another link...).

I used groundtruth data (which I have available for the whole Earth) as training data. For every model, from the simplest to the most complex, I ran the classification and calculated Cohen's Kappa statistics as a measure of goodness of fit. I got this training accuracy curve: training accuracy

As I had the suspicion that using all 24 variables model was overfitting, I wrote the code to perform a 3-fold cross validation (as explained on Wikipedia's Cross-validation article) and re-ran calculations to get the test (or validation) accuracy curve. I was expecting that this new curve would be decreasing after a global maximum for the effects of overfitting. Instead, test accuracy curve came out (almost) identical to the training curve. How to interpret this result? Does it mean that there was no overfitting in the first place?

Improve this question
$\endgroup$
6
  • $\begingroup$ It doesn't address your main question, but you have misapplied PCA by using it to rank variables like that. PCA is unsupervised, meaning that while the top components preserve the most variance in the data, they may have nothing at all to do with classification. The PCA rank is not a measure of predictive importance. $\endgroup$
    – Nuclear Hoagie
    Commented Oct 17, 2017 at 19:52
  • $\begingroup$ Thanks for pointing this out. What can I use in place of PCA if I wanted a measure of predictive importance? $\endgroup$
    – Damjam Kajim
    Commented Oct 17, 2017 at 20:15
  • $\begingroup$ There's a very serious issue here that needs to be addressed, and it is not even mentioned in the current accepted answer. That issue is whether the training and test data are truly independent. For example, if you know that the data are totally stationary at given a fixed time period but the predictor/response relationship is expected to change somewhat in time, then if your training/test data are from the same time period you could get exactly this result and the model could fail spectacularly when run out-of-sample with respect to the time period. In other words: we need more information. $\endgroup$
    – Josh
    Commented Oct 17, 2017 at 21:00
  • $\begingroup$ @Josh , the input data are averages of a 50 years period. As a matter of fact, second part of the thesis would be to apply the model on future-projected climatic data to estimate biomes shifts due to climate change...but yes, I adopt the assumption that the predictor/response relationship won't be changing radically. Strong and harsh assumption, but unfortunately there is no way to test it. Thanks for pointing this out. $\endgroup$
    – Damjam Kajim
    Commented Oct 17, 2017 at 21:53
  • $\begingroup$ @DamjamKajim, sounds like good news. As long as it's something you've taken into consideration you should be in good shape. On another note, you might want to consider letting a few answers come in before accepting one. Once a question has an accepted answer it's less likely to get additional answers, and sometimes you can learn a lot by hearing things expressed in different ways. $\endgroup$
    – Josh
    Commented Oct 18, 2017 at 15:32

1 Answer 1

Reset to default
2
$\begingroup$

Yes, if the test set performance does not dip below the training set performance under any circumstances, then you don't have over-fitting..

But it was still the right move to measure performance on unseen data. That should always be done. Training set performance is much more optional, most of the time you wouldn't even look at it.

I'm not entirely sure what you mean with ground truth data of the earth that composes your training set but that it seems you later divided into training set and test-set. It matters how you do this division. You cannot expect to train only on data regarding Europe and test on Asia for example. Your training data would not be representative of the test cases. (But as your test performance is good, you seem to be fine here.)

PS. PCA wouldn't sort your variables in order of importance. Importance would be defined in terms of how indicative they are of the class label and PCA does not look at the class label.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your quick answer. Yes of course I randomly divided into training set and test set. Thanks falso for your note about PCA. Do you have an indication of what techniques I could efficiently use to correctly rank variables in order of importance? $\endgroup$
    – Damjam Kajim
    Commented Oct 17, 2017 at 20:17
  • $\begingroup$ Look for supervised feature selection. There are multiple questions on the subject here. Never use the labels from the test set to order you variables, only those from the training set. $\endgroup$
    – David Ernst
    Commented Oct 17, 2017 at 20:56
  • $\begingroup$ If the question was worth answering, it's probably worth an upvote. $\endgroup$
    – Josh
    Commented Oct 17, 2017 at 21:14
  • $\begingroup$ I did upvote, though the vote didn't show up cause I've got less than 15 rep. :/ $\endgroup$
    – Damjam Kajim
    Commented Oct 17, 2017 at 21:57
  • $\begingroup$ I should have been more clear; that comment was directed at @DavidErnst. After all, he's the one who provided the answer. ;) $\endgroup$
    – Josh
    Commented Oct 18, 2017 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.