1
$\begingroup$

Situation

I am trying to predict the stand type of a forest 10 years following a clear cut. I am using GBM models to do the multinomial classification. The problem is I have several competing methods for how to group the stands in to types. For each stand I have data on the percent coverage of 19 different species. I have tried two different methods for assigning the stands to stand types. The first is using ecological knowledge to group similar stands together based on similar species attributes. The second is to use k-medoids clustering to create groups of similar stands. Both these methods can be used to produce a range of different numbers of Stand Type classes that the model will try to predict.

Questions

1) Say I have picked a number of clusters based on internal cluster quality and and number of ecological types based on domain knowledge ie 9 clusters and 11 ecological types. How can I compare the performance of the 2 gbm models created by using these as the response?

2) Would it be any different if I was comparing 2 models based on different numbers of clusters?

Discussion

I am avoiding using Accuracy because both models involve imbalanced classes. I have calculated Kappa and logloss but I wonder how these are impacted by having different numbers of classes. I have also calculated AUC and prAUC using multiClassSummary() from caret which I believe takes the average of the one-vs-all measure for each class. Are these performance metrics biased by the number of classes in the model? I would imagine that having more classes would make the outcome harder to predict, so if I have a model with 11 classes in the response performing better than one with 9 classes can I take that as an indication that the method of grouping which produced 11 classes is better? I am most interested comparing the resulting models as opposed to using different clustering or modeling methods but if you think this methodology is flawed then feel free to present an alternative.

$\endgroup$
  • $\begingroup$ Have a look at mroman.ch/guides/sensspec.html this might help you. It's essentially just a way to compute sensitivity and specificity and confidences based on a table and you can either compare those directly or use a metric derived from them such as the Youden's index. (You can skip directly to mroman.ch/guides/sensspec.html#n-ary-classification since you already know that accuracy isn't a useful metric). $\endgroup$ – mroman Nov 9 '18 at 16:16
  • $\begingroup$ Right but what are the implications of comparing two models with different numbers of classes in the response variable? $\endgroup$ – see24 Nov 9 '18 at 18:16
  • $\begingroup$ If you want to compare the overall performance the one with higher $Y$ is better. Otherwise if you have a system that recognizes classes A,B,C and one that recognizes A,B,C,D,E you can compute $Y$ for both using only the classes that you want to compare. System 2 might be better overall but worse than System 1 when it comes to the classes A,B,C. $\endgroup$ – mroman Nov 9 '18 at 22:42
  • $\begingroup$ The issue is that the classes aren't clearly distinct so if System 1 recognizes classes A, B, C and System 2 recognizes D, E, F, G, H then class A could be made up of D and half of E, B of the rest of E, F, and G, and C the same as H. So the data is a continuum which can be broken down in different ways represented by System 1 or 2 and I would like to evaluate the success of the two different methods of breaking down the data $\endgroup$ – see24 Nov 20 '18 at 13:32

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.