Cross-validation necessary when using Random Forest? I am new to this forum, but I am stuck with a couple questions related to image classification, and seeing the kind and highly useful messages that people provide, I had the hope that someone could help me out in clarifying something for me. I will try to be as explicit and clear as possible on my problem, but I am rather confused as you might see. So here goes:
Question 1: I am wanting to use an object based image analysis in ERDAS to assess differences in vegetation types (i.e. shrubs, grasses, trees, mosses) in a peatland landscape using spectral imagery, but I am unsure about the methods to use and which steps follow each other. Now you can just press on some buttons in a program, but I am wanting to understand better where my decision comes from and if this is credible. The questions therefore are: how do you determine which kind of classifier is best for your data, and if and what kind of resampling method to use? I read for example that it is necessary to resample your samples (e.g. Monte-Carlo cross-validation, or k-fold) and average accuracies of all splits you made to increase their validity. Besides, does resampling follow after classification? So would the steps then be:

*

*Subset your data (e.g. 67:33 into training, testing)

*Pick a classification algorithm that suits your needs and data

*Run the classification and note accuracy

*Resample (split) your data into new training and testing data using a resampling technique

*Re-run classification for all unique splits

*Average accuracies of all splits to get the "real" accuracy

Question 2:
Let's say I were to pick RandomForest, which I think is one classification algorithm; would I still need to cross-validate my data with a resampling technique? For example, I read that RandomForest sort of contains its' own cross-validation because it prevents overfitting through the use of bootstrapping of the variables used in the decision trees. Does this mean that instead of just a classification algorithm, RandomForest is a classification algorithm ánd cross-validation in one? Is it then still possible or needed to cross-validate the models by using for example five splits of data (67:33 training versus testing) and running the RandomForest with each of these unique splits, and then summarizing this information, or would that be overkill?
I am sorry if there is too many questions, and I am willing to clarify if needed. I hope someone could help me and I appreciate all the effort in advance!
Sincerely,
Jasper
 A: There are many fundamental issues you need to address first.  Very important is whether you conceptualize this as a forced-choice classification task or as a continuous prediction task.  "Classifier" denotes the former, and probability models are examples of the latter.  See here.  RF, since it can provide continuous outputs, is more in the prediction than the classification mode.
Random forests, which require incredibly large sample sizes in order to be successful, are notorious for overfitting data especially when there are many candidate predictive features.  Though random forest (RF) algorithms may build in some cross-validation to guide the algorithm, it is not correct to say that this "internal" cross-validation provides unbiased performance of RF.  When you do a proper validation that includes unbiased estimation of smooth calibration curves you will often be shocked at the appalling lack of calibration accuracy from RF.
If your sample size is larger than say 50,000 observations you can affort to have a large (n=20,000) holdout sample for stringent validation of RF predictions.  Otherwise you might look into bootstrapping the entire RF process to obtain unbiased estimates of likely future predictive performance of RF with respect to calibration and predictive discrimination.
A: 
 how do you determine which kind of classifier is best for your data,

The high level answer here is that you want to select a model (classification or regression) that suits

*

*your application task

*your data, and

*the available sample size.

Wrt. 1, @FrankHarrell already pointed out that you need to decide classification as in the output of the model being a class label vs. predicting the probability of the case to belong to a given class. There are further considerations here: e.g. are your classes mutually exclusive or not and how well-defined you expect your classes to be (rough guideline: can you describe characteristics of your class by "positive" words => usually well-defined), or is that description negative in the sense that you can mainly say what it is not => usually ill-defined)
As someone working with spectroscopic imaging data as well (my background is spectroscopy and chemometric data analysis), I'd recommend that you think even more about "proper" regression here. Such a regression model would e.g. predict for each pixel the "concentration" (or a fraction) of shrubs, grasses, trees and mosses. This would allow for pixels to contain mixtures such as half grasses half shrubs - whereas classification models cannot deal well with this situtation.
Which brings us also to point 2: spectroscopy (I guess UV/VIS/NIR reflectance?) is widely used because the spectra (if expressed in the proper unit, e.g. absorbance or log reflectance) are in good approximation linearly related to concentration of chemical species, and that in turn means that they are also linearly related to defined mixtures. Such as the phenotypcial composition of grasses vs. shrubs.
Which points to (bi)linear models being appropriate*. Also, assuming you have full spectra rather than only a few wavelengths, the underlying physics and chemistry of the spectroscopic technique allows to say e.g. that regularization via latent variables/factors (PCA/PLS/ridge) fits better with your particular type of data than trying to select a few wavelengths (LASSO).


*sample size. This is in a sense easy: the smaller your sample size, the more you need to take care that you don't overfit.


*

*the smaller the sample size, the more important it is that you bring in external knowledge via selecting an appropriate model ansatz (see 1.).

*you need to monitor/measure overfitting, which is possible e.g. via repeated cross validation or out-of-bootstrap valiation, but not with a single run of cross validation nor leave-one-out cross validation nor a single split test set.

*you then have two approaches available to reduce overfitting: either your regularize your model more, i.e. force it to be less complex (e.g. dimensionality reduction by PCA or PLS, or adding a ridge) or you can aggregate overfit models (e.g. random forest).

* Note that the statistician's "linear" in linear model refers to the parameters, whereas the chemist's "linear" refers to the concentration. Chemometrics uses the term bilinear to express that a model is linear in both senses.


 and if and what kind of resampling method to use?

The choice of "validation scheme" is IMHO less critical/fairly easy:

*

*you will pretty much always be OK with repeated k-fold cross validation or out-of-bootstrap.

*I tend to reserve "single" test sets for proper valiation studies. more here

*I avoid resampling or splitting "flavors" which do not allow to measure model stability/overfitting. E.g. I use leave one out only if there are so few independent samples that I cannot possibly leave out 2 at the same time - this boils down to doing LOO if there are < ≈5 independent cases.

The crucial point instead is to ensure independence. To speak in your example,  the data I get would typically contain thousands or millions of spectra, which are however, only from, say, 10 different (separate) peatland regions. This basically means we're speaking about sample size 10 for application scenarios that claim to predict for any other peatland area. Thus, it wouldn't matter whether you use repeated k-fold cross validation vs. out-of-bootstrap. But in both cases it would matter crucially that you split by peatland region.
(Even if your application scenario stays within one peatland region, neighbour pixels are likely not independent - establishing independence is more difficult then, but just as important)
For the sake of completeness, independence also means independent of any modeling choices. I.e., once you use resampling results to choose between modeling options, they cannot be used as performance estimate any more since they are now part of the model training process.

Let's say I were to pick RandomForest, which I think is one classification algorithm; would I still need to cross-validate my data with a resampling technique?

So, let's check the random forest against the points above.

*

*Random forests have been (successfully) used for spectroscopic data even though they don't fit that well with spectroscopic data since they look at single wavelengths while we expect the information to be spread out over many, in particular neighbouring, wavelength channels for physical and chemical reasons.


*(I'd recommend looking up random forests fairly in detail if you use them: they are aggregated models and thus act different from many other models in specific senses.)


*The usual random forest implementations implicitly assume that every row of the data is an independent case. Under this assumption, the internal out-of-bag performance estimates will be fine. But this assumption is clearly violated here (as in almost all real-world data I've met so far).
The overfitting and bad calibration (predicted class membership probability being severely off) @FrankHarrel points out are symptoms of this.
You now have two possibilities:

*

*use a random forest implementation that provides independent splits also for your data. (Which would be best, but likely boils down to you writing your own implementation of RF)


*You may be able to live with the fact that the RF is overfitted, and thus stays behind what would be possible with approach 1. This "only" requires that you show the RF you obtain is anyways good enough for your application purposes.
For this, you need an independent test of the (overfit) RF you obtain - e.g. by wrapping the RF training in your own repeated k-fold cross validation which provides independent splits for your data.
A: Many thanks for your quick reply Mr. Harrell. If I may recite, what you are saying here is that I should first ask myself whether I would want to use classification in general and if that would be a wise choice considering predicting probabilities would be a better fit as it is continuous rather than binary, and vegetation might never be just one type but rather is fuzzy? I think this is a valuable remark to make and something to think about.
Besides, what I am picking up from your remark on the RF is that there is still need for cross-validation to test for the bias in the model, and I understand what you are saying as the "internal" validation is never sufficient enough to provide unbiased results. I will look into the methods I can use to create an unbiased prediction, and I hope to learn these methods as I go, as I am quite new to this and I want to do this right. Thank you again!
