# Estimating model error in $k$-nearest neighbours with strongly spatially autocorrelated training data

In the palaeoclimate world, palaeoecologists have used spatial training sets of say sea-surface temperture (SST) and related this to micro-organisms living at the locations where SST was measured. A popular model to predict SST from species assemblages (so you can count the same species from sediment core and thus predict SST back in time) is the Modern Analogue Technique (MAT). MAT is our name for $k$ nearest neighbours, but we are using it in a regression rather than the more commonly encountered classification setting.

In MAT ($k$-NN) we select the $k$ closest samples in terms of species composition to infer SST. Under standard (leave-one-out, $k$-fold, or bootstrapping) crossvalidation these models have very good performance stats and low model error compared to other modelling approaches. It has been shown that much of this improvement is due to the large degree of spatial autocorrelation in the training data. MAT is more able to use this information than some other techniques, hence the favourable RMSEP).

One solution suggested to this problem (of getting a realistic RMSEP or at least a better understanding of model error/uncertainty or even quality) is to use $h$-block crossvalidation, where samples with distance $h$ of the target sample are removed during crossvalidation. However, with MAT, these will necessarily be the observations that the model needs in order to predict that SST; by throwing out the samples within distance $h$ you are excluding sites with similar species composition and hence forcing the model to extrapolate. This is much less of a problem where the relationships between species and SST are "modelled" for example.

I'm wondering if there are other ways in which spatial autocorrelation can be handled within a $k$-NN framework. Are there better/alternative ways to tackle the problem of estimating an RMSEP for a $k$-NN model in a situation as I describe (strong spatial dependence in the training data). I have been thinking about using a dumb null model using spatial distance against which to compare the $k$-NN model, but that doesn't get at estimating an RMSEP that is not biased due to the spatial autocorrelation.

• Bootstrapping for spatial (or time series) typically takes out whole contiguous blocks (not sure how you would do your h-block for pair-wise distances). Some quick googling brings up Lahiri 2010, which has a section of the last chapter devoted to spatial prediction (looks to me just estimates prediction intervals via bootstrapping). I don't really grok though what is problematic with leave-one-out. – Andy W Aug 21 '13 at 16:50
• @AndyW Thanks. Ordinary CV fails to account for the fact that just because SST is spatially smooth (as are many other important variables;salinity, nutrients) you'll find the $k$ closest species assemblages very close to the site(s) you are predicting for (in LOO or k-fold CV), and hence by definition they'll have similar SST to the left out site(s). That reduces the RMSEP - there may be little relationship between species and SST but due to the spatial autocorrelation you'll get a model that looks good in terms of error (it will be small). The fun begins when you apply this back in time. – Gavin Simpson Aug 21 '13 at 17:15
• @AndyW When you leave out samples within distance $h$ of the left out site (in $h$-block CV), by definition you are removing the most similar species assemblages and hence asking a model (that can only use observed species compositions) to predict SST based on closeness (in compositional terms) to species compositions not now found in the training set. In other words there are now no near neighbours in the cross-validation training set. There needs to be a half-way house, between $k$-fold CV and $h$-block CV; say retaining only 1 or 2 of the observations within $h$ of the left out site. – Gavin Simpson Aug 21 '13 at 17:20
• @AndyW Re "not sure how you would do your $h$-block for pair-wise distances", you can think of us as removing them from the training set before computing the pair-wise distances. Really what we do is just ignore those cells of the already computed matrix when choosing the $k$ nearest neighbours. I'll take a look at Lahiri; it appears we have a copy in the library here. Thanks again for your comments/thoughts. – Gavin Simpson Aug 21 '13 at 18:15