I have categorical, time-series data distributed in space. It is very noisy, but over the whole series there are big shifts in distribution - my goal is to see how these shifts progressed in space. So, an extract from a simpler one of these datasets looks like this (for other datasets there are more than two possible classes of observation, which is why I say 'simpler' here):

observation time        y               x
a           1446.129    60.3912628      5.3220544
a           1446.216    59.2675699      10.4075608
b           1446.233    59.9138688      10.7522454
a           1446.274    59.2556050296   10.1304655479
b           1446.337    59.538275       9.261831
a           1446.693    58.4476022      14.8902335

And with kernel smoothing of this whole time-series, we can see that there is a shift from 100% category 'a' at the beginning to close to 100% category 'b' at the end:

kernel smoothed time series data

I have been using k-nearest-neighbour smoothing and Gaussian kernel smoothing on these data to try to see how these shifts progressed in space - basically, using a Gaussian kernel to weight all the data for time and then either another Gaussian kernel or k-nearest neighbours to produce weights for space. This gives very plausible results. However, the data is highly unevenly sampled in space and (I think) this results in artefacts with either method.

k-nearest neighbour smoothed in space, Gaussian kernel smoothed time

This image represents average rates of category 'b' for a single point in time (i.e. weighting all the data by distance from the point in time of interest on a Gaussian curve), with each point in space an average of the k-nearest neighbours. Colour represents rate of 'b', point size is log(sample size). The problem is that the largest, central point represents over 1000 samples, where most other points represent just 1-5 samples. As a result, the appearance of a ring around that point with high rate of 'b' could just be the result of that point being included in the estimation. This problem is the same for kernel smoothing in space, other than that there the influence of oversampled points falls off gradually instead of producing a discrete ring.

What's more, I'm now trying to find ways to identify the optimum bandwidths for these smoothing functions, and I think I'm finding that having extreme differences in sample density produces problems there too. With datasets with extreme differences in sample density, the optimum bandwidths often seem to turn out to be close to zero - which is unhelpful, as the datasets are too noisy to see any patterns in without smoothing.

Is there a different smoothing method I should be using for this kind of data? Or a way to refine the approaches I've got that would avoid these artefacts - or at least allow me to identify where a phenomenon I see in the results is an artefact and where it's real?

(disclaimer: as I'm sure is very obvious, I'm not a statistician and have just been figuring this out as I go. I may be approaching this entirely wrongly!)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.