I have categorical, time-series data distributed in space. It is very noisy, but over the whole series there are big shifts in distribution - my goal is to see how these shifts progressed in space. So, an extract from a simpler one of these datasets looks like this (for other datasets there are more than two possible classes of observation, which is why I say 'simpler' here):
observation time y x
a 1446.129 60.3912628 5.3220544
a 1446.216 59.2675699 10.4075608
b 1446.233 59.9138688 10.7522454
a 1446.274 59.2556050296 10.1304655479
b 1446.337 59.538275 9.261831
a 1446.693 58.4476022 14.8902335
And with kernel smoothing of this whole time-series, we can see that there is a shift from 100% category 'a' at the beginning to close to 100% category 'b' at the end:
I have been using k-nearest-neighbour smoothing and Gaussian kernel smoothing on these data to try to see how these shifts progressed in space - basically, using a Gaussian kernel to weight all the data for time and then either another Gaussian kernel or k-nearest neighbours to produce weights for space. This gives very plausible results. However, the data is highly unevenly sampled in space and (I think) this results in artefacts with either method.
This image represents average rates of category 'b' for a single point in time (i.e. weighting all the data by distance from the point in time of interest on a Gaussian curve), with each point in space an average of the k-nearest neighbours. Colour represents rate of 'b', point size is log(sample size). The problem is that the largest, central point represents over 1000 samples, where most other points represent just 1-5 samples. As a result, the appearance of a ring around that point with high rate of 'b' could just be the result of that point being included in the estimation. This problem is the same for kernel smoothing in space, other than that there the influence of oversampled points falls off gradually instead of producing a discrete ring.
What's more, I'm now trying to find ways to identify the optimum bandwidths for these smoothing functions, and I think I'm finding that having extreme differences in sample density produces problems there too. With datasets with extreme differences in sample density, the optimum bandwidths often seem to turn out to be close to zero - which is unhelpful, as the datasets are too noisy to see any patterns in without smoothing.
Is there a different smoothing method I should be using for this kind of data? Or a way to refine the approaches I've got that would avoid these artefacts - or at least allow me to identify where a phenomenon I see in the results is an artefact and where it's real?
(disclaimer: as I'm sure is very obvious, I'm not a statistician and have just been figuring this out as I go. I may be approaching this entirely wrongly!)
