2
$\begingroup$

One theory holds that the increase in variance over time in a particular measure (let's call it $x$) should be linear.

I've collected a dataset in order to test this claim of the theory. What I've found, however, is that $x$ is almost always very small, but is occasionally quite large. The outliers in my dataset (the rare large values of $x$) seem to be causing fluctuations in the variance over time (independent datasets were collected for each time point) that make it difficult to see a clear pattern of increase.

What would statisticians recommend I do to reduce my analysis' vulnerability to outliers, while still addressing the claim of the theory I'm interested in?

One solution to reduce the influence of outliers would be to compute something like the interquartile range. But then, would I be looking for a linear increase in the interquartile range? (It seems to me the answer is no, but I'm not sure what to do about this.)

Another solution might be to fit a Gaussian distribution to my data, and derive the variance from the best-fitting width parameter. But this makes the assumption that my data are distributed normally.

Maybe the easiest thing to do would be to use some threshold for cutting outliers entirely from the dataset. But this seems like cheating to me. (Maybe it isn't cheating.)

Improve this question
$\endgroup$
5
  • $\begingroup$ i've found that removing values farther than 3 SDs from the mean cleans up my estimate of variance substantially, to the point that i feel i can address the claim of the theory. if anyone believes excluding outliers in this way is a bad thing to do that biases my results, please let me know. $\endgroup$
    – user28511
    Feb 19, 2017 at 0:07
  • 1
    $\begingroup$ If the outlier data points are reality, and are not due to measurement or recording error, perhaps your data are indicating the theory as written is not (exactly) correct. Maybe the outliers are important to understanding whatever the actual phenomenon is. It sounds like you data is not Normal. $\endgroup$
    – Mark L. Stone
    Feb 19, 2017 at 1:27
  • $\begingroup$ @MarkL.Stone i think the details of the experiment are probably important to working this out. the data are roughly normally distributed around zero. but there are occasional very large values > 6 SDs away from the mean. cutting these (a tiny percentage of the observations) gives a much cleaner picture of the change in the variance over time. it seems to me fine to cut them -- they are not due to measurement error, per se, but they are likely due to the occurrence of a phenomenon qualitatively different from the one that causes values to be clustered tightly around zero. $\endgroup$
    – user28511
    Feb 19, 2017 at 1:33
  • 2
    $\begingroup$ You have to decide whether you care about the qualitatively different outlying phenomenon. And then treat it accordingly. $\endgroup$
    – Mark L. Stone
    Feb 19, 2017 at 1:45
  • $\begingroup$ Removing values more than 3SDs from the mean is commonly attempted, but it's a terrible approach. One reason is that the SD is itself (tremendously) influenced by any outliers. That's why you need a resistant method like the MAD or Winsored SD. $\endgroup$
    – whuber ♦
    Feb 19, 2017 at 18:13

2 Answers 2

Reset to default
3
$\begingroup$

The median absolute deviation is one generally accepted measure of the spread of data points, robust in the sense that it is insensitive to the exact values of outliers unless outliers represent over half of the observations. This is a very useful alternative to variance/standard deviation in cases like yours. There are also additional robust measures of the spread (scale) of observations; see the references in the linked page for further information.

Improve this answer
$\endgroup$
2
  • $\begingroup$ @dbliss : Also, do consider the important caution raised by Mark L. Stone in his comments on the question: you might be better served by understanding and modeling the occasional extreme outliers, rather than ignoring them. $\endgroup$
    – EdM
    Feb 19, 2017 at 15:29
  • $\begingroup$ Non-mobile version of the link given by EdM: en.wikipedia.org/wiki/Median_absolute_deviation $\endgroup$
    – Oren Milman
    Jun 6, 2020 at 10:13
0
$\begingroup$

It looks like cutting your outliers is a bad idea if it changes your question of research. Principal component analysis is one way to figure this out, since it would allow you for disentangling how responsible any observation is in contributing to each orthogonal axis of variability entailed in your data. And you cannot cut an observation which gathers strongly to itself one of these orthogonal axis of variability without changing irremediably your question of research.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Since $x$ appears to be univariate (it is described as a time series in which repeated measures are obtained at each time), exactly how would one apply PCA to it? $\endgroup$
    – whuber ♦
    Feb 19, 2017 at 15:03
  • $\begingroup$ @w huber. The OP is talking about independent datasets collected for each time point. Which allows for increasing the dimensionality by horizontally stacking these independant $x$s. In this (likely) highly colinear system, cutting a true outlier would increase the condition index. Is this approach wrong or too questionable ? $\endgroup$
    – keepAlive
    Feb 19, 2017 at 17:16
  • $\begingroup$ That implicitly assumes you can pair up the data from one time to the next. That's a strong assumption that's not in evidence in the question. $\endgroup$
    – whuber ♦
    Feb 19, 2017 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.