0
$\begingroup$

Suppose I have two data sets. The average of one data set is 5.5 secs and the standard deviation is 0.0435. The average of the other data set is 5.2 secs but the standard deviation is 0.5123. I know that something is wrong with the second data set because the standard deviation is too high.

My question is how I should interpret the standard deviation value so that I could use the value to determine if data set is valid. Standard devation might not be the one I should use to do this. I will appreciate suggestions for any other methods to validate this data set.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Intrinsically there is no such thing as a standard deviation that is "too high", or a data set that is "not valid". "Too high" and "not valid" must be judgements depending on your assumptions of what is supposed to be the reality behind the data. $\endgroup$
    – Robert Israel
    Jul 31 '12 at 6:37
  • $\begingroup$ The problem is hopelessly overdetermined without knowing where the data come from. $\endgroup$
    – mstrfrdmx
    Jul 31 '12 at 7:18
3
$\begingroup$

Does 'Coefficient of Variation' fit the bill?

%CV = SD/MEAN x 100%

Your two samples have a %CV of 0.79% and 9.85% respectively. Suppose your expectation was that the %CV would always be below 1%, then 9.85% could be used to trigger an automated action to search for errors. For example, if the large %CV is actually due to a single error, it is easy to identify as the square of its deviation is the largest of any sample.

Improve this answer
$\endgroup$
2
$\begingroup$

First of all, what you are looking for is subjective and depends of what purpose your data is for. If you are eliminating data or filtering, this is actually changing the data which is analysed.

So the first step is to ask yourself why do you need to filter the data and if you really need it.

If you want to filter the data, you should first probably visually analyse the data using histograms to see the distribution, and after you have made sure you understood why you are having outliers then you can apply various techniques to remove the outliers.

Like any other filtering method, it may lead to false results and spurious conclusions/regressions if you do not understand the data you are analysing.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy