4
$\begingroup$

I'm working with data from a resistivity test. However, during the test it is common that a few measurement points are wrong due to technical failure. So I want to find and remove these points.

I have a data size of 1147, most of whos values are very close. When i simply plot it as a scatter plot I can immidetly detect around 10 points that are way of.

I calculated the Z-scores and found 12 points above ABS(3). However, is this a correct way of using the Z score? (since it's not a normal distribution)

I want to calculate the z-scores since it strikes me as a more systematic way to handle the problem, also then i can compare between different resistivity tests and see outliers and their values.

I have also thought about just doing it as a graphical test; plotting the scatter and perhaps a histogram or a boxplot. Any thoughts about that?

So more precisly; Is this a correct way to use the Z-score? Does it seem like a structured/accurate method? Is there any other way that is more commonly used?

Any help is very welcomed! Thank you! /Julia

So I added the data, and the large data points are the ones that are failures. Tyhe histogram shows a very strong positivly skewde distribution. (also the bin are adjusted so to be able to display it at all.
Data scatter

Histogram

$\endgroup$
  • $\begingroup$ How is your data distributed? Is it skewed, has it got heavy tails..? It is more important what it is rather than what it's not. Could you edit to add some more details? Maybe you could add a plot or some summary statistics? $\endgroup$ – Tim Nov 19 '15 at 15:01
  • $\begingroup$ What does this technical failure do to the values? (How will they behave differently to the rest?) -- this may be nearer to a classification problem. $\endgroup$ – Glen_b Nov 20 '15 at 0:36
  • $\begingroup$ They are extremly high, some of them so high that I can emdiatly know that they are wrong, but it is also kind of a grey zone were they could just be extremes and not "wrong". $\endgroup$ – Julia Nov 20 '15 at 10:31
  • 1
    $\begingroup$ I agree with Glen_b that this sounds more like a classification issue as opposed to a test of "fit" (or the lack of it) relative to a theoretical distribution, e.g., based on z-scores. One approach to this could involve finding the hyperplane separating "good" extremes from technical failures and "bad" extremes as in an SVM. Based on your last comment, this may not be possible with an acceptable level of accuracy. $\endgroup$ – Mike Hunter Nov 20 '15 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.