4
$\begingroup$

Is there a way to detect outliers in data that would fit a non-linear shape? For example, I have data that fits an exponential decay with an obvious outlier. I have simulated data with an example of this:

enter image description here

I want to know the index locations of where these outliers occur in the real data and currently I am doing this by applying an exponential decay fit to the data then finding the residual of each point the fit. I then find if any points go over a certain threshold that I find as some multiple of the standard deviation of the residuals and find any point with a residual over that threshold. Is there a better way to find outliers in a model like this?

Here are the points used in the above array:

 [10., 8.46481725, 7.16531311, 6.0653066,
 5.13417119, 4.34598209, 3.67879441, 3.11403224,
 2.63597138, 2.2313016 , 1.88875603, 1.59879746,
 1.35335283, 1.14558844, 0.96971968, 0.82084999,
 0.69483451, 0.58816472, 25, 0.42143844]
$\endgroup$
1
$\begingroup$

Residuals can be useful, but there are a few things to watch out for.

Your data might be heteroskedastic, i.e. the magnitude of residuals varies with the x-value. In this case, what counts as a "large" residual might depend on the x-value. (This is an issue even with linear relationships.)

You might also need to think about the potential for errors in the measurement of the x-value. For example, if y=2x, mismeasuring x by 1 will always cause an error of 2 in your prediction of y. But if y=2^x, mismeasuring x by 1 can cause very small or very large errors in predicting y, depending on what x is. Strictly speaking this isn't heteroskedasticity, but the results look very similar.

One common trick is transformation - either of the whole function, or of the residuals. In the example you give, I'd consider a log transformation, which will convert your exponential relationship to a linear one; depending on the sort of errors you encounter, it may or may not remove heteroskedasticity.

You can also look at neighbour-based ways to identify outliers. For example, for any given data point P, pick the closest n neighbours to the left and to the right, characterise the range of y-values in that neighbourhood, and then check whether P is consistent with that range.

...and there are many other methods, all the way from "eyeball it on a scatter plot" up to elaborate machine-learning methods like neural nets and SVMs. It's hard to give a definitive answer to your question since it depends on so many factors.

$\endgroup$
  • 1
    $\begingroup$ Can you explain how a neural net and SVM would be useful for the question posed? $\endgroup$ – Glen Mar 20 '17 at 22:45
  • $\begingroup$ Glen: I doubt they would be useful for the specific example given. I mentioned them in the context of the OP's more general question about detecting outliers with non-linear data. I'm aware that both NNs and SVMs have been used for outlier detection, but I'm not the right person to go into detail about that - I don't claim familiarity with those methods, just awareness that they exist. $\endgroup$ – Geoffrey Brent Mar 21 '17 at 2:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.