4
$\begingroup$

I have a given data set $D = \{ x_i, y_i \}_{i=1}^n$ for a regression problem. When I plot the data, it looks like there is an underlying parabola (2nd order linear model) and some outliers.

I want to design an approach using a probabilistic model with a latent binary variable $\{ 0,1 \}$ indicating whether a data point is an outlier or not.

Currently I have no idea what I could do, what would the parameters be in this cause and how are they optimized? Is Expectation Maximization an idea?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

This looks like a situation where you would like to model the errors using a mixture distribution, e.g.,

$e_i \sim pf_1(e_i|\theta_2) + (1-p)f_2(e_i|\theta_2)$

Often, in a simple model, $f_1$ and $f_2$ would be Normal with mean zero and standard deviations $\sigma_1$ and $\sigma_2$, where $\sigma_2$ might be set to 3-10 times $\sigma_1$. Thus, $f_2$ represents the outlier distribution, and $f_1$ represents the "regular" distribution. In the early days of robust analysis, 3 was considered "mild" contamination, and 10 "severe" contamination. You could of course estimate $\sigma_2$ along with $\sigma_1$.

For this type of problem, if you're not being Bayesian, some form of EM is the way to go. Within the framework of the EM algorithm, your missing data would be the latent indicator. You would get out estimates of the probabilities that the observations are drawn from the outlier distribution as well as parameter estimates for the model and the $f_i$. The EM algorithm and Extensions (free ebook download, why not?) will pretty much walk you through the algorithm for this case (section 2.7 in my edition, "Finite Normal Mixtures...", if you don't already have it.)

Of course, you should consider alternatives to mixture modeling as well, e.g., rlm in R, which, in its output, contains information that easily helps identify which observations are being flagged as likely outliers.

Improve this answer
$\endgroup$
5
$\begingroup$

I don't know exactly what you could do that is like what you suggest that makes sense. If the data fit a quadratic function in x but with a few outliers, I think you could simply fit a robust linear regression so that the outliers will be down-weighted. You don't need to detect and remove them.

Improve this answer
$\endgroup$
5
  • $\begingroup$ +1 Look into IRLS, for instance. $\endgroup$
    – whuber ♦
    Commented Jun 22, 2012 at 3:56
  • $\begingroup$ I wanted to detect them. $\endgroup$
    – Mahoni
    Commented Jun 22, 2012 at 21:22
  • $\begingroup$ One way to detect outliers is to look at the largest residuals from a robust regression fit. Outliers usually don't show up well when least squares is applied because least squares tends to be forced to fit closely to the outliers. Influence functions for the regression parameters are another way to find the outliers. the best way to learn about this is to read either Outliers in Statistical data by Barnett and Lewis. or Methods for Statistical Data Analysis of Multivariate Observations by Gnanadesikan: Links to follow $\endgroup$
    – Michael R. Chernick
    Commented Jun 22, 2012 at 21:59
  • $\begingroup$ Barnett and Lewis: amazon.com/Outliers-Statistical-Series-Probability-Statistics/… $\endgroup$
    – Michael R. Chernick
    Commented Jun 22, 2012 at 22:00
  • $\begingroup$ Gnanadesikan for influence functions to detect mulitvariate outliers: amazon.com/… $\endgroup$
    – Michael R. Chernick
    Commented Jun 22, 2012 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.