# Error measure and learning process: sensitivity to outliers

Problem

You have $N$ data points $y_1 \le \cdots \le y_N$ and wish to estimate a 'representative' value.

1) If your algorithm is to find the hypothesis $h$ that minimizes the in-sample sum of squared deviations,

$$E_{in}(h) = \sum_{n=1}^{N}{(h- y_n)^2},$$

then show that your estimate will be the in sample mean,

$$h_{mean} = \frac{1}{N} \sum_{n=1}^{N} { y_n }.$$

2) Suppose $y_N$ is perturbed to $y_N + E$, where $E \to \infty$. So the single data point $y_N$ becomes an outlier. What happens to your two estimators $h_{mean}$ and $h_{med}$?

I first minimized the function by taking the derivative and set it to zero :

$f(h) = E_{in}(h) = \sum_{n=1}^{N} {(h - y_n)^2}$

${{a}/{ay}} ((h - y)^2)$

$2y - 2h \to 2y = 2h \to y = h$

In this case, $h$ is just a number

$h_{mean} = \frac{1}{N} \sum_{n=1}^{N} {y_n}$

However, I am failing to see how this "proof" actually makes sense regarding $h_{mean}$. Would anyone clarify it please? Am I in the correct path?

And about the other question, I am really unsure but my answer is this :

Since a single point would obviously be the outlier to infinite, the $h_{mean}$ would increase; however, the $h_{med}$ would remain the same.

I find this answer too simple and I think I am missing something!

REFERENCE

https://work.caltech.edu/

https://work.caltech.edu/telecourse.html#lectures

• Is this a question from a course or textbook? If so, please add the [self-study] tag & read its wiki. Jul 4, 2016 at 18:33
• This sounds like homework or some type of self study. That's perfectly OK, but if so please add the self-study tag to your question. Also, why do you expect that the proof about the mean value with respect to minimizing residual sums of squares should have anything to do with the median? The median isn't in part (1) of the problem, just in part (2). There's a reason for that.
– EdM
Jul 4, 2016 at 18:37
• (+1) You seem to be thinking about this situation very well. Have confidence! Your conclusions are fundamentally sound. They might be off a little bit in the details, though. For instance, $h_{\text{med}}$ can vary as $E\to\pm\infty$. Look at some tiny datasets to see what actually happens.
– whuber
Jul 4, 2016 at 19:32
• Added self study tag and added reference of text book! I still dont get how $h_{med}$ can vary, doesnt make sense to me. Jul 4, 2016 at 19:50
• Sometimes things are almost as simple as they seem. Do consider, however, what @whuber recommended about tiny data sets. How are you defining the median if $N$ is even? What if $N=2$?
– EdM
Jul 4, 2016 at 20:49

As far as (2) goes, note that when $N \ge 3$ and $y_N$ is replaced by $y_N+E$ and $y_N + E \ge y_{\lfloor (N+1)/2\rfloor}$,

$$h_{\text{med}}(E) = \frac{1}{2}\left(y_{\lfloor (N+1)/2\rfloor} + y_{\lfloor N/2\rfloor}\right)$$

and

$$h_{\text{mean}}(E) = \left(\frac{1}{n}\sum_{i=1}^{N}y_i\right) + \frac{1}{n}E.$$

Therefore $h_{\text{mean}}$ is a linear function of $E$ with slope $1/N$: in particular, as $E\to\infty$, $h_{\text{mean}}\to \infty$.

However, $h_{\text{med}}$ is constant, so as $E\to\infty$, it remains unchanged.

The lesson is that when you have more than two data values, $h_{\text{med}}$ is unchanged when the largest of them is arbitrarily increased, but $h_{\text{mean}}$ changes without bound. The first one--the median of the data--is resistant to such "outliers" whereas the second--the mean of the data--is sensitive to outliers.