Error measure and learning process: sensitivity to outliers

Question

Problem

You have $N$ data points $y_1 \le \cdots \le y_N$ and wish to estimate a 'representative' value.

1) If your algorithm is to find the hypothesis $h$ that minimizes the in-sample sum of squared deviations,

$$E_{in}(h) = \sum_{n=1}^{N}{(h- y_n)^2},$$

then show that your estimate will be the in sample mean,

$$h_{mean} = \frac{1}{N} \sum_{n=1}^{N} { y_n }.$$

2) Suppose $y_N$ is perturbed to $y_N + E$, where $E \to \infty$. So the single data point $y_N$ becomes an outlier. What happens to your two estimators $h_{mean}$ and $h_{med}$?

---

My Answer

I first minimized the function by taking the derivative and set it to zero :

$f(h) = E_{in}(h) = \sum_{n=1}^{N} {(h - y_n)^2}$

$ {{a}/{ay}} ((h - y)^2)$

$2y - 2h \to 2y = 2h \to y = h$

In this case, $h$ is just a number

$h_{mean} = \frac{1}{N} \sum_{n=1}^{N} {y_n}$

However, I am failing to see how this "proof" actually makes sense regarding $h_{mean}$. Would anyone clarify it please? Am I in the correct path?

---

And about the other question, I am really unsure but my answer is this :

Since a single point would obviously be the outlier to infinite, the $h_{mean}$ would increase; however, the $h_{med}$ would remain the same.

I find this answer too simple and I think I am missing something!

REFERENCE

https://work.caltech.edu/

https://work.caltech.edu/telecourse.html#lectures

Answer 1

You answered (1) well.

As far as (2) goes, note that when $N \ge 3$ and $y_N$ is replaced by $y_N+E$ and $y_N + E \ge y_{\lfloor (N+1)/2\rfloor}$,

$$h_{\text{med}}(E) = \frac{1}{2}\left(y_{\lfloor (N+1)/2\rfloor} + y_{\lfloor N/2\rfloor}\right)$$

and

$$h_{\text{mean}}(E) = \left(\frac{1}{n}\sum_{i=1}^{N}y_i\right) + \frac{1}{n}E.$$

Therefore $h_{\text{mean}}$ is a linear function of $E$ with slope $1/N$: in particular, as $E\to\infty$, $h_{\text{mean}}\to \infty$.

However, $h_{\text{med}}$ is constant, so as $E\to\infty$, it remains unchanged.

The lesson is that when you have more than two data values, $h_{\text{med}}$ is unchanged when the largest of them is arbitrarily increased, but $h_{\text{mean}}$ changes without bound. The first one--the median of the data--is resistant to such "outliers" whereas the second--the mean of the data--is sensitive to outliers.