2
$\begingroup$

Let $X$ be a random variable with density $f_X(x)$. I want to find such $\theta$ that would minimize the expectation of the loss function $\mathbb{E}L(x,\theta)$ where $L(x,\theta) = |x - \theta|$ is a L1 loss function.

We can rewrite it as $\mathbb{E}L(x,\theta) = \int_{\mathbb{R}} |x - \theta|f_X(x)$. Then find a derivative:$$\frac{d}{d\theta}\int_{\mathbb{R}} |x - \theta|f_X(x) dx = \int_{\mathbb{R}} -\text{sign}(x-\theta)f_X(x)dx$$ But I don't understand what does it mean, how can I find $\theta$ from this expression?

$\endgroup$

2 Answers 2

5
$\begingroup$

As in many optimization problems, you can proceed by finding $\theta$ such that the derivative is equal to zero. On a side note, what you have is actually a subderivative, since the absolute value is nondifferentiable at $0$.

Set the (sub)derivative to zero:

$$\int_{-\infty}^\infty -\operatorname{sgn}(x-\theta) f_X(x) dx = 0$$

Multiply both sides by $-1$, then split the domain of integration into two halves:

$$\int_{-\infty}^\theta \operatorname{sgn}(x-\theta) f_X(x) dx + \int_\theta^\infty \operatorname{sgn}(x-\theta) f_X(x) dx = 0$$

$\operatorname{sgn}(x-\theta)$ is equal to $-1$ in the first integral (where $x < \theta$) and $+1$ in the second integral (where $x > \theta$):

$$-\int_{-\infty}^\theta f_X(x) dx + \int_\theta^\infty f_X(x) dx = 0$$

Therefore:

$$\int_{-\infty}^\theta f_X(x) dx = \int_\theta^\infty f_X(x) dx$$

This says that the probability mass to left of $\theta$ is equal to the probability mass to the right of $\theta$. Therefore, $\theta$ is the median of the distribution $f_X$.

$\endgroup$
4
$\begingroup$

Another resolution that avoids taking the derivative of the absolute value is to notice that \begin{align*}\int_{-\infty}^{+\infty} |x-\theta| f_X(x)\,\text{d} x&=\int_{-\infty}^{\theta} (\theta-x)f_X(x)\,\text{d} x+\int^{+\infty}_{\theta} (x-\theta)f_X(x)\,\text{d} x\\&=\int_{-\infty}^{\theta} F_X(x)\,\text{d} x+\int^{+\infty}_{\theta} (1-F_X(x))\,\text{d} x\end{align*} where $F_X$ is the cdf and then differentiate $$F_X(\theta)-(1-F_X(\theta))=0$$ leading to$$F_X(\theta)=1/2$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.