# Minimizing the expectation of the loss function

Let $$X$$ be a random variable with density $$f_X(x)$$. I want to find such $$\theta$$ that would minimize the expectation of the loss function $$\mathbb{E}L(x,\theta)$$ where $$L(x,\theta) = |x - \theta|$$ is a L1 loss function.

We can rewrite it as $$\mathbb{E}L(x,\theta) = \int_{\mathbb{R}} |x - \theta|f_X(x)$$. Then find a derivative:$$\frac{d}{d\theta}\int_{\mathbb{R}} |x - \theta|f_X(x) dx = \int_{\mathbb{R}} -\text{sign}(x-\theta)f_X(x)dx$$ But I don't understand what does it mean, how can I find $$\theta$$ from this expression?

As in many optimization problems, you can proceed by finding $$\theta$$ such that the derivative is equal to zero. On a side note, what you have is actually a subderivative, since the absolute value is nondifferentiable at $$0$$.

Set the (sub)derivative to zero:

$$\int_{-\infty}^\infty -\operatorname{sgn}(x-\theta) f_X(x) dx = 0$$

Multiply both sides by $$-1$$, then split the domain of integration into two halves:

$$\int_{-\infty}^\theta \operatorname{sgn}(x-\theta) f_X(x) dx + \int_\theta^\infty \operatorname{sgn}(x-\theta) f_X(x) dx = 0$$

$$\operatorname{sgn}(x-\theta)$$ is equal to $$-1$$ in the first integral (where $$x < \theta$$) and $$+1$$ in the second integral (where $$x > \theta$$):

$$-\int_{-\infty}^\theta f_X(x) dx + \int_\theta^\infty f_X(x) dx = 0$$

Therefore:

$$\int_{-\infty}^\theta f_X(x) dx = \int_\theta^\infty f_X(x) dx$$

This says that the probability mass to left of $$\theta$$ is equal to the probability mass to the right of $$\theta$$. Therefore, $$\theta$$ is the median of the distribution $$f_X$$.

Another resolution that avoids taking the derivative of the absolute value is to notice that \begin{align*}\int_{-\infty}^{+\infty} |x-\theta| f_X(x)\,\text{d} x&=\int_{-\infty}^{\theta} (\theta-x)f_X(x)\,\text{d} x+\int^{+\infty}_{\theta} (x-\theta)f_X(x)\,\text{d} x\\&=\int_{-\infty}^{\theta} F_X(x)\,\text{d} x+\int^{+\infty}_{\theta} (1-F_X(x))\,\text{d} x\end{align*} where $$F_X$$ is the cdf and then differentiate $$F_X(\theta)-(1-F_X(\theta))=0$$ leading to$$F_X(\theta)=1/2$$