deriving the optimal distribution

Question

Let the input variable $X \in \mathcal{X}$ and the target variable $Y \in \mathcal{Y}$. For a fixed hypothesis $h \in \mathcal{H}$ I want to solve \begin{equation} \min_{p(X,Y)} \int_{\mathcal{X}}\int_{\mathcal{Y}}p(x,y)\ell(h(x),y) \,dy \,dx \\ \textrm{s.t.} \quad \quad \int_{\mathcal{X}}\int_{\mathcal{Y}}p(x,y) \,dy \,dx =1 \end{equation}

The Lagrangian here is: \begin{equation} \mathcal{L}(p, \lambda)= \int_{\mathcal{X}}\int_{\mathcal{Y}}p(x,y)\ell(h(x),y) +\lambda p(x,y) \,dy \,dx - \lambda \end{equation}

Is there a way to derive the optimal distribution $p^*(X,Y)$ by solving $\frac{\partial\mathcal{L}}{\partial p(x,y)}=0$ (e.g., as we can do in entropy maximisation problems)? Else, is there another technique I could use to get some insights for the optimal distribution ?

Answer 1

Assuming $\ell$ has a minimum at a point $(x_0,y_0)$, $\ell(h(x),y) \ge \ell(h(x_0),y_0) \equiv \ell_0$, then

$$ \iint p(x,y)\ell(h(x),y) \ge \ell_0\iint p(x,y) = \ell_0$$

which means that $\ell_0$ is the minimal possible value of the integral, and it is attained by $\hat p(x,y)=\delta(x-x_0,y-y_0)$.

Note that the problem with trying to solve this via finding a stationary point of the Lagrangian, is that since $p(x,y)$ is a distribution it is bounded, namely $p(x,y) \ge 0$. When a minimization problem is bounded, the solution can be either at a stationary point in the interior of the region (a local minimum), or on the boundary, which is the case in your problem (since $\hat p(x,y) = 0$ for $(x,y) \ne (x_0,y_0)$).

(In fact the Lagrangian in this case doesn't have any stationary points, since equating $\frac{\delta\mathcal L}{\delta p(x,y)}$ to zero will just give $\ell(h(x),y)=\lambda$, which has no solutions unless $\ell(h(x),y)$ is a constant.)

when analyzing such problems it is also a good idea to think about the discrete analogs, which have the same features but are easier to handle. In this case it would be for example

$$\min \sum_i p_i w_i \;\;\; \text{s.t.} \sum_i p_i = 1,\;\; 0\le p_i \le 1 $$

which is clearly solved by setting to 1 the probability that corresponds to the minimal element $w_{\text {min}}$.