deriving the optimal distribution

Let the input variable $$X \in \mathcal{X}$$ and the target variable $$Y \in \mathcal{Y}$$. For a fixed hypothesis $$h \in \mathcal{H}$$ I want to solve $$$$\min_{p(X,Y)} \int_{\mathcal{X}}\int_{\mathcal{Y}}p(x,y)\ell(h(x),y) \,dy \,dx \\ \textrm{s.t.} \quad \quad \int_{\mathcal{X}}\int_{\mathcal{Y}}p(x,y) \,dy \,dx =1$$$$

The Lagrangian here is: $$$$\mathcal{L}(p, \lambda)= \int_{\mathcal{X}}\int_{\mathcal{Y}}p(x,y)\ell(h(x),y) +\lambda p(x,y) \,dy \,dx - \lambda$$$$

Is there a way to derive the optimal distribution $$p^*(X,Y)$$ by solving $$\frac{\partial\mathcal{L}}{\partial p(x,y)}=0$$ (e.g., as we can do in entropy maximisation problems)? Else, is there another technique I could use to get some insights for the optimal distribution ?

• The solution for this problem is trivial - it's a $\delta$-function at the point that minimizes $\ell(h(x),y)$ Commented Jun 8, 2022 at 19:36
• Thanks for your response @J. Delaney!! I was not aware about that… Can you please explain this to me with more details as an answer?
– appa
Commented Jun 8, 2022 at 22:53

Assuming $$\ell$$ has a minimum at a point $$(x_0,y_0)$$, $$\ell(h(x),y) \ge \ell(h(x_0),y_0) \equiv \ell_0$$, then

$$\iint p(x,y)\ell(h(x),y) \ge \ell_0\iint p(x,y) = \ell_0$$

which means that $$\ell_0$$ is the minimal possible value of the integral, and it is attained by $$\hat p(x,y)=\delta(x-x_0,y-y_0)$$.

Note that the problem with trying to solve this via finding a stationary point of the Lagrangian, is that since $$p(x,y)$$ is a distribution it is bounded, namely $$p(x,y) \ge 0$$. When a minimization problem is bounded, the solution can be either at a stationary point in the interior of the region (a local minimum), or on the boundary, which is the case in your problem (since $$\hat p(x,y) = 0$$ for $$(x,y) \ne (x_0,y_0)$$).

(In fact the Lagrangian in this case doesn't have any stationary points, since equating $$\frac{\delta\mathcal L}{\delta p(x,y)}$$ to zero will just give $$\ell(h(x),y)=\lambda$$, which has no solutions unless $$\ell(h(x),y)$$ is a constant.)

when analyzing such problems it is also a good idea to think about the discrete analogs, which have the same features but are easier to handle. In this case it would be for example

$$\min \sum_i p_i w_i \;\;\; \text{s.t.} \sum_i p_i = 1,\;\; 0\le p_i \le 1$$

which is clearly solved by setting to 1 the probability that corresponds to the minimal element $$w_{\text {min}}$$.