Let the input variable $X \in \mathcal{X}$ and the target variable $Y \in \mathcal{Y}$. For a fixed hypothesis $h \in \mathcal{H}$ I want to solve \begin{equation} \min_{p(X,Y)} \int_{\mathcal{X}}\int_{\mathcal{Y}}p(x,y)\ell(h(x),y) \,dy \,dx \\ \textrm{s.t.} \quad \quad \int_{\mathcal{X}}\int_{\mathcal{Y}}p(x,y) \,dy \,dx =1 \end{equation}

The Lagrangian here is: \begin{equation} \mathcal{L}(p, \lambda)= \int_{\mathcal{X}}\int_{\mathcal{Y}}p(x,y)\ell(h(x),y) +\lambda p(x,y) \,dy \,dx - \lambda \end{equation}

Is there a way to derive the optimal distribution $p^*(X,Y)$ by solving $\frac{\partial\mathcal{L}}{\partial p(x,y)}=0$ (e.g., as we can do in entropy maximisation problems)? Else, is there another technique I could use to get some insights for the optimal distribution ?

  • $\begingroup$ The solution for this problem is trivial - it's a $\delta$-function at the point that minimizes $\ell(h(x),y)$ $\endgroup$
    – J. Delaney
    Jun 8, 2022 at 19:36
  • $\begingroup$ Thanks for your response @J. Delaney!! I was not aware about that… Can you please explain this to me with more details as an answer? $\endgroup$
    – appa
    Jun 8, 2022 at 22:53

1 Answer 1


Assuming $\ell$ has a minimum at a point $(x_0,y_0)$, $\ell(h(x),y) \ge \ell(h(x_0),y_0) \equiv \ell_0$, then

$$ \iint p(x,y)\ell(h(x),y) \ge \ell_0\iint p(x,y) = \ell_0$$

which means that $\ell_0$ is the minimal possible value of the integral, and it is attained by $\hat p(x,y)=\delta(x-x_0,y-y_0)$.

Note that the problem with trying to solve this via finding a stationary point of the Lagrangian, is that since $p(x,y)$ is a distribution it is bounded, namely $p(x,y) \ge 0$. When a minimization problem is bounded, the solution can be either at a stationary point in the interior of the region (a local minimum), or on the boundary, which is the case in your problem (since $\hat p(x,y) = 0$ for $(x,y) \ne (x_0,y_0)$).

(In fact the Lagrangian in this case doesn't have any stationary points, since equating $\frac{\delta\mathcal L}{\delta p(x,y)}$ to zero will just give $\ell(h(x),y)=\lambda$, which has no solutions unless $\ell(h(x),y)$ is a constant.)

when analyzing such problems it is also a good idea to think about the discrete analogs, which have the same features but are easier to handle. In this case it would be for example

$$\min \sum_i p_i w_i \;\;\; \text{s.t.} \sum_i p_i = 1,\;\; 0\le p_i \le 1 $$

which is clearly solved by setting to 1 the probability that corresponds to the minimal element $w_{\text {min}}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.