I have a loss function given by:

$$ \mathcal{L}(x, y) = \frac{\sum_{i=0}\sqrt{x_{i}y_{i}}(y_{i} -g(x_{i}))^{2}}{\sum_{i=0}\sqrt{x_{i}y_{i}}} $$

where $g(x)$ is some function to be minimized, $y \in [0,1]$ and $x \in [0, \infty]$

Apparently, this loss function overestimates the value of $g(x)$ when values of $x$ are large and underestimates the the value of $g(x)$ when it is small, but this was not qualified further in my source.

I would like to try and see if this is the case, analytically. My limited understanding would say that by finding the value of $g(x)$ that minimizes the expectation value and observe how it related to the value of $x$, I would be able to argue this point.

I have attempted, following the example here, to derive the value of $g(x)$ that minimizes the expectation value for the loss, $\mathbf{E}(\mathcal{L})$. I am very rusty on my probabilities, so I would appreciate any advice if I have messed up. Particularly, I am not sure whether I have treated $x$ correctly throughout.

My attempt goes as follows:

$$\mathbf{E}(\mathcal{L}) = \frac{\int f(x|y) \sqrt{xy}(y - g(x)))^{2} \ dy}{\int f(x|y) \sqrt{xy} \ dy} $$

Starting with the denominator, we integrate over $y$ to get:

$$\int f(y|x) \sqrt{xy} \ dy = \sqrt{x}\mathbf{E}(\sqrt{y}|x)$$

Then, we look at the numerator and multiply it out:

$$\int f(y|x) \sqrt{xy}(y - g(x))^{2}.dy = \int f(y|x) \sqrt{xy}(y^{2} - 2yg(x) + g(x)^{2}) \ dy$$

We then group the terms:

$$\int f(y|x) \sqrt{xy}y^{2} \ dy - \int f(y|x) 2y\sqrt{y}g(x) \ dy + \int f(y|x)\sqrt{y}g(x)^{2} \ dy$$

We can then integrate over $y$ to get:

$$\sqrt{x}(\mathbf{E}(\sqrt{y}y^{2} | x) - 2g(x)\mathbf{E}(\sqrt{y}y | x)) + g(x)^{2}\mathbf{E}(\sqrt{y} | x))$$

We divide the numerator by the denominator to get:

$$\mathbf{E}(\mathcal{L}) = \frac{(\mathbf{E}(\sqrt{y}y^{2} | x) - 2g(x)\mathbf{E}(\sqrt{y}y | x)) + g(x)^{2}\mathbf{E}(\sqrt{y} | x))}{\mathbf{E}(\sqrt{y}|x)}$$

We take the derivative with respect to $g(x)$:

$$\frac{d\mathbf{E}(\mathcal{L})}{dg(x)} = \frac{-2\mathbf{E}(\sqrt{y}y|x))}{\mathbf{E}(\sqrt{y}|x)} + 2g(x) = 0$$

Thus we conclude that the value of $g(x)$ that leads to the optimal expectation value, $\min \mathbf{E}(\mathcal{L})$, of this loss function is given by:

$$g(x) = \frac{\mathbf{E}(\sqrt{y}y|x)}{\mathbf{E}(\sqrt{y}|x)}$$

Assuming this derivation is correct, I do not see how I can infer the reasoning of the source to this result.

Have I achieved the result correctly? If not, where have I messed up?

Is the hypothesis given in the source a valid one, given this result?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.