Return to Answer

Your question inspired me to have a look on loss function from point of view of mathematical analysis. This is a disclaimer - my background is in physics, not in statistics.

Let's rewrite $\rm-loss$ as a function of NN output $x$ and find its derivative:

\begin{align} f(x) &= a \ln x + (1-a) \ln (1-x)\\ f^\prime(x) &= \frac{a-x}{x(1-x)} \end{align}

where $a$ is the target value. Now we put $x = a + \delta$ and assuming that $\delta$ is small we can neglect terms with $\delta^2$ for clarity:

$$ f^\prime(a+\delta) = \frac{\delta}{a(a-1) + \delta(2a-1)} $$

This equation let us get some intuition how loss behaves. When target value $a$ is (close to) zero or one, derivative is constant $-1$ or $+1$. For $a$ around 0.5 the derivative is linear in $\delta$.

In other words, during backpropagation this loss cares more about very bright and very dark pixels, but puts less effort on optimizing middle tones.

Regarding assymetry - when NN is far from optimum, it does not matter probably, as you will converge faster or slower. When NN is close to optimum ($\delta$ is small) assymetry disappears.

Your question inspired me to have a look on loss function from point of view of mathematical analysis. This is a disclaimer - my background is in physics, not in statistics.

Let's rewrite $\rm-loss$ as a function of NN output $x$ and find its derivative:

\begin{align} f(x) &= a \ln x + (1-a) \ln (1-x)\\ f^\prime(x) &= \frac{a-x}{x(1-x)} \end{align}

where $a$ is the target value. Now we put $x = a + \delta$ and assuming that $\delta$ is small we can neglect terms with $\delta^2$ for clarity:

$$ f^\prime(\delta) = \frac{\delta}{a(a-1) + \delta(2a-1)} $$

This equation let us get some intuition how loss behaves. When target value $a$ is (close to) zero or one, derivative is constant $-1$ or $+1$. For $a$ around 0.5 the derivative is linear in $\delta$.

In other words, during backpropagation this loss cares more about very bright and very dark pixels, but puts less effort on optimizing middle tones.

Regarding assymetry - when NN is far from optimum, it does not matter probably, as you will converge faster or slower. When NN is close to optimum ($\delta$ is small) assymetry disappears.

Your question inspired me to have a look on loss function from point of view of mathematical analysis. This is a disclaimer - my background is in physics, not in statistics.

Let's rewrite $-loss$ as a function of NN output $x$ and find its derivative:

$ f(x) = a \ln x + (1-a) \ln (1-x) $

$ f'(x) = \frac{a-x}{x(1-x)} $

where $a$ is the target value. Now we put $x = a + \delta$ and assuming that $\delta$ is small we can neglect terms with $\delta^2$ for clarity:

$ f'(a+\delta) = \frac{\delta}{a(a-1) + \delta(2a-1)} $

This equation let us get some intuition how loss behaves. When target value $a$ is (close to) zero or one, derivative is constant $-1$ or $+1$. For $a$ around 0.5 the derivative is linear in $\delta$.

In other words, during backpropagation this loss cares more about very bright and very dark pixels, but puts less effort on optimizing middle tones.

Regarding assymetry - when NN is far from optimum, it does not matter probably, as you will converge faster or slower. When NN is close to optimum ($\delta$ is small) assymetry disappears.

Edit: Fixed last formula

Your question inspired me to have a look on loss function from point of view of mathematical analysis. This is a disclaimer - my background is in physics, not in statistics.

Let's rewrite $\rm-loss$ as a function of NN output $x$ and find its derivative:

\begin{align} f(x) &= a \ln x + (1-a) \ln (1-x)\\ f^\prime(x) &= \frac{a-x}{x(1-x)} \end{align}

where $a$ is the target value. Now we put $x = a + \delta$ and assuming that $\delta$ is small we can neglect terms with $\delta^2$ for clarity:

$$ f^\prime(a+\delta) = \frac{\delta}{a(a-1) + \delta(2a-1)} $$

This equation let us get some intuition how loss behaves. When target value $a$ is (close to) zero or one, derivative is constant $-1$ or $+1$. For $a$ around 0.5 the derivative is linear in $\delta$.

In other words, during backpropagation this loss cares more about very bright and very dark pixels, but puts less effort on optimizing middle tones.

Regarding assymetry - when NN is far from optimum, it does not matter probably, as you will converge faster or slower. When NN is close to optimum ($\delta$ is small) assymetry disappears.

Your question inspired me to have a look on loss function from point of view of mathematical analysis. This is a disclaimer - my background is in physics, not in statistics.

Let's rewrite $-loss$ as a function of NN output $x$ and find its derivative:

$ f(x) = a \ln x + (1-a) \ln (1-x) $

$ f'(x) = \frac{a-x}{x(1-x)} $

where $a$ is the target value. Now we put $x = a + \delta$ and assuming that $\delta$ is small we can neglect terms with $\delta^2$ for clarity:

$ f'(\delta) = \frac{\delta}{a(a-1) + \delta(2a-1)} $

This equation let us get some intuition how loss behaves. When target value $a$ is (close to) zero or one, derivative is constant $-1$ or $+1$. For $a$ around 0.5 the derivative is linear in $\delta$.

In other words, during backpropagation this loss cares more about very bright and very dark pixels, but puts less effort on optimizing middle tones.

Regarding assymetry - when NN is far from optimum, it does not matter probably, as you will converge faster or slower. When NN is close to optimum ($\delta$ is small) assymetry disappears.

Your question inspired me to have a look on loss function from point of view of mathematical analysis. This is a disclaimer - my background is in physics, not in statistics.

Let's rewrite $-loss$ as a function of NN output $x$ and find its derivative:

$ f(x) = a \ln x + (1-a) \ln (1-x) $

$ f'(x) = \frac{a-x}{x(1-x)} $

where $a$ is the target value. Now we put $x = a + \delta$ and assuming that $\delta$ is small we can neglect terms with $\delta^2$ for clarity:

$ f'(a+\delta) = \frac{\delta}{a(a-1) + \delta(2a-1)} $

This equation let us get some intuition how loss behaves. When target value $a$ is (close to) zero or one, derivative is constant $-1$ or $+1$. For $a$ around 0.5 the derivative is linear in $\delta$.

In other words, during backpropagation this loss cares more about very bright and very dark pixels, but puts less effort on optimizing middle tones.

Regarding assymetry - when NN is far from optimum, it does not matter probably, as you will converge faster or slower. When NN is close to optimum ($\delta$ is small) assymetry disappears.

Edit: Fixed last formula