Timeline for Using cross-entropy for regression problems
Current License: CC BY-SA 4.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 15, 2020 at 18:13 | vote | accept | Josh | ||
| Jul 15, 2020 at 16:44 | comment | added | Sebastian | Exactly, you got it:) | |
| Jul 15, 2020 at 16:42 | comment | added | Josh | Thanks @Sebastian. That makes sense. I suppose the above is equivalent to arguing that minimizing CE is always the same as doing MLE, and of course, the MLE solution does not have to be the one that minimizes the L2 loss (i.e. it's model dependent) right? | |
| Jul 15, 2020 at 13:43 | comment | added | Sebastian | Yes exactly, they are minimizing the same loss (or to be more precise their solution is the same (CE loss has this additional constant $n\log(\frac{1}{\sqrt{2\pi}})$ that does not matter for the optimization No without a normal distribution this does not hold. If you substitute normal distribution with Laplace distribution this will result in the minimization of the $L1$ loss | |
| Jul 15, 2020 at 13:03 | comment | added | Josh | Thanks - so in relation to "What's the benefit of using MSE for regression instead?", when you say "minimizing CE with the assumption of normality is equivalent to the minimization of the 𝐿2 loss" what's the relationship then between CE and MSE, e.g. under the assumption of normality? Would the Empirical Risk be the same and we would be minimizing the same loss? and would that relationship hold when we don't use a Normal distribution? | |
| Jul 15, 2020 at 6:13 | comment | added | Sebastian | @Josh as Eweler already pointed out: we imagine that $H(q, f_\theta)$ has some fixed but unknown value (for a fixed $\theta$) by the term $1/n\sum_{i=1}^n-\log(f_\theta(x_i)) \approx \int -\log(f_\theta(x)) q(x)dx$ we approximate this quantity empirically and minimize this quantity as a proxy because we have no way to minimize the term we actually care about, i.e. the underlying real cross-entropy. This is generally referred to as empirical risk minimization (the risk is the theoretical value). Note that we usually drop the $1/n$ because is irrelevant to the optimization. | |
| Jul 15, 2020 at 4:18 | comment | added | Eweler | That means approximating the expectation with respect to the true data distribution $p^*(x)$ with a Monte Carlo estimate using a set of samples $S$: $ \int dx \, p^*(x) f(x) \approx \frac{1}{\vert S \vert}\sum_{i \in S} f(x_i), x_i \sim p^*(x)$. Since typically we are unable to evaluate the integral analytically or in reasonable (polynomial) time for most problems. | |
| Jul 14, 2020 at 19:49 | comment | added | Josh | Thanks Sebastian. What do you mean by "empirical approximation to the cross entropy" ? | |
| Jul 14, 2020 at 19:22 | comment | added | Sebastian | The goal was just to illustrate that minimizing CE results in the minimization of L2 loss when we assume normality. | |
| Jul 14, 2020 at 19:21 | history | edited | Sebastian | CC BY-SA 4.0 |
added 18 characters in body
|
| Jul 14, 2020 at 19:19 | comment | added | Dave | Then what's the point of the Gaussian assumption? In linear regression, we make a Gaussian assumption to do parameter inference, which is less important in neural networks. | |
| Jul 14, 2020 at 19:15 | comment | added | Sebastian | the estimate for the mean would not change | |
| Jul 14, 2020 at 19:10 | comment | added | Dave | What if we don't have (identical) normal error terms? | |
| Jul 14, 2020 at 18:58 | history | answered | Sebastian | CC BY-SA 4.0 |