Understanding the difference between MLE and MSE

Question

When using MSE in linear regression, I understand that we aim to minimize the average of the square errors between the predicted value $f(x)$ and the actual label y. My understanding is that we aim to reduce the geometric difference between $f(x)$ and the real value $y.$ I think this comes from $f(x)-y.$

My initial thought was that like MSE, the logistic loss function also tries to minimize the distance between the data points and the prediction. Especially when in lecture a fitted sigmoid curve was plotted to the data set for extreme input values, the predicted values match the dataset almost perfectly.

Now I have two questions:

1. I thought the output from the logistic model the outputs are probabilities. So how can I even map them on the data set, where the y-labels are units.

2. If not geometric difference, what is essentially measured by the logistic loss function. What discrepancy is the logistic loss function aiming to quantify and minimize in the context of binary classification problems?

Answer 1

I thought the output from the logistic model the outputs are probabilities.

Good!

If you believe this---and I mean really believe it, not just know to say this as the full-credit answer to an exam question---you are quite a bit in front of many people who use logistic regression for "classification" problems and (mistakenly) believe logistic regressions to make explicit classifications. (Perhaps refer to my answer here for more detail.)

Yes, the output of a logistic regression is either the predicted probability or a predicted log-odds that can be converted to a probability. Since either can be converted into the other, it is not worth arguing about the true output of a logistic regression model. The important fact to know is what your software gives as its predictions. predict_proba in sklearn will predict probabilities, while glm in R gives log-odds. If you use a different software package, the output should be documented somewhere.

The reason the classical logistic regression gives probabilities is because it aims to predict the probability parameter $p$ of a binomial distribution. A binomial distribution also has an $n$ parameter, meaning that it can be thought of like multiple flips of the same coin that lands on heads with probability $p$ and tails with probability $1-p$.

In a binary "classification" problem where it is common to use a logistic regression (especially for "classification" problems), that binomial distribution is taken to have an $n=1$, so it is just one flip of that coin. You would consider a successful flip to take the value $1$ and an unsuccessful flip to take the value $0$.

This leads to the answer to your first question. You do not input the category names like "heads" and "tails" into the log loss. You code them as $0$ and $1$. Then you have the predictions, which are the probabilities of falling in the category that is coded as $1$. You now have values that you can input into the log loss equation, where $y$ is your $0/1$-coded observations and $\hat p$ is your predicted probabilities.

$$ \text{LogLoss}\left(y, \hat p\right) = -\dfrac{1}{N}\overset{N}{\underset{i=1}{\sum}}\bigg[ y_i\log(\hat p_i) + (1 - y_i)\log(1 - \hat p_i) \bigg]\\ y_i \in\{0, 1\}\\ \hat p_i \in (0, 1) $$

Regarding the second question, the log loss comes from maximum likelihood estimation of the logistic regression parameter (the regression coefficients) for a binomial distribution, not a distance. It happens that maximum likelihood estimation for a Gaussian distribution coincides with minimizing Euclidean distance between predicted and observed values, but this does not generalize so nicely to all distributions (such as binomial). In fact, the log loss does not even obey the axioms of a metric in the sense of a metric space, which is typically taken as the most general mathematical space where "distance" has a reasonable meaning.

However, there is still something distance-like about the log loss, in that, as the predicted probability of event $1$ approaches $1$ when the observation really was event $1$, the loss decreases toward $0$. Likewise, as the predicted probability of event $1$ approaches $0$ when the observation really was event $0$, the loss decreases toward $0$.

library(ggplot2)
p <- seq(0.01, 0.99, 0.005)
y0 <- rep(0, length(p))
y1 <- rep(1, length(p))
L0 <- -(y0*log(p) + (1 - y0)*log(1 - p))
L1 <- -(y1*log(p) + (1 - y1)*log(1 - p))
d0 <- data.frame(
  Prediction = p,
  Loss = L0,
  Truth = "0"
)
d1 <- data.frame(
  Prediction = p,
  Loss = L1,
  Truth = "1"
)
d <- rbind(d0, d1)
ggplot(d, aes(x = Prediction, y = Loss, col = Truth)) +
  geom_line(linewidth = 2) +
  xlab(“Predicted P(1)”

Answer 2

Here is some data:

When $x<9$ we have a very small probability that $y = 1$. When $x > 11$ we have a very large probability that $y=1$. In between things are murkier.

We could get an approximation of the probability by looking at the empirical probability within a "moving window" of size 1:

In the interval $x \in [9,10]$ there are 20 samples, of which $5$ have $y=1$. So the height of the green point at $x = 9.5$ is $\frac{5}{20} = 0.25$.

We decide to fit a sigmoid curve to this:

This is a function of the form

$$ p_{m,b}(x) = \frac{1}{1+e^{-(mx+b)}} $$

We fit logistic regression using maximum likelyhood estimation. Out of all of the different choices of the parameters $m$ and $b$ in our model, we are looking for the choices which will maximize the probability that our sample was generated by those parameters.

For an individual data point $(x,y)$ with $y = 1$, the probability that the model generated that data point is $p_{m,b}(x)$. For an individual data point $(x,y)$ with $y = 0$, the probability that the model generated that data point is $1 - p_{m,b}(x)$. So the probability that the model generated your sample is just the product of all of these:

$$ \prod_{i} \begin{cases} p_{m,b}(x_i) \textrm{ if $y_i = 1$}\\ 1 - p_{m,b}(x_i) \textrm{ if $y_i = 0$} \end{cases} $$

We try to find the values of $m$ and $b$ which make this as large as possible.

In practice we instead take the negative logarithm of this quantity, call it the logistic loss, and try to minimize it. This is equivalent, but nicer for numerical reasons.

$$ \begin{align*} -\log \left(\prod_{i} \begin{cases} p_{m,b}(x_i) \textrm{ if $y_i = 1$}\\ 1 - p_{m,b}(x_i) \textrm{ if $y_i = 0$} \end{cases}\right) & = -\sum_{i} \begin{cases} \log(p_{m,b}(x_i)) \textrm{ if $y_i = 1$}\\ \log(1 - p_{m,b}(x_i)) \textrm{ if $y_i = 0$} \end{cases} \end{align*} $$

Now we can use a notational trick: since $y_i$ is either $0$ or $1$ we can "multiply by $y_i$ to do case analysis".

$$ \ell(m,b)= -\sum_i y_i\log(p_{m,b}(x_1)) +(1- y_i)\log(1 - p_{m,b}(x_1) ) $$

When $y_i = 1$ this simplifies to the first case and when $y_i = 0$ it simplifies to the second case.

This is logistic loss! Hopefully it makes sense why you would want to minimize it now. Minimizing this quantity maximizes the likelihood that your data was generated by the distribution $p_{m,b}(x)$!