Do we always assume cross entropy cost function for logistic regression solution unless stated otherwise?

Question

I am using Matlab glmfit for logistic regression. Now I know that usually people use the cross entropy to evaluate the error in predictions against the true labels ( which different than the linear regression where they use Mean Square Error (MSE) or Sum of Squares SS). My questions:

1) Is true that whenever I use builtin logistic regression function (as in glmfit in matlab) I would assume they use the cross entropy as a cost function? I read through the documentation of glmfit here but nothing mentioned about that. https://www.mathworks.com/help/stats/glmfit.html

2) And in general if someone handed me a logistic regression solution (or read a paper that says they implemented a logistic regression) is it fair to assume that they used cross entropy as a cost function unless stated otherwise?

Note my question is mathematical, rather it is about the implementation in matlab and the norm in this field..

Answer 1

In Matlab 2014a yes, cross entropy is used for logistic regression (see line 282 in glmfit.m):

% Define variance and deviance for binomial, now that N has NaNs removed.
if isequal(distr, 'binomial')
    sqrtN = sqrt(N);
    sqrtvarFun = @(mu) sqrt(mu).*sqrt(1-mu) ./ sqrtN;
    devFun = @(mu,y) 2*N.*(y.*log((y+(y==0))./mu) + (1-y).*log((1-y+(y==1))./(1-mu)));
end

and I guess this is also the norm in the field and in other versions of matlab (but maybe there is someone more expert in the field with a different opinion), as there are good reasons to not use MSE (this is discussed for example here: Using MSE instead of log-loss in logistic regression or here why sum of squared errors for logistic regression not used and instead maximum likelihood estimation is used to fit the model?).

Answer 2

TLDR; Logistic regression model uses logistic loss function by definition.

Logistic regression is a kind of generalized linear model, so as any other GLM, it is defined in terms of three components:

1. Linear combination

   $$ \eta = \beta_0 + \beta_1 X_1 + \dots + \beta_k X_k $$

   is not very interesting, as it is the same for all the generalized linear models.

2. Link function

   $$ g(\mu) = \eta $$

   where $\mu$ is the conditional mean of $Y$,

   $$ E(Y|X) = \mu $$

   is the feature of GLMs that makes them so flexible. It is important because it let's you to go beyond simple linearity in modeling relation between predictors and the conditional mean of the target variable. There are many different link functions. The most simple link function is identify function $g(x) = x$, it is the default link function used in Gaussian GLM and leads to linear regression. For logistic regression logit function $g(x) = \log(\tfrac{x}{1-x})$ is the default choice, but there are also other possible choices like probit function $g(x) = \Phi(x)$ (in such case, people often call it probit regression to make it clear that non-default link function was used). In the case of logistic regression link is important because it bounds $\mu$ in $(0, 1)$ interval, what is needed for it to be a valid probability, as we model the probabilities of success.

3. Likelihood function for the response variable

   $$ Y|X \sim f(\mu, V(\mu)) $$

   is the probabilistic part of the model, as it describes the probability distribution $f$ of the response variable. Notice that the probability distribution have also additional parameters (e.g. standard deviation for Gaussian distribution), so we need additional variance function $V(\mu)$, but that's not the case for Bernoulli distribution and logistic regression, since it has only a single parameter.

   GLMs are parametric models, i.e. they are defined in terms of probability distributions, with unknown parameters to be estimated. The parameters are estimated by maximizing the likelihood function. Instead of maximizing the likelihood function, we could minimize some loss function, e.g. minimizing squared loss is equivalent to maximizing Gaussian likelihood; minimizing absolute loss, to maximizing Laplace likelihood; and minimizing log loss, to maximizing the Bernoulli likelihood (see also Which loss function is correct for logistic regression?), etc. What follows, minimizing different loss would be equivalent with maximizing different likelihood function. While link function only transforms the linear function of predictors, the likelihood function actually defines the relation of the conditional mean with your data.

   (But be warned, that having those three components does not automatically make the model a proper GLM, e.g. beta regression is not considered as a GLM.)

To put those pieces together, the logistic regression model is

$$ Y|X \sim \mathcal{B}(\,g^{-1}(X\beta)\,) $$

where $Y$ is a binary random variable following a Bernoulli distribution (alternatively as a Binomial distribution over $y$ successes in $n$ trials happening with probability $\mu$), where the "probability of success" is a linear function of $X$ passed through the link function $g$. The key point is that we assume a distribution over $Y$, that has mean depending on $X$, the link function is just about applying some transformations to the "raw" linear function of $X$.