3
$\begingroup$

I am using Matlab glmfit for logistic regression. Now I know that usually people use the cross entropy to evaluate the error in predictions against the true labels ( which different than the linear regression where they use Mean Square Error (MSE) or Sum of Squares SS). My questions:

1) Is true that whenever I use builtin logistic regression function (as in glmfit in matlab) I would assume they use the cross entropy as a cost function? I read through the documentation of glmfit here but nothing mentioned about that. https://www.mathworks.com/help/stats/glmfit.html

2) And in general if someone handed me a logistic regression solution (or read a paper that says they implemented a logistic regression) is it fair to assume that they used cross entropy as a cost function unless stated otherwise?

Note my question is mathematical, rather it is about the implementation in matlab and the norm in this field..

$\endgroup$
  • 2
    $\begingroup$ I marked it as a duplicate of two other similar questions. Logistic regression is a model with Bernoulli likelihood, what leads to minimizing logistic loss. If you work with binary data and optimize something else, then this is not a logistic regression. $\endgroup$ – Tim May 24 '18 at 7:08
  • $\begingroup$ Thanks for the quick mark! But not sure how these two question related to mine? Do they say anything about the implementation of the cost function in the machine learning toolbox in matlab? (using glmfit) $\endgroup$ – MrX May 24 '18 at 7:32
  • 2
    $\begingroup$ Logistic regression is logistic regression. The model is the same no matter of implementation details. $\endgroup$ – Tim May 24 '18 at 7:56
  • $\begingroup$ I am asking about the cost function here, so you mean if someone used MSE to evaluate the error during the training phase we do not have logistic regression ?! I thought the logtic regression is about the link function! and has nothing to do with the choice of the cost function!! $\endgroup$ – MrX May 24 '18 at 8:00
  • 2
    $\begingroup$ Thanks for the comments, but seems we are talking past each other here, maybe because I am new in this field, but you know that we can use MSE (mean square error) or MAE (mean absolute error) or cross entropy or .. to evaluate the error of the prediction against the labels. You can train the logistic regression model (or any special case of GLM) to minimize any of these metrics (MAE MSE etc.), for instance typically the solution of LR minimizes the MSE, but you can still fit the parameters to minimize MAE instead (maybe using non linear optimization technique) $\endgroup$ – MrX May 24 '18 at 8:21
0
$\begingroup$

In Matlab 2014a yes, cross entropy is used for logistic regression (see line 282 in glmfit.m):

% Define variance and deviance for binomial, now that N has NaNs removed.
if isequal(distr, 'binomial')
    sqrtN = sqrt(N);
    sqrtvarFun = @(mu) sqrt(mu).*sqrt(1-mu) ./ sqrtN;
    devFun = @(mu,y) 2*N.*(y.*log((y+(y==0))./mu) + (1-y).*log((1-y+(y==1))./(1-mu)));
end

and I guess this is also the norm in the field and in other versions of matlab (but maybe there is someone more expert in the field with a different opinion), as there are good reasons to not use MSE (this is discussed for example here: Using MSE instead of log-loss in logistic regression or here why sum of squared errors for logistic regression not used and instead maximum likelihood estimation is used to fit the model?).

$\endgroup$
2
$\begingroup$

TLDR; Logistic regression model uses logistic loss function by definition.

Logistic regression is a kind of generalized linear model, so as any other GLM, it is defined in terms of three components:

  1. Linear combination

    $$ \eta = \beta_0 + \beta_1 X_1 + \dots + \beta_k X_k $$

    is not very interesting, as it is the same for all the generalized linear models.

  2. Link function

    $$ g(\mu) = \eta $$

    where $\mu$ is the conditional mean of $Y$,

    $$ E(Y|X) = \mu $$

    is the feature of GLMs that makes them so flexible. It is important because it let's you to go beyond simple linearity in modeling relation between predictors and the conditional mean of the target variable. There are many different link functions. The most simple link function is identify function $g(x) = x$, it is the default link function used in Gaussian GLM and leads to linear regression. For logistic regression logit function $g(x) = \log(\tfrac{x}{1-x})$ is the default choice, but there are also other possible choices like probit function $g(x) = \Phi(x)$ (in such case, people often call it probit regression to make it clear that non-default link function was used). In the case of logistic regression link is important because it bounds $\mu$ in $(0, 1)$ interval, what is needed for it to be a valid probability, as we model the probabilities of success.

  3. Likelihood function for the response variable

    $$ Y|X \sim f(\mu, V(\mu)) $$

    is the probabilistic part of the model, as it describes the probability distribution $f$ of the response variable. Notice that the probability distribution have also additional parameters (e.g. standard deviation for Gaussian distribution), so we need additional variance function $V(\mu)$, but that's not the case for Bernoulli distribution and logistic regression, since it has only a single parameter.

    GLMs are parametric models, i.e. they are defined in terms of probability distributions, with unknown parameters to be estimated. The parameters are estimated by maximizing the likelihood function. Instead of maximizing the likelihood function, we could minimize some loss function, e.g. minimizing squared loss is equivalent to maximizing Gaussian likelihood; minimizing absolute loss, to maximizing Laplace likelihood; and minimizing log loss, to maximizing the Bernoulli likelihood (see also Which loss function is correct for logistic regression?), etc. What follows, minimizing different loss would be equivalent with maximizing different likelihood function. While link function only transforms the linear function of predictors, the likelihood function actually defines the relation of the conditional mean with your data.

    (But be warned, that having those three components does not automatically make the model a proper GLM, e.g. beta regression is not considered as a GLM.)

To put those pieces together, the logistic regression model is

$$ Y|X \sim \mathcal{B}(\,g^{-1}(X\beta)\,) $$

where $Y$ is a binary random variable following a Bernoulli distribution (alternatively as a Binomial distribution over $y$ successes in $n$ trials happening with probability $\mu$), where the "probability of success" is a linear function of $X$ passed through the link function $g$. The key point is that we assume a distribution over $Y$, that has mean depending on $X$, the link function is just about applying some transformations to the "raw" linear function of $X$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.