2
$\begingroup$

Say we measure a variable dependent on a dichotomic class $X|_\text{class}$ where the distributions conditional on that class are

$$X|_\text{class =1} \sim N(\mu_1,\sigma^2)$$ $$X|_\text{class =2} \sim N(\mu_2,\sigma^2)$$

With this situation we can apply a logistic model, as explained in:

The question here is about which method would create the lowest mean squared error for the classification boundary

  1. We could fit the normal distributions based on the two populations and use that to compute a classification boundary.
  2. We could perform a logistic regression and use that to compute a classification boundary.

These methods do not give the same result as demonstrated with the code below (fitting normal distributions is better, with lower deviations from the ideal boundary, than fitting a logistic regression model).

Question: Apparently, logistic regression is not always the best performer. Are there systems/methods to decide on the use of logistic regression instead of straightforward fitting of normal distributions?

set.seed(1)


sample = function() {
  n = 30
  x = rnorm(n,0)
  y = rnorm(n,1)
  xy = c(x,y)
  z = rep(c(0,1), each = n)
  
  mod = glm(z ~xy, family = binomial())
  
  ### boundary based on average between group means
  boundary1 = mean(c(x,y))    
  ### boundary from logistic model        
  boundary2 = -mod$coefficients[1]/mod$coefficients[2]  
  
  return(c(boundary1,boundary2))
}


b = replicate(10^3, sample())

plot(b[1,],b[2,])
mean((b[1,]-0.5)^2)  ### error from straightforward classification = 0.0166966
mean((b[2,]-0.5)^2)  ###            error from logistic regression = 0.01713042
$\endgroup$
11
  • $\begingroup$ I think I would describe 1. as a parametric Gaussian classifier, rather than LDA, which would seem to me to be a more general algorithm (as it still works for non-Gaussian data)? $\endgroup$ Mar 15 at 15:10
  • 1
    $\begingroup$ @DikranMarsupial I agree and I have changed it. Interestingly, your comment is very close to the idea of my question. You state that a LDA still works for non-Gaussian data, but a Gaussian classifier does as well still work for non-Gaussian data (and I view LDA as just a special case of a Gaussian classifier). The idea behind logistic regression is not well motivated when it is applied to classification instead of regression. It seems to be often used because it is the tool lying around, but not because it is neccesarily the best tool. $\endgroup$ Mar 15 at 15:27
  • 1
    $\begingroup$ Do you know this paper: ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf $\endgroup$ Mar 15 at 15:33
  • $\begingroup$ @CagdasOzgenc not yet. $\endgroup$ Mar 15 at 15:35
  • $\begingroup$ @SextusEmpiricus LDA can be derived in terms of within and between class scatter matrices, without the assumption of Gaussian class conditional distributions, so it isn't necessarily violating any assumptions when used on non-Gaussian data. I'm not sure about the classification-v-regression though as classification can be viewed as just applying a threshold after probability estimation. Personally I really like kernel logistic regression ;o) $\endgroup$ Mar 15 at 15:54

1 Answer 1

0
$\begingroup$

The question might have been badly stated. The answer to the question 'when to choose the more indirect method?' is that this choice is to be made when the conditions as stated in the example of the question are not right.

  • If the conditional distribution of the categories is indeed a normal distribution $X|Y∼N(\mu_Y,\sigma)$, then this is equivalent to a binomial model with logit link $Y|X∼\text{Bernoulli}(p = (1+e^{-\alpha+ -\beta X})^{-1})$.
  • However, the other way around is not true. The logistic model can be related to more different situations, and also when $X|Y$ is not normal distribution

See below an example.

In the first case we have two normal distributions for the categories 1 and 2. This leads to $Y|X$ being following a logistic model

$$X|Y=1 \sim N(1,1) \\ X|Y=2 \sim N(3,1) \\$$

In the second case we also have a logistic model, but this time it has a different process behind the data generation; By drawing $X$ from a uniform distribution, and determine $Y$ based on a Bernoulli distribution

$$X \sim U(-1.5,4.5) \\ Y|X∼\text{Bernoulli}(p = (1+e^{-2(X-2)})^{-1})$$

two different cases

So, which model to choose, the generative Gaussian classifier, or the discriminative logistic model, depends on our believes about the data generating process.

If X is dependent on the category and conditional on that category more or less normal distributed, then the Gaussian classifier can be preferred. But not all cases with a logistic model are like that.

On the other hand if the two classes are normal distributed, but with different variance $\sigma^2$ (in the example equal variance was assumed), then logistic regression might be a bad choice.

Code for the image:

layout(matrix(c(1,2)))
par(mar = c(3,4,2,2), mgp = c(2,1,0))


fit = function(z1,z2, title) {
   r = length(z1)/(length(z2)+length(z1))
   d1 = density(z1, bw = 0.3)
   x1 = d1$x
   y1 = d1$y
   d2 = density(z2, bw = 0.3)
   x2 = d2$x
   y2 = d2$y
   plot(-1,-1, xlim = c(-2,5), ylim = c(0,0.3), xlab = "x", ylab = "density of predictor x", main = title, cex.main = 1)
   polygon(c(x1,rev(x1)), c(y1*r,y1*0), 
           col = rgb(1,0,0,0.3))
   polygon(c(x2,rev(x2)), c(y2*(1-r),y2*0), 
           col = rgb(0,0,1,0.25))
   lines(x1,y1*r, col = 2)
   lines(x2,y2*(1-r), col = 4)

   z = c(z1,z2)
   u = c(rep(1,length(z1)),rep(2,length(z2)))
   w = u-1

   mod = glm(w ~ z, family = binomial)
   plot(w ~ z, col = u*2, xlab = "x", ylab = "data and estimated probability", xlim = c(-2,5), pch = "|")
   zs = seq(-3,7,0.01)
   pu = predict(mod, type = "response", newdata = list(w = zs, z = zs))
   lines(zs,pu)
} 


set.seed(1)

n = 500
x = rnorm(n,1)
y = rnorm(n,3)
fit(x,y, "1st case: Gaussian distributed X")

z = runif(n*2,-1.5,4.5)
pu = (1+exp(-2*(z-2)))^{-1}
u = rbinom(n*2,1,pu)
x = z[u == 0]
y = z[u == 1]
fit(x,y, "2nd case: non Gaussian distributed X")
$\endgroup$
5
  • $\begingroup$ X doesn’t follow a Gaussian distribution. It is a mixture. Secondly decision boundary is not linear in case of correlation. So your generalization is too forward in my opinion. $\endgroup$ Mar 18 at 7:02
  • $\begingroup$ @CagdasOzgenc the conditional distribution X|Y follows a Gaussian distribution. In the multivariate situation the decision boundary remains linear when the covariance matrices of the conditional Gaussian distributions are considered to be the same. So the distributions have the same $\sigma$ but different $\mu_1$ and $\mu_2$ $$X|_\text{class =1} \sim N(\mu_1,\sigma^2) \\ X|_\text{class =2} \sim N(\mu_2,\sigma^2)$$ $\endgroup$ Mar 18 at 7:22
  • $\begingroup$ You are right that the Gaussian classifier is not just a special case of logistic regression. It is only the Gaussian classifier with the additional assumption of equal covariance. So it's not like the logistic regression is more general than the Gaussian classifier, but it was not my intention to state this. $\endgroup$ Mar 18 at 7:27
  • $\begingroup$ Even in case of mathematical equivalency Gaussian estimation is more efficient. However if you consider for example Gaussian case with some outliers, both methods are pretty bad as shown in this paper: proceedings.mlr.press/v28/nguyen13a.pdf. But the arguments from the paper are somewhat moot, because these days you can easily plug in a multi layer neural network or XGBoost and find a space where things are easier to classify. So I don't know why comparing a Gaussian classifier to a logistic regression is relevant anymore unless you are sure about your data generation process (never). $\endgroup$ Mar 18 at 9:29
  • $\begingroup$ @GagdasOzgenc "So I don't know why comparing a Gaussian classifier to a logistic regression is relevant anymore" there are two motivations. (1) Purely theoretical interest. There is an equivalence between logistic regression and two Gaussian distributions for the classes. They give estimates with different accuracy, why and how is that occurring. (2) People tend to use a logistic regression for classification. Given that it is not better in the case of the example with a perfect specification, on may wonder when it is usefully instead. $\endgroup$ Mar 18 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.