2
$\begingroup$

I know that in quadratic discriminant analysis (QDA) we use the variance of each class, so is the formula different than that in linear discriminant analysis (LDA)?

Is it $$\frac{1}{N-K} \sum (x - \mu)(x - \mu)^T$$ or $$\frac{1}{N} \sum (x - \mu)(x - \mu)^T,$$

and how can I generate a quadratic boundary equation from this?

$\endgroup$
3
$\begingroup$

The first formula should be

$$\frac{1}{N-K} \sum_c \sum_{i \in c} (x_i - \hat \mu_c)^T (x_i - \hat \mu_c)$$

and is for calculating the pooled variance, to be used if you're tying the covariance matrix across classes (as in LDA). The $N-K$ term arises from Bessel's correction.

If you're not tying the covariance matrices (as in QDA), then the covariance matrix for class $c$ is

$$\frac{1}{N_c - 1} \sum_{i \in c} (x_i - \hat \mu_c)^T (x_i - \hat \mu_c)$$

if you want an unbiased estimate of the variance, or

$$\frac{1}{N_c} \sum_{i \in c} (x_i - \hat \mu_c)^T (x_i - \hat \mu_c)$$

if you want an MSE estimate of the variance.

Either way, usually you don't calculate the equation of the decision boundary in QDA. Given a test point you just evaluate the posterior probability of each class, and pick the highest.

$\endgroup$
  • 1
    $\begingroup$ I think this answer will become more clear if you briefly explain the difference between LDA and QDA. E.g. your first formula is for LDA, not for QDA. $\endgroup$ – amoeba says Reinstate Monica Dec 14 '14 at 16:23
  • 1
    $\begingroup$ Exactly. +1. $\endgroup$ – amoeba says Reinstate Monica Dec 14 '14 at 16:28
1
$\begingroup$

To answer the second part of your question:

and how can I generate a quadratic boundary equation from this?

You must first understand the difference between LDA and QDA:

  • In Linear Discriminant Analysis (LDA) we assume that the observations within each class are drawn from a multivariate Gaussian distribution with a class-specific mean vector, but a covariance matrix that is common to all $K$ classes.

  • Quadratic Discriminant Analysis (QDA) provides an alternative approach by assuming that each class has its own covariance matrix $\Sigma_k$.

To generate the boundary equation you must know the scoring or discriminant function in the case of QDA. As you don't explicitly ask for the derivation I will state it here as:

$$ \delta_k(x) = \log \pi_k - \frac{1}{2} \log |\Sigma_k| - \frac{1}{2} (x - \mu_k)^T \Sigma_k^{-1} (x - \mu_k) $$

The boundary equation is given by the function of $x$ you obtain by equating the scoring function of two different classes:

$$ \delta_k(x) = \delta_l(x)$$

Which can be very painful... also note that it will be a quadratic function of $x$ - hence the name quadratic discriminant analysis

This is why @Andy advises against calculating the function explicitly - you are better off calculating the score function at each point on a grid (for each class) and then selecting the class that has the highest value. Or you can calculate the probability, which should give the same result

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.