GDA and LDA terminology

Question

Can the terms LDA (Linear Discriminant Analysis) and GDA (Gaussian Discriminant Analysis) be used interchangeably? Do they often refer to the same thing?

Answer 1

According to Wikipedia:

"The terms Fisher's linear discriminant and LDA are often used interchangeably, although Fisher's original article actually describes a slightly different discriminant, which does not make some of the assumptions of LDA such as normally distributed classes or equal class covariances."

LDA is equivalent to maximum likelihood classification assuming Gaussian distributions for each class. I would have said that "Gaussian Discriminant Analysis" could actually refer to either formulation, as it's an ambiguous name.

However, normally I would associate GDA with the acronym "General Discriminant Analysis" or "Generalised Discriminant Analysis". General Discriminant Analysis is called a "general" because it applies the methods of the general linear model (see also General Linear Models (GLM)) to the discriminant function analysis problem. Generalised Discriminant Analysis also refers to nonlinear methods, but it instead achieved using kernel functions (see e.g. Generalized Discriminant Analysis Using a Kernel Approach), and is also known variously as Kernel Discriminant Analysis (KDA) and Kernel Fisher Discriminant Analysis (KDFA).

Answer 2

GDA (Gaussian Dcriminant Analysis) is a general term for both LDA (Linear Discriminant Analysis) and QDA (Quadratic Discriminant Analysis) where the likelihood probability of each observation given the class, i.e., P(x|y) can be modeled by the multivariate Gaussian distribution.

Here, x is a n-dimensional observation random vector (n=number of features in each observation). y is a label, i.e., one of K classes (say, y=1, y=2, .... y=K).

---

LDA is a case where each observation is drawn from the multivariate Gaussian distribution with class-specific mean vector and 'shared' covariance matrix.

i.e., P(x|y=1) ~ N(u1, sigma), P(x|y=2) ~ N(u2, sigma), . . . . P(x|y=K) ~ N(uK, sigma).

where, N(uK, sigma) denotes Gaussian distribution with mean uK (nx1 vector) and covariance sigma (nxn matrix).

---

QDA is a special case where each observation is drawn from the multivariate Gaussian distribution with class-specific mean vector and 'class-specific' covariance matrix.

i.e., P(x|y=1) ~ N(u1, sigma1), P(x|y=2) ~ N(u2, sigma2), . . . . P(x|y=K) ~ N(uK, sigmaK).

QDA is more flexible (i.e., curvy) than LDA. However, it may suffer from the overfitting problem (i.e., high variance).

Answer 3

In the theory of classification the linear form of the discriminant function is optimal when the class conditional densities are Gaussian with the same covariance matrix. So in that sense I could understand the term Gaussian sneaking in as a term for LDA. However QDA quadratic discrimination is optimal when the class conditional densities are Gaussian and the covariances are different. So that could be termed Gaussian too. So what tdc said is not quite right because he left off the equal covariance matrix condition.