13 votes

Meaning of "reconstruction error" in PCA and LDA

For PCA what you do is that you project your data on a subset of your input space. Basically, everything holds on this image above: you project data on the subspace with maximum variance. When you ...
Vince.Bdn's user avatar
  • 785
13 votes
Accepted

Gaussian Discriminant Analysis and sigmoid function

I'm not sure what it means that GDA "leads to the sigmoid hypothesis" but perhaps it refers to the relationship between GDA (aka Linear Discriminant Analysis, LDA) and logistic regression. Before ...
ilanman's user avatar
  • 4,787
12 votes

When would you use PCA rather than LDA in classification?

You are missing something deeper: PCA isn't a classification method. PCA in machine learning is treated as a feature engineering method. When you apply PCA to your data you are guaranteeing there'll ...
Firebug's user avatar
  • 19.1k
10 votes

When would you use PCA rather than LDA in classification?

Whereas the previos answer by Firebug is correct, I want add another perspective: Unsupervised vs. supervised learning: LDA is very useful to find dimensions which aim at seperating cluster, thus ...
Nikolas Rieble's user avatar
9 votes

What is a Gaussian Discriminant Analysis (GDA)?

I think Andrew Ng's notes on GDA (https://web.archive.org/web/20200103035702/http://cs229.stanford.edu/notes/cs229-notes2.pdf) are the best explanation I have seen of the concept, but I want to "...
Casey L's user avatar
  • 231
9 votes

Estimating the covariance matrix in linear discriminant analysis

You are right. The equation for the shared variance-covariance matrix comes from Pooled Variance The shared covariance matrix $\Sigma$ is taken as a weighted average of individual covariance matrices,...
Binu Jasim's user avatar
9 votes

Why linear discriminant analysis is sensitive to cross validation (LDA overfit problem)?

Looks like your sample size is not a lot bigger than the dimensionality of the data (feature set size). That can be a problem for LDA and it can overfit. Since it relies on computing the within-class ...
Karolis Koncevičius's user avatar
9 votes
Accepted

Why is my LDA performance a non-monotonic function of the amount of training data?

You discovered an interesting phenomenon. LDA computations rely on inverting within-class scatter matrix $\mathbf S_W$. Usually LDA solution is presented as an eigenvalue decomposition of $\mathbf ...
amoeba's user avatar
  • 105k
8 votes

What is "Discriminant hotelling"?

Discriminant hotelling or discriminant/hotelling is somewhat ambiguous (@amoeba) and seems to be poor word choice (@whuber). An example of that usage appears here. The context in that paper is the ...
Carl's user avatar
  • 13.1k
8 votes
Accepted

What is the difference between PCA and PLS-DA?

Quick answer which I will expand in few days is PLS-DA is a supervised method where you supply the information about each sample's group. PCA, on the other hand, is an unsupervised method which means ...
gunakkoc's user avatar
  • 1,532
8 votes
Accepted

Why linear discriminant analysis is sensitive to cross validation (LDA overfit problem)?

When I even use leave one out (LOOCV) to calculate LDA projection matrix, it is calculated by holding out just one observation. My question is why even in this case the projection matrix ($W$) is so ...
cbeleites unhappy with SX's user avatar
8 votes
Accepted

Difference in scaling of Linear Discriminant Analysis coefficients between manual calculation and R

You have correctly computed $$\mathbf {b} = \mathbf{S}_W^{-1}(\boldsymbol\mu_1-\boldsymbol\mu_2),$$ where $\boldsymbol\mu_i$ are class means and $\mathbf{S}_W$ is the within-class pooled covariance ...
amoeba's user avatar
  • 105k
8 votes
Accepted

Why with two classes, LDA gives only one dimension?

Like @amoeba, I don't understand your difference between Q1 and Q2: in any case, LDA obtains at most $k-1$ dimensions. For $k=2$ that's one dimension, for $k=8$ it would be 7. And of course, if the ...
cbeleites unhappy with SX's user avatar
7 votes

What are "coefficients of linear discriminants" in LDA?

If you multiply each value of LDA1 (the first linear discriminant) by the corresponding elements of the predictor variables and sum them ($-0.6420190\times$...
Tim's user avatar
  • 3,411
7 votes
Accepted

Variable selection using cross-validated PLS model when permutation test shows lack of significance

For the benefit of other readers I will briefly explain what the permutation test is in this context. In this specific example there is a binary dependent variable $y$, a large number of independent ...
amoeba's user avatar
  • 105k
7 votes
Accepted

Autoencoders and Collaborative Filtering: Is one network per training sample really necessary?

[...] each training sample will be provided to its own neural network [...] Actually there is only one autoencoder shared by all samples and updated by all samples. This is convention in neural ...
THN's user avatar
  • 598
7 votes
Accepted

Is classification using linear regression called logistic regression or linear disriminant analysis?

They are both close, but in different ways If you run ordinary least-squares regression with a binary class variable as the outcome (label) variable, you get exactly the 2-class case of linear ...
Thomas Lumley's user avatar
7 votes

Intuition for why LDA is a special case of naive Bayes

Here's my intuition: The LDA classifier assumes that across all classes, the $p$ predictors $\boldsymbol{X}_k$ (for $k=1, \dots,p$) all share some covariance matrix ${\boldsymbol \Sigma}$, but may ...
Wesley's user avatar
  • 580
6 votes

Discriminant Function Analysis in SPSS: Sorts effectively, but Box's M is still 0.000. Is the analysis worthless?

Box's M test is to check if the groups' covariances matrices are the same in the population; this homogeneity of covariances is as assumption of linear discriminant analysis (LDA). The test is quite ...
ttnphns's user avatar
  • 57.5k
6 votes

Three versions of discriminant analysis: differences and how to use them

I find it hard to agree that FDA is LDA for two-classes as @ttnphns suggested. I recommend two very informative and beautiful lectures on this topic by Professor Ali Ghodsi: LDA & QDA. In ...
zyxue's user avatar
  • 1,125
6 votes

Why is there a sharp elbow in my ROC curves?

I agree with John, in that the sharp curve is due to a scarcity of points. Specifically, it appears that you used your model's binary predictions (i.e. 1/0) and the observed labels (i.e. 1/0). Because ...
Tyler's user avatar
  • 61
6 votes

Why is there a sharp elbow in my ROC curves?

Although this question was asked about 3 years ago, I find it useful to answer it here after coming across it and getting puzzled by it for some time. When your ground truth output is 0,1 and your ...
user3545810's user avatar
6 votes
Accepted

Interpretation of the cluster criterion $\operatorname{tr}(S_W^{-1}S_B)$

$S_W^{-1}S_B$ can be interpreted as multivariate signal-to-noise ratio. The between-class scatter matrix $S_B$ tells us how far from each other class means are located. The within-class scatter ...
amoeba's user avatar
  • 105k
6 votes

When would you use PCA rather than LDA in classification?

LDA is used to carve up multidimensional space. PCA is used to collapse multidimensional space. PCA allows the collapsing of hundreds of spatial dimensions into a handful of lower spatial dimensions ...
Brad's user avatar
  • 600
5 votes
Accepted

Difference between GMM classification and QDA

If you're given class labels $c$ and fit a generative model $p(x, c) = p(c) p(x|c)$, and use the conditional distribution $p(c|x)$ for classification, then yes you're essentially performing QDA (the ...
Yibo Yang's user avatar
  • 682
5 votes
Accepted

Differences between linear and canonical discriminant analyses (LDA and CDA)

These are two names for the same thing. Linear discriminant analysis (LDA) is called a lot of different names. I have seen canonical discriminant analysis canonical linear discriminant analysis ...
amoeba's user avatar
  • 105k
5 votes
Accepted

Difference between LDA and PLS-DA?

First of all, PLS-DA means that you perform a PLS regression and then apply a threshold to assign class labels. Now, there are two very different situations where this is done: the underlying ...
cbeleites unhappy with SX's user avatar
5 votes
Accepted

What is the difference between PCA and LDA?

Principal Component Analysis is an unsupervised method, with the resulting latent variables depending only on the values in the supplied X matrix. Linear Discriminant Analysis is a supervised method, ...
jaketmp's user avatar
  • 216
4 votes

GDA and LDA terminology

GDA (Gaussian Dcriminant Analysis) is a general term for both LDA (Linear Discriminant Analysis) and QDA (Quadratic Discriminant Analysis) where the likelihood probability of each observation given ...
east's user avatar
  • 41
4 votes

Comparison of LDA vs KNN time complexity

Time complexity depends on the number of data and features. LDA time complexity is $O\left(Nd^{2}\right)$ if $N>d$, otherwise it's $O\left(d^{3}\right)$ (see this question and answer). It's mostly ...
Firebug's user avatar
  • 19.1k

Only top scored, non community-wiki answers of a minimum length are eligible