13
votes
Accepted
Gaussian Discriminant Analysis and sigmoid function
I'm not sure what it means that GDA "leads to the sigmoid hypothesis" but perhaps it refers to the relationship between GDA (aka Linear Discriminant Analysis, LDA) and logistic regression.
Before ...
- 4,787
12
votes
When would you use PCA rather than LDA in classification?
You are missing something deeper: PCA isn't a classification method.
PCA in machine learning is treated as a feature engineering method. When you apply PCA to your data you are guaranteeing there'll ...
- 19.4k
10
votes
When would you use PCA rather than LDA in classification?
Whereas the previos answer by Firebug is correct, I want add another perspective:
Unsupervised vs. supervised learning:
LDA is very useful to find dimensions which aim at seperating cluster, thus ...
- 3,491
9
votes
What is a Gaussian Discriminant Analysis (GDA)?
I think Andrew Ng's notes on GDA (https://web.archive.org/web/20200103035702/http://cs229.stanford.edu/notes/cs229-notes2.pdf) are the best explanation I have seen of the concept, but I want to "...
- 231
9
votes
Why linear discriminant analysis is sensitive to cross validation (LDA overfit problem)?
Looks like your sample size is not a lot bigger than the dimensionality of the data (feature set size). That can be a problem for LDA and it can overfit. Since it relies on computing the within-class ...
- 5,104
9
votes
Accepted
Why is my LDA performance a non-monotonic function of the amount of training data?
You discovered an interesting phenomenon.
LDA computations rely on inverting within-class scatter matrix $\mathbf S_W$. Usually LDA solution is presented as an eigenvalue decomposition of $\mathbf ...
- 106k
8
votes
What is "Discriminant hotelling"?
Discriminant hotelling or discriminant/hotelling is somewhat ambiguous (@amoeba) and seems to be poor word choice (@whuber). An example of that usage appears here. The context in that paper is the ...
- 13.2k
8
votes
Accepted
What is the difference between PCA and PLS-DA?
Quick answer which I will expand in few days is
PLS-DA is a supervised method where you supply the information about each sample's group. PCA, on the other hand, is an unsupervised method which means ...
- 1,532
8
votes
Accepted
Why linear discriminant analysis is sensitive to cross validation (LDA overfit problem)?
When I even use leave one out (LOOCV) to calculate LDA projection matrix, it is calculated by holding out just one observation. My question is why even in this case the projection matrix ($W$) is so ...
- 39.4k
8
votes
Accepted
Difference in scaling of Linear Discriminant Analysis coefficients between manual calculation and R
You have correctly computed $$\mathbf {b} = \mathbf{S}_W^{-1}(\boldsymbol\mu_1-\boldsymbol\mu_2),$$ where $\boldsymbol\mu_i$ are class means and $\mathbf{S}_W$ is the within-class pooled covariance ...
- 106k
8
votes
Accepted
Why with two classes, LDA gives only one dimension?
Like @amoeba, I don't understand your difference between Q1 and Q2: in any case, LDA obtains at most $k-1$ dimensions. For $k=2$ that's one dimension, for $k=8$ it would be 7. And of course, if the ...
- 39.4k
7
votes
What are "coefficients of linear discriminants" in LDA?
If you multiply each value of LDA1 (the first linear discriminant) by the corresponding elements of the predictor variables and sum them ($-0.6420190\times$...
- 3,401
7
votes
Accepted
Autoencoders and Collaborative Filtering: Is one network per training sample really necessary?
[...] each training sample will be provided to its own neural network [...]
Actually there is only one autoencoder shared by all samples and updated by all samples. This is convention in neural ...
- 598
7
votes
Accepted
Is classification using linear regression called logistic regression or linear disriminant analysis?
They are both close, but in different ways
If you run ordinary least-squares regression with a binary class variable as the outcome (label) variable, you get exactly the 2-class case of linear ...
- 43.7k
7
votes
Intuition for why LDA is a special case of naive Bayes
Here's my intuition:
The LDA classifier assumes that across all classes, the $p$ predictors $\boldsymbol{X}_k$ (for $k=1, \dots,p$) all share some covariance matrix ${\boldsymbol \Sigma}$, but may ...
- 580
6
votes
Three versions of discriminant analysis: differences and how to use them
I find it hard to agree that FDA is LDA for two-classes as @ttnphns suggested.
I recommend two very informative and beautiful lectures on this topic by Professor Ali Ghodsi:
LDA & QDA. In ...
- 1,176
6
votes
Why is there a sharp elbow in my ROC curves?
I agree with John, in that the sharp curve is due to a scarcity of points. Specifically, it appears that you used your model's binary predictions (i.e. 1/0) and the observed labels (i.e. 1/0). Because ...
- 61
6
votes
Discriminant Function Analysis in SPSS: Sorts effectively, but Box's M is still 0.000. Is the analysis worthless?
Box's M test is to check if the groups' covariances matrices are the same in the population; this homogeneity of covariances is as assumption of linear discriminant analysis (LDA). The test is quite ...
- 58.4k
6
votes
When would you use PCA rather than LDA in classification?
LDA is used to carve up multidimensional space.
PCA is used to collapse multidimensional space.
PCA allows the collapsing of hundreds of spatial dimensions into a handful of lower spatial dimensions ...
- 600
6
votes
Accepted
Difference between LDA and PLS-DA?
First of all, PLS-DA means that you perform a PLS regression and then apply a threshold to assign class labels.
Now, there are two very different situations where this is done:
the underlying ...
- 39.4k
5
votes
Accepted
Difference between GMM classification and QDA
If you're given class labels $c$ and fit a generative model $p(x, c) = p(c) p(x|c)$, and use the conditional distribution $p(c|x)$ for classification, then yes you're essentially performing QDA (the ...
- 692
5
votes
Accepted
Differences between linear and canonical discriminant analyses (LDA and CDA)
These are two names for the same thing.
Linear discriminant analysis (LDA) is called a lot of different names. I have seen
canonical discriminant analysis
canonical linear discriminant analysis
...
- 106k
5
votes
Accepted
What is the difference between PCA and LDA?
Principal Component Analysis is an unsupervised method, with the resulting latent variables depending only on the values in the supplied X matrix.
Linear Discriminant Analysis is a supervised method, ...
- 216
4
votes
GDA and LDA terminology
GDA (Gaussian Dcriminant Analysis) is a general term for both LDA (Linear Discriminant Analysis) and QDA (Quadratic Discriminant Analysis) where the likelihood probability of each observation given ...
- 41
4
votes
How do I calculate the structure-matrix and standardized discriminant coefficients from a Linear Discriminant model in R?
It's a bit weird to answer your own question, I suppose, but hopefully this will be a useful bit of information for folks looking to do this in the future. I used the great package candisc, which has ...
4
votes
how to calculate observation within and between groups variances of PCA scores
You want the Procrustes variance, also called 'morphological disparity' in geometric morphometrics. You can compute Procrustes variances and test for significant differences among groups using the <...
- 1,086
4
votes
Accepted
Why is within class scatter matrix in LDA singular?
This is a common problem with LDA, as the number of measurements of each sample (i.e., the dimensionality of each data vector) exceeds the number of samples in each class. In this case, the ...
4
votes
Accepted
Is a linear discriminant function actually "linear"?
Affinity is clearly enough for (4.12) to hold. Since
$$\hat{\mathbf{x}}=\lambda\mathbf{x}_A + (1-\lambda)\mathbf{x}_B $$
then, multiplying LHS & RHS by $\mathbf{w}^T_k$ and adding $w_{k0}$, we ...
- 18.3k
4
votes
Accepted
The discriminant function in linear discriminant analysis
it can be shown that this is equivalent to assigning the observation to the class for which the below equation is largest
Deriving the discriminant function for LDA
For LDA we assume that the random ...
- 9,866
4
votes
Accepted
LDA dimensionality reduction
In LDA, unlike PCA, the maximum number of features after dimensionality reduction is $c - 1$, where $c$ is the number of classes. Since you have 5 classes, the maximum number of features is 4. You can ...
- 442
Only top scored, non community-wiki answers of a minimum length are eligible