# Tag Info

8

As AdamO suggests in the above comment, you can't really do better than read Chapter 4 of The Elements of Statistical Learning (which I will call HTF) which compares LDA with other linear classification methods, giving many examples, and also discusses the use of LDA as a dimension-reduction technique in the vein of PCA which, as ttnphns points out, is ...

7

If you mean LDA I would say the name, linear discriminant analysis, can be explained historically dating back at least to Fisher's paper from 1936, which, to the best of my knowledge, precedes the current terminology and distinction in machine learning between a discriminative and a generative model. Not that Fisher called it linear discriminant analysis ...

7

Discriminants are the axes and the latent variables which differentiate the classes most strongly. Number of possible discriminants is $min(k-1,p)$. For example, with k=3 classes in p=2 dimensional space there can exist at most 2 discriminants such as on the graph below. (Note that discriminants are not necessarily orthogonal as axes drawn in the original ...

7

The obvious "common sense" way to resolving your problem is to Get the conclusion using the full data set. i.e. what results will you declare ignoring intermediate calculations? Get the conclusion using the data set with said "outliers" removed. i.e. what results will you declare ignoring intermediate calculations? Compare step 2 with step 1 If there is ...

6

While "The Elements of Statistical Learning" is a brilliant book, it requires a relatively high level of knowledge to get the most from it. There are many other resources on the web to help you to understand the topics in the book. Lets take a very simple example of linear discriminant analysis where you want to group a set of two dimensional data points ...

6

Classification in LDA goes as follows (Bayes' approach). [About extraction of discriminants one might look here.] According to Bayes theorem, the sought-for probability that we're dealing with class $k$ while observing currently point $x$ is $P(k|x) = P(k)*P(x|k) / P(x)$, where $P(k)$ – unconditional (background) probability of class $k$; $P(x)$ – ...

5

I don't know of any papers on this. I've used this approach, for descriptive purposes. DFA provides a nice way to summarize group differences and dimensionality with respect to the original variables. One might more easily just profile the groups on the original variables, however, this loses the inherently multivariate nature of the clustering problem. ...

5

I think that Multi-class LDA classifier always (well, in most practical tasks) out-performs 2 class LDA. And I will try to describe why. Have a look at the example dataset: You have three classes here. And let's say you want to build one-vs-other classifier with LDA for the blue class. The estimated mean for class "blue" is zero, but the estimated mean ...

5

Regarding the "Real Values" The "Real Values" are better called "confidences" or (from my pov the most common term) "scores". Such scores are often normalized so that they sum up to 1 for all classes. They represent a measure how, well, confident the model is that the presented example belongs to a certain class. They are highly dependent on the general ...

5

I take it that the question is about LDA and linear (not logistic) regression. There is a considerable and meaningful relation between linear regression and linear discriminant analysis. In case the DV consisting just of 2 groups the two analyses are actually identical. Despite that computations are different and the results - regression and discriminant ...

4

Here is a reference to one of Efron's papers: http://www.jstor.org/discover/10.2307/2285453?uid=3739864&uid=2129&uid=2&uid=70&uid=4&uid=3739256&sid=47699111262887 Here is an abstract that mentions O'Neill's papers related to his Ph D dissertation: Comparison of generative and discriminative classifiers is an ever-lasting topic. ...

4

The idea to get a classifier for new cases after the classes had been identified in a cluster analysis is in itself natural and sane. Discriminant analysis could be such a tool, especially after K-means clustering. However, the DA's classifier will work well if assumptions of DA hold, such as (1) continuous data with approximately multivariate normal ...

4

You can try the lda() function in package MASS which should come with all R installations as it is a Recommended package. Load the package via: library("MASS") Then read the help page for lda() ?lda paying particular attention to the example section. When back at the prompt you can run the examples using example(lda)

4

"Fisher's Discriminant Analysis" is simply LDA in a situation of 2 classes. When there is only 2 classes computations by hand are feasible and the analysis is directly related to Multiple Regression. LDA is the direct extension of Fisher's idea on situation of any number of classes and uses matrix algebra devices (such as eigendecomposition) to compute it. ...

3

You can get rid of some by looking for pairs that are very highly correlated and randomly deleting one of the pair. Then you can look at partial least squares, and pick variables that are important in the PLS solution. I did this with a similar problem and it worked pretty well (that is, the resulting discriminant function did pretty well)

3

OK, since nobody answered I think that, after some experimentation, I can do it myself. Following discriminant analysis guidelines, let T be the whole cloud's (data X, of 2 variables) sscp matrix (of deviations from cloud's centre), and let W be the pooled within-cluster sscp matrix (of deviations from a cluster centre). B=T-W is the between-cluster sscp ...

3

For many reasons, classification is not a good goal for most problems; prediction is. Logistic regression (LR) is a more direct probability model to use for prediction, with fewer assumptions. Linear discriminant analysis (LDA) assumes that X has a multivariate normal distribution given Y. Using Bayes' rule to get Prob(Y|X) you get a logistic model. So ...

3

I have 3 points to say. First, what is "good" documentation for you? There is a lot of pages on DA and downloadable books on multivariate statistics (where DA is discussed) in the Internet. Some texts are superficial and easy, some are more sophisticated. Second, because you have just 2 groups your DA will be virtually equivalent to the multiple linear ...

3

As you say, LDA is supervised. How does your supervisor define "learning"? But yes, usually it is counted as supervised learning. Reference, e.g. first 2 pages of The Elements of Statistical Learning You can use LDA models for prediction of new cases. (I'd say that implies that something has been learned However, you can also put emphasis on the ...

3

I believe (and someone will correct me if I'm wrong) that "classification function" can be used much more broadly - for any function that does classification, whether from logistic regression, discriminant analysis or whatever. "Discriminant function" is restricted to those from discriminant analysis. At least, that's how I've seen them used.

2

The main difference between the coefficients and the correlations (elements of structure matrix) is not that these are less stable than those. Coefficient shows partial (i.e. unique) contribution of the variable to the discriminant function score, it is like regression coefficient. Correlation shows omnibus (i.e. unique + shared with other variables) ...

2

I'm afraid that your expectance that the data cluster in space hierarchically, as classes and subclasses, is an illusion. If it had been true then the 9-class LDA could have not concealed the fact that the subclasses are closer to each other than the classes. The presumption must be that there are 9 classes, and if the analysis trying to distinguish them all ...

2

There are several nonparametric methods for discriminant analysis: rank methods, classifiers based on robust estimators of location and scale (M-estimators or MCD-estimators, for instance), and so on. Have you decided what kind of method that you want to use? As for the impact of non-normality on LDA and QDA, I'd recommend that you have a look at the ...

2

The first question is whether you already know which frog belongs to which morphotype If you do know, and your goal is to use these frogs to better analyze how the morphotypes vary on these variables, then you want discriminant analysis. This might enable later investigators to accurately place frogs into morphotypes based on these variables. If you do not ...

2

I think you are very confused about what principal component analysis is and the Chang paper has added to your confusion. First we have multivariate data in say k dimensions the principal components are a particular transformation of the coordinates such that the first principal component exhibits the largest variation in the data for any one component. ...

2

The normality assumption is only a criteria for optimality in the sense that if the class conditional densities for the feature vector are normally distributed with known mean vectors and known equal covariance matrices the linear discriminant function is the Bayes rule (optimal) when the two types of errors have equal loss weights. The actual Fisher linear ...

2

Essentially these two constraints basically require the training data to be correctly classified, and at least a certain distance from the decision threshold 0. The hyperplane that fulfils these constraints with the smallest norm of the weights will have the maximal margin. The value $\pm 1$ is essentially arbitrary, you could replace it with $\pm$ any ...

2

wx + b = -1 and wx + b = 1; These equations represent two parallel hyperplanes that are formed based on samples class (-1, +1). These two hyperplanes are used to optimize the distance between classes and to get optimal hyperplane. For optimization problem Lagrange multipliers are used. You can find brief description on resources listed below. Online ...

2

Here is a short tale about Linear Discriminant Analysis (LDA) as a reply to the question. When we have one variable and $k$ groups to discriminate by it, this is ANOVA. The discrimination power of the variable is $SS_{between groups} / SS_{within groups}$, or $B/W$. When we have $p$ variables, this is MANOVA. If the variables are uncorrelated neither in ...

2

PCA calculates the eigenvalues that explain most of the variation across the data, in this case it would operate per feature vector and does not take account of class labels. LDA maximizes Fishers discriminant ratio (or Mahalaobis distance), i.e. it maximizes the distance between classes. If you define the feature vector for each observation (case) as the ...

Only top voted, non community-wiki answers of a minimum length are eligible