# Does PCA followed by LDA make sense, when there is more data available for PCA than for LDA?

This is a question about classification. I am a neuroscience student with little experience of classification methods and I'd be grateful for any advice about the best way to implement a linear classifier (LDA) on this data.

I have a magnetoencephalography dataset, recorded from people as they perform a cognitive task. This has the following properties:

• 306 channels of data, but a preprocessing step has reduced the dimensionality to 64, and then reprojecting the data onto the sensors.

• The data is fine-grained in the time domain (1000Hz)

• The data is chopped up into short segments termed 'trials'. These correspond to the cognitive task the subjects were performing. The trials can differ, for example, in trial type A (of which I will have ~50) the subject may have been asked to pay attention to stimuli on the left hand side of space, and in trial type B (of which I have also 50) to pay attention to stimuli on the right hand side of space).

I want to classify my data at particular timepoints within trials (e.g. 0.5 seconds after subjects are told to attend left/right), training the classifier to discriminate between trials of type A and type B. The feature vector for each observation (i.e. trial) is the vector of instantaneous activity at each sensor at that timepoint. I have more features than trials (306 sensors) so I either need to do feature selection or use regularized LDA (or use something like Hastie's sparse discriminant analysis which seems to me to do both).

This I could do, BUT it seems like it throws away the information about the statistical structure of the data contained in the datapoints NOT from the exact time I am trying to classify. Also, I know the data has a lower dimensionality than 306 - less than 50 components will likely capture the vast majority of the variance in the data.

I was thinking therefore about using a dimensionality reduction step, probably PCA, before passing the reduced-dimensionality data to an unregularized LDA or naive bayes classifier. The idea is that the dimensionality reduction step exploits the fact that I have a large amount of data sampled over time.

And that's where I got confused. PCA projects the data onto orthogonal dimensions, so does it makes sense to then do LDA (which estimates a covariance matrix in order to use information about the correlation between the features)? Or should I do naive bayes following the dimensionalty reduction? In which case, I could probably have just done naive bayes to start with, given that it's not sensitive to the number of features in the way that LDA is.

If anyone can advise on a good approach here, given the particular structure of this data, I'd be very grateful. The key question is: does PCA followed by LDA make sense?

• Without reading the whole question but in reply to the last ask The key question is: does PCA followed by LDA make sense? I'd reply "Often, not". In a sense, the two techniques are alternative and may impede each other. For example, if you retain just the 1st principal component shown on pic1 you won't be able then to separate the classes along it as well as the discriminant shown on pic2 could do it. – ttnphns Feb 27 '13 at 17:10
• @ttnphns: In your linked example the number of points is much higher than the number of dimensions. If the situation is opposite (like in this question), LDA will hopelessly overfit. Reducing the dimensionality with PCA before doing LDA will often act as a sort of regularization, preventing overfitting and dramatically increasing the LDA performance. – amoeba Jan 25 '14 at 14:03
• @amoeba, I agree with the logic you imply. However, in the situation where n<p LDA simply won't work. When n>p but approaches p, better techniques, such as partial squares discriminant analysis, are somewhat better than pre-processing with PCA. – ttnphns Jan 25 '14 at 22:55
• this is an interesting question. In the field chemometrics where one usually has few samples and many highly correlated parameters (spectral wavelengths from e.g. 400 - 2500nanometer in 1nm steps) I occasionally saw a "segmented PCA", where the cut the whole spectrum in spectral regions (segments, often 4) and took the first 3 PCs of each segment as parameters in the LDA. Here is one example: gcl.csrsr.ncu.edu.tw/publication/hyperIntlRSdist.pdf – Jens Aug 14 '14 at 9:51

PCA calculates the eigenvalues that explain most of the variation across the data, in this case it would operate per feature vector and does not take account of class labels. LDA maximizes Fishers discriminant ratio (or Mahalaobis distance), i.e. it maximizes the distance between classes.

If you define the feature vector for each observation (case) as the data at an instantaneous time point, then the temporal components of the data are not relevant. In this case you can apply PCA as pre-processing stage to each feature vector to reduce dimensionality prior to classification.

If however, you define each trial as a 10s epoch or segment around the point of interest, you could then calculate a summary statistic for each sensor across all time samples in the epoch. Each feature in your feature vector would then be a summary of the behaviour of each sensor over the 10s (e.g. mean amplitude across each 10s epoch). You could then apply PCA as pre-processing step to reduce the dimensionality of the feature vector from 306 to a more manageable number.

This second approach assumes that summary statistics calculated over each 10s epoch contains more information relevant to your problem than the instantaneous feature detailed above.

• Thanks @BGreene! I am indeed classifying at instants in time. With 306 features, I can get two-class classification accuracy of about 78% for one comparison of interest, using either Naive Bayes or regularized LDA. However, I'm interested in making the most sensitive possible classifier (without overfitting!) as many effects of interest in the data are likely to be rather subtle. Therefore, I was hoping to use the fact that I have a great deal more data than that from the precise time window of interest to my advantage - e.g. by permitting better estimation of the covariance matrix. – orpetil Mar 4 '13 at 21:54
• If anyone know of techniques which could exploit this property, that'd be great. for example, provided the covariance matrix wasn't too non-stationary, perhaps I could estimate the covariance matrix for the LDA from many time points, and feed this to the classifier for a single time point? the idea with using PCA was partly that since the PCA decomposition could be made over the entire dataset (all time points) that would 'implicitely' calculate the covariance matrix better than it can be estimated across trials at a single time only. But perhaps that doesn't make sense! – orpetil Mar 4 '13 at 21:57

Your problem is most likely overfitting - it is typical when you use small datasets with large number of dimensions. In case of LDA what usually happens that the covariance matrix is biased because of the small sample. In order to avoid it you may try to use regularization. One useful technique is to sphere covariance matrix by adding small terms to the diagonal (also called shrinkage). In this case there is no need to use PCA. You can read about the technique in this paper on single trial EEG analysis.

• Welcome to the site, @btel. There is no need to sign your posts, in fact, we'd prefer you don't. Your avatar is automatically added to every post, along with your user name hyperlinked to your userpage, & other info. Since you're new here, you may want to read our about page & our FAQ, which cover info like this. – gung - Reinstate Monica Feb 28 '13 at 1:39

I am doing the similar thing with neuronal spikes. Some papers I am following did PCA and selected components that explain 90% variance then did LDA, with the main purpose being "avoiding singular matrix", which to some extent makes sense, but can be avoided by "eigen solver" and "shrinkage".

There are 3 reasons why I chose not to do so:

1. very practically, your dataset is temporal, if you perform PCA at each epoch you lose the potential to compare the change across time (or at least harder). It's also harder to track back to individual neurons, but I think in your case it's not that important.
2. as is suggested by another answer, PCA may impair your decoding performance - it is unsupervised.
3. More importantly, I think scientifically it is interesting to see how different regions collectively give rise to a function. I don't think PCA before a classifier will help it.

while I'm typing this answer, I just thought of that people also pool the dimensions of trials and epochs together as sample dimension for PCA. If you perform PCA this way, then at each time point your components are not orthogonal, and you somehow still can compare different epochs.

BGreene's answer is a good one, but I would just add that it might make even more sense to go from PCA to the more closely related LDA classifier, which is Latent Dirichlet Allocation. As the name implies, it's a latent variable technique, like PCA. It's trending highly in the classification literature (I'm using it myself right now) particularly in text mining and unstructured data.

Using PCA then Fischer's LDA is statistically valid ceteris paribus, but it's not necessarily a good idea. Though I and plenty of other people do use linear discriminate in some contexts because it's so quick and easy, in most applications it's not a very statistically efficient or practically effective choice. As a classifier it often performs particularlly poorly; I usually run and rank many classifiers for any algorithm I'm building and linear discriminate is usually at the bottom of the list. It's poor performance may be the reason for needing PCA to reduce dimensional, but I would just choose a single, more appropriate classifier.

• It's not obvious to me how latent Dirichlet allocation (topic modeling) could be applied here. The OPs data are continuous signals not categorical variables, so it's not clear what the "words" or the "documents" would be. – jerad Feb 27 '13 at 23:22