Throw out data from covariance/scatter matrix calculation when there is only one observation of a class?

Question

If you only have one example of a class would it be better to throw out the data from the beginning (prior to covariance matrix calculation and feature reduction) and not consider it at all?

I am using Uncorrelated Linear Discriminant Analysis(ULDA) for feature reduction and am currently using all of my data to calculate scatter matrices. However, when I get to the calculation of a pooled covariance matrix when I am generating my model, the covariance of the class a single observation cannot be calculated and yields a matrix of NaNs.

So for example, if there are 100 features and 10 classes, when we do ULDA we will have around 10 features left to represent the data. if two of those 10 classes only have one observation, those can't be used to calculate the covariance matrix. Would it make more sense to throw them out at the beginning, meaning after ULDA there would be around 8 features to represent the data? Or do we need the 10 features to represent the data but then when we calculate the covariance, just add a covariance matrix that is all 0s?

My question is conceptually what would it mean to leave out the data from the scatter matrix calculation and what would it mean to define the covariance matrix as all zeros for the case where there is only one observation of the class?

Answer 1

It's not possible to estimate a covariance matrix from a single observation. So, for LDA, you would need to omit classes with only one data point. In your example, this would leave 8 classes, LDA would yield 8 features, and you wouldn't be able to say anything about the remaining 2 classes.

The zero matrix isn't a valid covariance matrix. But, the limit of a Gaussian PDF as the covariance matrix approaches zero is a Dirac delta function (i.e. infinitely thin, infinitely tall spike) centered at the mean.