Soft classification after Fisher LDA with MLE

Question

EDIT: any help is appreciated!

I am using Fisher's LDA for an image classification application. Let's say I have labeled data of 20 [100x100]px face images per person with a total of 100 persons, every pixel is a feature so we're in a 10000 dimensional space with 2000 samples.

We have 100 different persons/classes, my algorithm calculates a Transformation matrix with the criteria to maximize between class scatter and minimize within class scatter from all these samples (kind of training). New (test) samples are projected to this Fisher space (c-1=99 dimensional space) using the above mentioned transformation matrix.

Now instead of using a kNN classifier (with an euclidean distance metric) that doesn't take into account class information such as the spread of the data, I would like to write a soft classification algorithm. This algorithm should calculate the probability/likelihood of the test sample belonging to each class. I would then classify based on the highest probability.

Now comes the part with which I have problems with insight, bear with me.

While computing the transformation matrix, I keep 29 eigenvectors (shape 10000,29) with the largest eigenvalues. These eigenpairs are computed via numpy's

np.linalg.eig

How do I compute the probability/likelihood of a sample belonging to each class in Fisher space?

Do I project every sample in facespace into the eigenvector with the largest eigenvalue and fit an univariate normal distribution for every class there (using MLE)? Or do I try to fit a multivariate gaussian distribution in Fisher space for every class? (I'd need a covariance matrix for that? I'd like to work with what I have and use eigenpairs instead) I looked at scipy's scipy.stats.multivariate_normal but I don't have the covariance matrix (I could calculate it from the scatter matrix but it's not like I can use that info in fisher space? The shape of the within class scatter matrix is (N-c) = 2000-100= [1900x1900] matrix)

TL;DR version

How do I compute the probability of a sample belonging to each class in fisher space (after projecting the sample to this space of course)

PS; In the FLDA implementation, PCA is used to go to (N-c) space to solve the singularity problem of the scatter matrix and in that space LDA is performed to go to (C-1) - space with C the number of classes.

Answer 1

Fisher's LDA version maximizes a class-separability criterion, The ratio of the within class scatter to between class scatter. Where samples from the same class should be clustered tight together, and different classes should be as far apart from each other as possible.

Recent approaches to LDA for classification use a Bayesian framework (like sklearn's implementation);

LDA can be used for dimensionality reduction AND for classification.

In my original question I was wondering how to compute class probabilities in fisher space, one way to do this is computing the means and covariances of every class and fitting a multivariate distribution to each class, this enables us to get class probabilties of a sample belonging to each class after projection to fisherspace.

Now for the relation with the eigenpairs, I'm still uncertain about this, clarification about the relation with these would be awesome.

But I have completely switched over to the bayesian framework so there's no rush.

I want to refer to this post and encourage you to study the answers posted by that user and this user regarding LDA, they're excellent.