I've really hit the wall here and need help with direction :).
I am trying to use mvnpdf as part of basic EM algorithm but the covariance matrix of data seems to be not positive definite. There are many discussion on this topic and I think I have followed all the major one’s without success. Also I want to understand how this problem should be dealt with real world data.
I have face-image data (2800 images) where each image is 60x60x3 = 10800 variable vector. The pixel data is normalised between [0,1].
Following code Ive used to calculate covariance
% x is the 2800x10800 data matrix % I = 2800 dataset_mean = sum(x,1)./I; sig = zeros (dimensionality, dimensionality); for i = 1 : I mat = x (i,:) - dataset_mean; mat = mat' * mat; sig = sig + mat; end sig = sig ./ I;
Following points I have thought,
[~,p] = chol(sig) gives me ~2800 hence because 'p' is positive it is definitely not Positive Definite. No idea why it always close to number of data points/rows.
Tried, [R,err]=cholcov(s, 0); because I've read that it is the method used inside mvnpdf and I get err +ve indicating that it is Positive Definite
Eigenvector has a lot of negative values but all of them are all very close to 0 (i.e. range of e-12). What I understand that it is numerical residual error of some sort hence I am using my own code to calculate covariance (above ) rather than an in-built function.
The determinant of covariance matrix (sig) is also 0, as I understand that it means that the variable is the data matrix and too correlated. Am I right here?
From point 3 and 4 I thought this is a case of Multivariate dependencies, where several variables together perfectly predict another variable.
Follow things I have tried,
- To fix eigenvector problem -
1.Added small amount on the diagonal of covariance matrix.
2.Make all near zero -ve eigenvector 0 and reconstruct covariance matrix Ref : http://comisef.wikidot.com/tutorial:repairingcorrelation
% compute eigenvectors/-values [V,D] = eig(C); % replace negative eigenvalues by zero D = max(D, 0); % reconstruct correlation matrix BB = V * D * V'; % rescale correlation matrix T = 1 ./ sqrt(diag(BB)); TT = T * T'; C = BB .* TT;
- Used SVD to reconstruct covariance matrix
- nearestSPD by John D’Errico Ref : https://in.mathworks.com/matlabcentral/fileexchange/42885-nearestspd
All above three methods gave Positive Definite matrix which I confirmed by using, [R,err]=cholcov(s, 0); but determinant was still 0.
Additionally, mvnpdf gave me Inf value and I think it is because determinant of covariance matrix was 0 and it messed up the inverse calculation for covariance inside mvnpdf.
So, my question is that what do I need to do next, am I missing something? and I am sure this problem exists with real world data hence how is it tackled ?