4
$\begingroup$

I have a $n \times p$ matrix $A$ where $n$ is the number of observables and $p$ is the number of observations. $n \gg p$

In my code, I have done $[E,V] \,=\, eig(A)$ and doing a least squares operation on the eigenvector matrix $V$, which is $p \times p$. This is using the mldivide operator in MATLAB, such that X = V\b;.

My data is fairly huge, and I am getting warnings from MATLAB that $V$ is rank deficient, and thus the condition number is too high to get a reasonable answer.

Are there any standard techniques/tools out there to overcome this? I have tried shifting and scaling $V$, and reading threads here on this topic, but no avail.

The mldivide works for smaller datasets, where $p \sim O(10^1)$. However, when I go to $O(10^2)$ and beyond, I face rank deficiency issues. I would welcome suggestions.

A few useful points about my data (I can provide more details if required):

  1. This data is coming from an experiment of an extremely chaotic physical phenomenon, and therefore is noisy.
  2. The $V$ matrix consists of eigenvectors in complex conjugates, and there is a lot of variation in every eigenvector (order of magnitude ranges from $10^{-14}$ to $10^{-1})$.
$\endgroup$
2
  • $\begingroup$ Are you sure it's a numerical problem, not the real issue with rank deficiency? $\endgroup$ – Aksakal Jun 4 '15 at 13:19
  • $\begingroup$ I have solved that problem since its been a year. But I still have the same question for some of my datasets, especially econometric statistical data like weekly gas prices over a decade etc. I tried mean-subtraction, normalizing by 1, scaling etc. but the rank deficiency is unchanged. So I am looking for general pointers how to approach this. $\endgroup$ – atmaere Jun 4 '15 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.