Return to Answer

All covariance matrices are positive semi-definite. If the covariance matrix isn't positive definite, that implies that it is singular.

In this case, either or both of these is true:

1. A singular covariance matrix occurs when the source data contains linearly dependent features. You need to remove linearly dependent features from your source data. The gold-standard way to verify the rank of your feature matrix is to perform SVD. If the number of positive singular values is smaller than the number of features, it's rank-deficient.

2. Your matrix is numerically singular. This can happen even if all of the columns are linearly independent. You can verify this by checking the condition number of the matrix. You state that you have the case that $(x-y)^\top\Sigma^{-1}(x-y)<0$; if the results are only "slightly" negative, then this is a result of accumulated numerical error. (If the results are very negative, there's a large programming error somewhere.)

You could try modifying $\Sigma$ to enforce non-singularity. A common way to do this is to compute $\Sigma=USV^\top$ the singular value decomposition of $\Sigma$, and then "adjust" the singular values $S_{ii}$ so that they are pinned to some small, positive multiple of the largest singular value: $\tilde{S_{ii}}=\max(S_{ii}, \epsilon \max{S_{ii}})$. A reasonable choice is $\epsilon=10^{-6}$ for double-floats. This will distort your covariance matrix a little bit, but alleviate your negative distance problem.

All covariance matrices are positive semi-definite. If the covariance matrix isn't positive definite, that implies that it is singular.

In this case, either or both of these is true:

1. A singular covariance matrix occurs when the source data contains linearly dependent features. You need to remove linearly dependent features from your source data. The gold-standard way to verify the rank of your feature matrix is to perform SVD. If the number of positive singular values is smaller than the number of features, it's rank-deficient.

2. Your matrix is numerically singular. This can happen even if all of the columns are linearly independent. You can verify this by checking the condition number of the matrix. You state that you have the case that $(x-y)^\top\Sigma^{-1}(x-y)<0$; if the results are only "slightly" negative, then this is a result of accumulated numerical error. (If the results are very negative, there's a large programming error somewhere.)

You could try modifying $\Sigma$ to enforce non-singularity. A common way to do this is to compute $\Sigma=USV^\top$ the singular value decomposition of $\Sigma$, and then "adjust" the singular values $S_{ii}$ so that they are pinned to some small, positive multiple of the largest singular value: $\tilde{S_{ii}}=\max\left\{S_{ii}, \epsilon \max{S_{ii}}\right\}$. A reasonable choice is $\epsilon=10^{-6}$ for double-floats. This will distort your covariance matrix a little bit, but alleviate your negative distance problem.

All covariance matrices are positive semi-definite. If the covariance matrix isn't positive definite, that implies that it is singular.

In this case, either or both of these is true:

1. A singular covariance matrix occurs when the source data contains linearly dependent features. You need to remove linearly dependent features from your source data. The gold-standard way to verify the rank of your feature matrix is to perform SVD. If the number of positive singular values is smaller than the number of features, it's rank-deficient.

2. Your matrix is numerically singular. This can happen even if all of the columns are linearly independent. You can verify this by checking the condition number of the matrix; if it's on the order of $10^{12}$ or larger, it's poorly conditioned. You state that you have the case that $(x-y)^\top\Sigma^{-1}(x-y)<0$; if the results are only "slightly" negative, then this is a result of accumulated numerical error. (If the results are very negative, there's a large programming error somewhere.)

You could try modifying $\Sigma$ to enforce non-singularity. A common way to do this is to compute $\Sigma=USV^\top$ the singular value decomposition of $\Sigma$, and then "adjust" the singular values $S_{ii}$ so that they are pinned to some small, positive multiple of the largest singular value: $\tilde{S_{ii}}=\max(S_{ii}, \epsilon \max{S_{ii}})$. A reasonable choice is $\epsilon=10^{-6}$ for double-floats. This will distort your covariance matrix a little bit, but alleviate your negative distance problem.

All covariance matrices are positive semi-definite. If the covariance matrix isn't positive definite, that implies that it is singular.

In this case, either or both of these is true:

1. A singular covariance matrix occurs when the source data contains linearly dependent features. You need to remove linearly dependent features from your source data. The gold-standard way to verify the rank of your feature matrix is to perform SVD. If the number of positive singular values is smaller than the number of features, it's rank-deficient.

2. Your matrix is numerically singular. This can happen even if all of the columns are linearly independent. You can verify this by checking the condition number of the matrix. You state that you have the case that $(x-y)^\top\Sigma^{-1}(x-y)<0$; if the results are only "slightly" negative, then this is a result of accumulated numerical error. (If the results are very negative, there's a large programming error somewhere.)

You could try modifying $\Sigma$ to enforce non-singularity. A common way to do this is to compute $\Sigma=USV^\top$ the singular value decomposition of $\Sigma$, and then "adjust" the singular values $S_{ii}$ so that they are pinned to some small, positive multiple of the largest singular value: $\tilde{S_{ii}}=\max(S_{ii}, \epsilon \max{S_{ii}})$. A reasonable choice is $\epsilon=10^{-6}$ for double-floats. This will distort your covariance matrix a little bit, but alleviate your negative distance problem.

All covariance matrices are positive semi-definite. If the covariance matrix isn't positive definite, that implies that it is singular. A singular covariance matrix occurs when the source data contains linearly dependent features. You need to remove linearly dependent features from your source data.

All covariance matrices are positive semi-definite. If the covariance matrix isn't positive definite, that implies that it is singular.

In this case, either or both of these is true:

1. A singular covariance matrix occurs when the source data contains linearly dependent features. You need to remove linearly dependent features from your source data. The gold-standard way to verify the rank of your feature matrix is to perform SVD. If the number of positive singular values is smaller than the number of features, it's rank-deficient.

2. Your matrix is numerically singular. This can happen even if all of the columns are linearly independent. You can verify this by checking the condition number of the matrix; if it's on the order of $10^{12}$ or larger, it's poorly conditioned. You state that you have the case that $(x-y)^\top\Sigma^{-1}(x-y)<0$; if the results are only "slightly" negative, then this is a result of accumulated numerical error. (If the results are very negative, there's a large programming error somewhere.)

You could try modifying $\Sigma$ to enforce non-singularity. A common way to do this is to compute $\Sigma=USV^\top$ the singular value decomposition of $\Sigma$, and then "adjust" the singular values $S_{ii}$ so that they are pinned to some small, positive multiple of the largest singular value: $\tilde{S_{ii}}=\max(S_{ii}, \epsilon \max{S_{ii}})$. A reasonable choice is $\epsilon=10^{-6}$ for double-floats. This will distort your covariance matrix a little bit, but alleviate your negative distance problem.