8

The condition number of a correlation matrix is not of great interest in its own right. It comes into its own when that matrix gives the coefficients of a set of linear equations, as happens for multiple linear regression using standardized regressors. Belsley, Kuh, and Welsh--who were among the first to point out and systematically exploit the relevance ...


3

When the columns of $X$ are standardized, the condition number Definitions By definition, the condition number of $X$ is obtained by considering the effect of $X$ (qua linear transformation) on all possible nonzero vectors $e.$ If we (for the purposes of this thread only) define the "stretch" of $X$ at $e$ to be the amount by which $X$ changes its length, ...


3

This UCLA page covers this issue in the context of logistic regression. But the phrasing of this question suggests some misunderstanding of multicollinearity and condition numbers. The condition number and multicollinearity are functions of the design matrix of independent variable values. They bear no relation to the dependent variable or to the type of ...


2

If you cannot use your original matrix $K$, you might want to use a $K^{\prime}$ that is the closest you can get in terms of Frobenius norm. To do that find the smallest eigenvalues and set them to a higher number positive number. This will give you a new P(S)D matrix that is closer than any other matrix in Frobenius norm, see N.J. Higham 1988 for details. ...


2

It's a simple fact and easily verified that multiplying one column of X by a scaling factor of say 1,000,000 can dramatically change the condition number of X. Your intuition about the effect of scaling is wrong.


2

what you are looking for is ridge regression- basically it adds a regularisation term $\alpha \|w\|^2 $, where $w$ is the coefficient vector, to the mean squared error (see eg wiki Tikhonov regularisation). This term penalises large weights (and so rescaling now has some benefit): in particular it penalises solutions with large opposing positive and negative ...


2

Compare: norm(m, type='F') With: norm(m, type='2') At first I thought (as you probably did) that with the 2-norm they meant the 2-norm when considering the matrix as a vector. However I found it odd that they used 2 notations in different places ('F' for Frobenius and '2' for 2-norm). Then I saw that the norm function has both as options (see:...


2

Combining several questions/comments, I believe I have the answer. This is just a scaling problem. The condition number is the ratio of the largest eigenvalue in the design matrix to the smallest. This large condition number results from scaling rather than from multicollinearity. If we have just one variable with units in the thousands (ie, a large ...


1

I think you're confused with the usage of the word input. Naturally, in deep learning context we mean a vector $x$ by input. However, in this passage it is the matrix $\textbf A$ that is referred to as input. Think of the matrix $\textbf A$ not as a constant predetermined matrix, but as of a parameter that is estimated. Maybe you estimate $\textbf A$ from ...


1

When we estimate GLM/Logit/... by IRLS, then is based on weighted least squares. The analogue to the condition number of the design matrix would be the condition number of the weighted design matrix which does depend on the distributional assumption for the response. In other nonlinear models, e.g nonlinear least squares, we could look at the condition ...


1

The rbf kernel is extremely smooth and despite it being possibly the most popular choice of kernel it is rarely the best choice (not just IMO but this also seems to be the opinion of most well know researchers in the GP community - watch talks from GPSS for example). However, no matter what kernel you use there is always a chance that your covariance ...


1

Let $A$ be the matrix that R claims is computationally singular. For the purpose of this answer lets assume that it is really mathematically non-singular but close to singular (otherwise lowering the tolerance will do nothing). Let $\epsilon$ be the tolerance in use. For condition numbers see: https://en.wikipedia.org/wiki/Condition_number. The condition ...


1

Super high condition number would mean that some variables are highly correlated. 70 is not that big of a condition number to me. High or low condition number doesn't mean that one correlation matrix is "better" than the other. All it means is that variables are more correlated or less. Whether it's good or not depends on the application. UPDATE: I'm ...


Only top voted, non community-wiki answers of a minimum length are eligible