Questions tagged [condition-number]
The condition number (also known as condition index) is a diagnostic tool for collinearity in regression models. A regression model has as many condition numbers as it has independent variables. Each is defined as the square root of the ratio of the largest eigenvalue to the eigenvalue for the respective variable. Condition numbers over 30 are considered to be signs of problematic collinearity.
30
questions
9
votes
1
answer
177
views
"Unbiased" (at least ballpark) Estimate of Condition Number of True Covariance Matrix being Estimated & other Symmetric Matrices (e.g.,Hessian)
Are there any known ways of getting an unbiased estimate of the condition number of the true covariance matrix being estimated, or at least correct within a small number of orders of magnitude? For ...
- 12.8k
0
votes
0
answers
26
views
Does adding more covariates always increase the condition number of the Hessian, and can you have a high-condition number but higher log-likelihood?
Does adding more covariates always increase the condition number of the Hessian, and can you have a high-condition number but higher log-likelihood/more optimal model?
Should we ever not use the model ...
user318514
3
votes
0
answers
128
views
Does the correlation matrix always have a smaller condition number than the covariance matrix?
In my experience and the experience of others (for example: https://stats.stackexchange.com/a/287737/193216), a covariance matrix
$\textrm{COV}=\langle {\bf x}_t{\bf x}_t'\rangle$
has a larger ...
- 195
8
votes
1
answer
101
views
Explaining conditioning number in statistics to non-statisticians
I work these days as a statistician and a lot of what I do is evaluating design of experiments; I started this job less than a year ago, after getting a PhD in mathematical statistics. I remember once ...
- 1,233
1
vote
1
answer
744
views
Computing condition number for OLS
In https://www.statsmodels.org/0.8.0/examples/notebooks/generated/ols.html, if you scroll down to the section "Multicollinearity" (bolded), it shows that the condition number of $X$ is $4....
- 1,057
0
votes
1
answer
195
views
Variable Selection : Removing Linear Dependency by SVD using the Condition number and then eliminating the variable causing multicollinearity
I am trying to perform regression with over 5000 feature variables(X) and I would like to eliminate multicollinearity. Incremental VIF computation is expensive. Incremental PCA works but I might lose ...
- 23
3
votes
1
answer
642
views
Why does the condition number of the covariance matrix explode as number of variables increases?
From asset returns of $N$ stocks, the symmetric covariance matrix sized $N\times N$ is constructed, which treats the asset returns as variables.
When the number of variables $N$ is fairly low like $N=...
- 3,403
0
votes
0
answers
190
views
What is the meaning of the regressor characteristic root?
As described by Greene's Econometric Analysis (7th Edition), the regressor matrix's condition number measures how singular the matrix is. Therefore, the condition number is a measure of ...
- 111
1
vote
0
answers
27
views
Multicollinearity negating $\beta$
In Regression coefficients that flip sign after including other predictors, ars's answer states that "Basically, if your variables are positively correlated, then the coefficients will be ...
- 1,057
1
vote
0
answers
61
views
Comparing numerical stability and computing bounds on the condition number of learned weights
I have an empirical risk minimization problem with two equivalent losses that solves it, $f_1(x; \theta_1)$ and $f_2(x ; \theta_2)$, where $x$ is the data and $\theta$ are the model parameters (in ...
- 13
0
votes
0
answers
96
views
QR decomposition used for estimating condition numbers
I have read that the QR decomposition is often used to estimate the condition number of a matrix but I don't understand... what is the benefit of using the QR decomposition for this? Is it purely ...
- 618
1
vote
1
answer
618
views
Kappa condition number in R
I have read that the kappa function in R does not always explicitly calculate the condition number of a matrix, but rather, estimates the 2 norm of a matrix or a QR ...
- 618
3
votes
1
answer
666
views
How are the condition numbers of a design matrix and its correlation matrix related?
Given a design matrix $X$ for a linear regression model, what is the relationship between the condition number of $X$ and its correlation matrix $R$?
I would be interested in the case of a centered ...
- 229
4
votes
1
answer
3k
views
Why does matrix condition number change drastically when a constant is added?
If I create a regression model design matrix with 3 uncorrelated variables, I get a small condition number as expected. MWE:
...
- 343
5
votes
0
answers
883
views
Choosing the basis functions in a linear regression
I have two random variables $X$ and $Y$ and I'm trying to model $\mathbb{E}[Y|X]$.
To this end, I'd like to pick a collection of functions $f_1, f_2 \dots f_n : \mathbb{R} \to \mathbb{R}$ and then ...
- 413
4
votes
2
answers
4k
views
Deep Learning: Condition Number and Poor Conditioning
I am reading the following section of the book Deep Learning.
Can you provide an intuitive explanation of the above section? I don't quite understand the statement "When this number is large, matrix ...
- 1,189
4
votes
0
answers
395
views
Proximal Gradient Descent and Proximal Coordinate descent for Lasso Problem
Why is proximal coordinate descent much less affected by bad conditioning than proximal gradient descent?
For example, we can consider this problem : $\min_x \frac{1}{2}\|Ax-b\|^2_2 + \lambda\|x\|_1$
...
- 76
1
vote
0
answers
90
views
Is condition number important when fitting splines?
I am fitting data using p-splines. The authors of p-splines, Eilers and Marx, remark that there is no technical requirement to have a small number of knots and in fact you can have many more knots ...
- 2,442
2
votes
2
answers
4k
views
Multicolinearity and Condition number of logistic regresison
It seems to be common to take a "high" condition number as a sign for multicolinearity in regression analysis. For linear models I'm totally convinced that this is a good idea, but is there any ...
- 555
15
votes
1
answer
18k
views
Cause of a high condition number in a python statsmodels regression?
I'm pretty new to regression analysis, and I'm using python's statsmodels to look at the relationship between GDP/health/social services spending and health outcomes (DALYs) across the OECD. Just to ...
- 151
0
votes
1
answer
2k
views
Condition number calculation in R
If I understood correctly, the condition number should be a product of Frobenious norms of a matrix and its inverse.
In R if I do the following:
...
- 171
1
vote
1
answer
416
views
Wrong choice of covariance function?
I am applying Gaussian process regression (GPR) model on some data assuming covSEiso (a.k.a. RBF) covariance function. I made sure there are no identical data points (but there are similar data ...
- 975
5
votes
2
answers
2k
views
When is it appropriate to override the default reciprocal condition number tolerance for solve() in R?
I am estimating a GMM IV model, where I'm creating a weighting matrix by taking the inverse of Z'Z, where Z is a matrix of instruments. For certain combinations of instruments, when I try to compute ...
- 151
2
votes
0
answers
590
views
Should the intercept be included when you check the condition index?
Many sources state that a condition index >30 constitutes a multicollinearity problem. When I've tried to implement this check in practice, I've realized that the condition index (and VIFs) change ...
- 155
1
vote
0
answers
58
views
Problems with ill-conditioned matrix for Dantzig Selector
I'm using the Dantzig Selector on high dimensional data and my matrix is ill conditioned (condition number on the order of magnitude of 10^17). I know there are ways to improve the conditioning, ...
- 661
1
vote
0
answers
776
views
How to interpret differences in VIF and condition number?
In my present data, the Variance Inflation Factors suggest lack of substential multicolliniearity (<1,7). However, the condition number of 28 is almost at the critical value of 30.
How do I ...
- 31
14
votes
2
answers
20k
views
How do you interpret the condition number of a correlation matrix
I have two correlation matrices, one with a condition number of 9 and the other with a condition number of 70. From what i have read, it will appear that the first matrix is better conditioned than ...
- 143
5
votes
1
answer
386
views
Automatically fixing ill-conditioning or collinearity
I'm backtesting a regression model, which entails running it on a bunch of bootstrap samples of a "rewound" version of our data set. Unfortunately, in some of these resamplings, I end up getting some "...
- 5,608
1
vote
1
answer
1k
views
Propogation of error in a matrix inversion
I'm trying to find the deterministic error bounds for some parameters calculated through distance geometry.
The equation can be simplified to the following form:
$ \left[\begin{matrix} x_1 \\ x_2 \\ ...
- 139
2
votes
2
answers
2k
views
Condition number of data matrix and stability of OLS estimates
I have a multivariate regression model $Y=X\beta ' + \epsilon$. The variables in the $X$ matrix have very different scales and hence the condition number of $X'X$ is huge (order of trillions).
I ...
- 781