Questions tagged [matrix-inverse]
The inverse of a given square matrix, $A$, is the matrix $A^{-1}$ such that $AA^{-1}$ is the identity matrix.
124
questions
1
vote
0answers
36 views
Why is least squares failing on my exact linear equation? (sanity checking) [closed]
I have a problem with 3 free variables. Right now I am in a simulation phase, so I can create an arbitrary number of example cases for regression.
This is a linear problem, so I can write my data in ...
0
votes
0answers
9 views
How to overcome the issue of singular within-cluster scatter matrices in clustering using entropy-based feature ranking?
I am trying to implement the entropy-based feature selection method for clustering by Dash and Liu. In this method, features are ranked in importance based on an entropy minimisation procedure and ...
0
votes
0answers
14 views
Squared Multiple Correlation (SMC) of my correlation matrix tend towards 1. How to interpret this?
In order to be able to conduct exploratory factor analysis, I want to carry out parallel analysis to determine the number of factors to be extracted. To do so, I want to extract the eignevalues of the ...
0
votes
1answer
49 views
Upper bound trace of inverse of covariance matrix
Let C be the covariance matrix from any normal distribution. If the trace of C is upper-bounded by a constant k (i.e., tr(C)<=k), can I find an upper bound for the trace of the inverse of C (i.e., ...
0
votes
0answers
21 views
Fast inverse of positive definite matrix subtracted by its Nystrom approximation
Assume a positive definite symmetric covariance matrix $$C_{n,n}$$ and let its Nystrom approximation be $$\hat{C}_{n,n}=C_{n,q} C^{-1}_{q,q} C_{q,n}$$ for some $q<n$
Inverting $C_{n,n}$ is of $\...
0
votes
0answers
14 views
Matrix inversion and $\mathcal{O}_p$ notation: $(\mathbf{A} + \mathcal{O}_p(f(n)))^{-1} = \mathbf{A}^{-1} + \mathcal{O}_p(??)$
Suppose that $\mathbf{A}$ is square invertible matrix and that $\hat{\mathbf{A}}$ is an estimator of $\mathbf{A}$ based on a sample of size $n$ such that:
$\hat{\mathbf{A}}$ is invertible,
$\hat{\...
4
votes
1answer
181 views
Generalized Least Squares using Moore Penrose pseudo inverse
I'm using GLS to fit a model where some independent variables are strongly correlated. Therefore my covariance matrix is singular. I have found that Moore-Penrose pseudo inverse can be used to find an ...
1
vote
1answer
116 views
Bypassing inverse matrix calculation and the comparison of Gradient Descent and Newton Descent
I am currently reading Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.
In the 5th chapter the Gradient Descent algorithm is introduced with the following notations :
$...
1
vote
0answers
65 views
Distribution of the Sum of an AR(1) Model Time Series
I have the following model for my model
$\Delta X_{t} = \mu \Delta t + \rho \Delta X_{t-1} + \sigma \sqrt{\Delta t} Z_t$
with the following initial conditions -
$\Delta X_{1} = \mu \Delta t + \...
0
votes
0answers
23 views
Finding inverse of $X'X$ in the case of two regressors [duplicate]
Variance of OLS etimator in matrix form look like this: $Var(\hat{\beta_j})=\sigma^2(X'X^{-1})$
I'm struggling to derive inverse matrix for the case with two independent variables.
$X'X$ $=$ $\...
2
votes
1answer
43 views
numerically stable sparse Gaussian process regression (matrix inversion)
In sparse approximations of GP for large data set $(X,\mathbf{y})$ with $n$ samples, usually $m$ inducing points are chosen such that the true covariance matrix is approximated by $K_{nn}\to K_{nm}K_{...
4
votes
1answer
143 views
Incremental solution for matrix inverse using Shermann-Morrison in $O(n^2)$
I have been reading a presentation on Value Function Approximation by David Silver (Introduction to Reinforcement Learning Course). On page 43 he finds a solution for linear least squares for an ...
4
votes
1answer
444 views
Which is more numerically stable for OLS: pinv vs QR
If I am doing standard OLS and want to calculate beta values (OLS estimators), which of the following is the more numerically stable method? And why?
Assuming that the columns of $X$ are already mean-...
2
votes
1answer
88 views
Use $X^{-1}Y$ instead of $(𝑋^T𝑋)^{−1}𝑋^T Y$ to calculate $\beta$ when $X$ is already a square matrix in the least square problem
In the least squares problem $X\beta = Y$, the solution is $\hat{\beta} = (𝑋^T𝑋)^{−1}𝑋^TY$. I learned that two facts:
$𝑋^T𝑋$ is square matrix so that the definition of matrix inversion is ...
1
vote
1answer
74 views
Proof of Pearson-Aitken selection formula
I am trying to understand the proof of the Pearson-Aitken selection formula, widely used in statistical genetics. A proof that the formula is general is given by Aitken (1936). However, I failed to ...
1
vote
1answer
56 views
Why the representation in the form of $Z'X(X'X)^{-1}X'Z$ can not be simplified into $Z'Z$
Representation similar to $Z'X(X'X)^{-1}X'Z$ frequently appear to e.g. 2SLS.
I think that $Z'X(X'X)^{-1}X'Z = Z'XX^{-1}X'^{-1}X'Z = Z'(XX^{-1})(X'^{-1}X')Z = Z'Z$. So why it seems that in the context ...
0
votes
1answer
23 views
matrix inequality related to finance
I'm trying to show that, for certain investment strategies, it pays to have more precise estimates of the covariance matrix of your returns. I have always took this for granted, but I've been having ...
0
votes
1answer
254 views
How are biases updated when 'batch size' > 1?
This is my network represented in matrices: (a dot represents an arbitrary number)
Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity)
Backpropagation
The ...
3
votes
1answer
1k views
Matrices: system that is “computationally singular” versus “exactly singular” [closed]
I would like to know the mathematical concepts behind singular matrices. Matrices that do not have inverses in R throw one of two errors. I have provided some examples of both errors below:
Error in ...
3
votes
1answer
148 views
Explanation of generalization of Newton's Method for multiple dimensions
I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
1
vote
0answers
21 views
Recursively expressing matrix inverse
Let $X$ be an $D \times N$ matrix. Let $I$ be a $D \times D$ identity matrix. Also let $y$ be a $N \times 1$ column vector. Suppose we are trying to solve $(X X ^T + k I) w = Xy$ for a $D$ dimensional ...
3
votes
1answer
911 views
How do we know $X'X$ is nonsingular in OLS?
I am currently working through understanding the mechanics of OLS estimates and the hat matrix. One thing I have been searching for without luck is how we know that the term $X'X$ is invertible where $...
10
votes
3answers
1k views
Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression
The closed form of w in Linear regression can be written as
$\hat{w}=(X^TX)^{-1}X^Ty$
How can we intuitively explain the role of $(X^TX)^{-1}$ in this equation?
0
votes
0answers
49 views
Why does higher variance of independent variables decrease standard errors of the estimator?
Some time ago, reading Advanced Data Analysis
from an Elementary Point of View by Cosma Rohilla Shalizi, I found this statement in a section about significant coefficients:
Moreover, at a fixed ...
2
votes
1answer
190 views
Use of Moore-Penrose Inverse in Likelihood Computation
I've been running MLE on a mixed-Gaussian model (let's just focus on the multivariate Gaussian case for now). In order to take the inverse of the covariance matrix, I've been using MATLAB's pinv() ...
0
votes
1answer
53 views
Newton-Raphson Error
According to Agresti(2013) pg 364-365, iterative methods such as Newton-Raphson methods, $
\begin{aligned}
\beta^\text{new} &= \beta^\text{old} + (X^{T}WX)^{-1}X^{T}(V)
\end{aligned}
$ help to ...
1
vote
0answers
61 views
Are the Inverses of two asymptotically equivalent matrices themselves asymptotically equivalent
Suppose $M_n = P_n + op(1)$.
Is it the case that $M_n^{-1} = P_n^{-1} + op(1)$, if both $M_n^{-1}$ and $P_n^{-1}$ exist with probability going to 1 as $n$ increases? Can the Continuous Mapping ...
2
votes
2answers
324 views
Obtaining the possible least squares solutions when $X^TX$ is not invertible
If $X^TX$ is not invertible, what is the set of solutions for the least squares estimator $\hat{\beta_1}$ in the below?
$Y_i = \beta_0 +\beta_1(x_i-\bar{x}) +\epsilon_i$
I got as far as writing out ...
1
vote
1answer
193 views
“Undefined real result” with inverse() function in WinBUGS
Using WinBUGS, I'd like to generate the posterior distribution for a function of several other variables (stochastic nodes). One step in building up this function involves taking the inverse of a ...
1
vote
0answers
91 views
Expectation of a Matrix Raised to a Power
I have been able to find little literature related to the topic online (e.g., A note on the Expected Value of an Inverse Matrix such that E($\frac{1}{X})$ $\geq$ ($\frac{1}{E(X)}$), though, vaguely, I ...
0
votes
1answer
72 views
Least Square Solution to Linear Regression Problem
Given two column-vectors, A = [0; 1.6818; 2.8284; 3.8337; 4.7568; 5.6234] and B = [0; 984.7; 1590.7; 2029.1; 2251.9; 2254.45], I ...
1
vote
0answers
282 views
Section on Moore-Penrose Pseudoinverse in the Deep Learning book
I am reading the following section in the Deep Learning book by Goodfellow, Bengio and Courville.
I have some questions.
If $A\in\mathbb{R}^{m\times n}$ and $Ax=y$, then we can have left inverse ...
6
votes
1answer
485 views
Why can't we cancel these two matrices in the OLS estimator?
When I have a multivariable linear regression model for sample element $i$:
$$y_i=\beta_1x_{1,i}+...+\beta_kx_{k,i}+\epsilon_i,$$
then the OLS estimator is determined by the following equation:
$$X^Ty=...
3
votes
1answer
186 views
Why forward and reverse transforms for PPCA are so different?
In probabilistic PCA (PPCA), they model dimensionality reduction as probabilistic model
$t = Wx + \mu + \epsilon$
where
$W$ is non-swuare matrix of size $(d \times q)$, $q < d$
$x$ is a vector ...
1
vote
1answer
51 views
Under-constrained models and invertibility of covariance matrix
In Goodfellow et al.'s Deep Learning, the authors write on page 232:
[$\mathbf{X^\top X}$] can be singular whenever the data-generating distribution truly has no variance in some direction, or when ...
1
vote
1answer
792 views
The probability limit of an inverse matrix
all:
I am considering a question regarding the calculation of the probability limit for an inverse matrix.
Specifically, suppose we have a non-singular and squared matrix $M$ with dimension $2\times ...
5
votes
1answer
644 views
Usefulness of convexity of linear regression when there is no closed form solution
The optimisation problem in linear regression, $f(\beta) = ||y-X\beta||^2$ is convex (as it is a quadratic function), and when $(X^TX)$ is invertible, we have a unique solution which we can calculate ...
1
vote
1answer
2k views
Non-Singularity due to inclusion of non-zero lambda in ridge regression [duplicate]
There were many similar questions on this site , related to this but none were exactly to the point I wanted to ask
So the question is relates to ridge regression and This link where there is a ...
1
vote
0answers
329 views
Solving an inverse problem with machine learning
I am running up against a very tough inverse problem that I suspect might be solvable using machine learning. Here is the problem.
Overview
I am studying an object $X$ which, internally, is ...
2
votes
1answer
119 views
Invert singular matrix for design of experiments and regression
I have a matrix, $X'X$, which is singular meaning that I cannot invert it. I need the inverse of this matrix to perform two independent things. I need it for the design of experiments, in R using ...
4
votes
1answer
489 views
How to show that demeaning the data in design matrix does not change the hat matrix
When I have a design matrix $$X =
\begin{bmatrix}
1 & x_{11} & \ldots & x_{1k}\\
1 & x_{21} & \ldots & x_{2k}\\
\vdots & \vdots & \ddots & \vdots\\
1 & ...
2
votes
1answer
253 views
Spectral Decomposition of a symmetric matrix times a diagonal matrix
I would like to find the inverse of the sum of a Kronecker product and a diagonal matrix. I found this answered here, but I don't see how the last step is valid.
Because of the fact that $C$ ...
1
vote
0answers
387 views
How to compare diagonal elements of precision matrix (the inverted correlation matrix)?
Let $$C=\begin{pmatrix}C_{11} & C_{12}\\ C_{21} & C_{22}\end{pmatrix}$$ be a $p\times p$ correlation matrix with positive entries, where $C_{11}$ is a $q\times q$ matrix. Define $D=C^{-1}=(d_{...
2
votes
0answers
206 views
when can I substitute an inverse with a pseudo-inverse in an estimator
Short Version: can I substitute the Moore-Penrose generalized inverse of a matrix (R function ginv()) for a matrix inverse (R function ...
2
votes
0answers
96 views
Generalised inverses - Solve normal equations
I am still not fully able to handle the concept of generalised inverses when applied to OLS.
Is there a way to show that
$\beta$ = $\ (X'X)^- X'y + (I_k−(X'X)^-X'X)z $
solves the normal equations:
...
3
votes
0answers
1k views
Efficient/feasible sparse matrix inversion in R
I am looking to perform a 2-stage least-squares estimation with sparse matrices in R, in the style of Bramoulle et al (J. Econometrics 2009). Specifically, let:
G be a very sparse block-diagonal ...
0
votes
1answer
660 views
Solving simple linear equation using Matrices [closed]
I need to find a matrix A whose dimensions will be 1 x n and I have input matrix X whose ...
4
votes
1answer
616 views
When is it appropriate to override the default reciprocal condition number tolerance for solve() in R?
I am estimating a GMM IV model, where I'm creating a weighting matrix by taking the inverse of Z'Z, where Z is a matrix of instruments. For certain combinations of instruments, when I try to compute ...
8
votes
1answer
3k views
What is the physical significance of inverse of a matrix? [closed]
I was asked this question in an interview. Though I tried my best to answer the question in whatever way I could (I was explaining in terms of mathematics), the professor looked upset.
Any idea?
The ...
2
votes
2answers
952 views
Residual Sum of squares in Weighted regression
The Residual Sum of squares (RSS) in Weighted regression is written as
$$(\mathbf{y-X\hat{\boldsymbol\beta}})^{'}\mathbf{C}^{-1}(\mathbf{y-X\hat{\boldsymbol\beta}})$$
Where $$\hat{\boldsymbol\beta}=(\...
