The inverse of a given square matrix, $A$, is the matrix $A^{-1}$ such that $AA^{-1}$ is the identity matrix.
1
vote
1answer
46 views
Matrices: system that is “computationally singular” versus “exactly singular”
I would like to know the mathematical concepts behind singular matrices. Matrices that do not have inverses in R throw one of two errors. I have provided some examples of both errors below:
Error in ...
3
votes
1answer
63 views
Explanation of generalization of Newton's Method for multiple dimensions
I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
1
vote
0answers
18 views
Recursively expressing matrix inverse
Let $X$ be an $D \times N$ matrix. Let $I$ be a $D \times D$ identity matrix. Also let $y$ be a $N \times 1$ column vector. Suppose we are trying to solve $(X X ^T + k I) w = Xy$ for a $D$ dimensional ...
3
votes
1answer
102 views
How do we know $X'X$ is nonsingular in OLS?
I am currently working through understanding the mechanics of OLS estimates and the hat matrix. One thing I have been searching for without luck is how we know that the term $X'X$ is invertible where $...
2
votes
1answer
46 views
Use of Moore-Penrose Inverse in Likelihood Computation
I've been running MLE on a mixed-Gaussian model (let's just focus on the multivariate Gaussian case for now). In order to take the inverse of the covariance matrix, I've been using MATLAB's pinv() ...
0
votes
1answer
41 views
Newton-Raphson Error
According to Agresti(2013) pg 364-365, iterative methods such as Newton-Raphson methods, $
\begin{aligned}
\beta^\text{new} &= \beta^\text{old} + (X^{T}WX)^{-1}X^{T}(V)
\end{aligned}
$ help to ...
0
votes
0answers
11 views
What is the use of right inverse of non-square matrix in context of solutions to linear system?
I know that left inverse gives us least squares solution in case matrix A is full column rank.
I am not able to think of any use of right inverse of A matrix in case of under determined systems and ...
0
votes
0answers
17 views
Why does the vectorization of the sum of variance-weighed least square equal $R^{T}\Sigma R$?
The sum of the variance-weighed least square errors of $n$ independent observations is given by
$$\sum_{i=1}^{n}\frac{(y_i-\hat{y}_i)^2}{\sigma_i^2}$$
where
$\begin{cases}
y_i \mathrm{\,\,is\,\,the\,\...
1
vote
0answers
32 views
Are the Inverses of two asymptotically equivalent matrices themselves asymptotically equivalent
Suppose $M_n = P_n + op(1)$.
Is it the case that $M_n^{-1} = P_n^{-1} + op(1)$, if both $M_n^{-1}$ and $P_n^{-1}$ exist with probability going to 1 as $n$ increases? Can the Continuous Mapping ...
1
vote
1answer
49 views
Obtaining the possible least squares solutions when $X^TX$ is not invertible
If $X^TX$ is not invertible, what is the set of solutions for the least squares estimator $\hat{\beta_1}$ in the below?
$Y_i = \beta_0 +\beta_1(x_i-\bar{x}) +\epsilon_i$
I got as far as writing out ...
1
vote
1answer
73 views
“Undefined real result” with inverse() function in WinBUGS
Using WinBUGS, I'd like to generate the posterior distribution for a function of several other variables (stochastic nodes). One step in building up this function involves taking the inverse of a ...
1
vote
0answers
38 views
Expectation of a Matrix Raised to a Power
I have been able to find little literature related to the topic online (e.g., A note on the Expected Value of an Inverse Matrix such that E($\frac{1}{X})$ $\geq$ ($\frac{1}{E(X)}$), though, vaguely, I ...
0
votes
1answer
42 views
Least Square Solution to Linear Regression Problem
Given two column-vectors, A = [0; 1.6818; 2.8284; 3.8337; 4.7568; 5.6234] and B = [0; 984.7; 1590.7; 2029.1; 2251.9; 2254.45], I ...
1
vote
0answers
109 views
Section on Moore-Penrose Pseudoinverse in the Deep Learning book
I am reading the following section in the Deep Learning book by Goodfellow, Bengio and Courville.
I have some questions.
If $A\in\mathbb{R}^{m\times n}$ and $Ax=y$, then we can have left inverse ...
6
votes
1answer
454 views
Why can't we cancel these two matrices in the OLS estimator?
When I have a multivariable linear regression model for sample element $i$:
$$y_i=\beta_1x_{1,i}+...+\beta_kx_{k,i}+\epsilon_i,$$
then the OLS estimator is determined by the following equation:
$$X^Ty=...
3
votes
1answer
115 views
Why forward and reverse transforms for PPCA are so different?
In probabilistic PCA (PPCA), they model dimensionality reduction as probabilistic model
$t = Wx + \mu + \epsilon$
where
$W$ is non-swuare matrix of size $(d \times q)$, $q < d$
$x$ is a vector ...
1
vote
1answer
36 views
Under-constrained models and invertibility of covariance matrix
In Goodfellow et al.'s Deep Learning, the authors write on page 232:
[$\mathbf{X^\top X}$] can be singular whenever the data-generating distribution truly has no variance in some direction, or when ...
1
vote
1answer
423 views
The probability limit of an inverse matrix
all:
I am considering a question regarding the calculation of the probability limit for an inverse matrix.
Specifically, suppose we have a non-singular and squared matrix $M$ with dimension $2\times ...
5
votes
1answer
464 views
Usefulness of convexity of linear regression when there is no closed form solution
The optimisation problem in linear regression, $f(\beta) = ||y-X\beta||^2$ is convex (as it is a quadratic function), and when $(X^TX)$ is invertible, we have a unique solution which we can calculate ...
1
vote
1answer
481 views
Non-Singularity due to inclusion of non-zero lambda in ridge regression [duplicate]
There were many similar questions on this site , related to this but none were exactly to the point I wanted to ask
So the question is relates to ridge regression and This link where there is a ...
1
vote
0answers
249 views
Solving an inverse problem with machine learning
I am running up against a very tough inverse problem that I suspect might be solvable using machine learning. Here is the problem.
Overview
I am studying an object $X$ which, internally, is ...
2
votes
1answer
73 views
Invert singular matrix for design of experiments and regression
I have a matrix, $X'X$, which is singular meaning that I cannot invert it. I need the inverse of this matrix to perform two independent things. I need it for the design of experiments, in R using ...
4
votes
1answer
292 views
How to show that demeaning the data in design matrix does not change the hat matrix
When I have a design matrix $$X =
\begin{bmatrix}
1 & x_{11} & \ldots & x_{1k}\\
1 & x_{21} & \ldots & x_{2k}\\
\vdots & \vdots & \ddots & \vdots\\
1 & ...
3
votes
1answer
189 views
Spectral Decomposition of a symmetric matrix times a diagonal matrix
I would like to find the inverse of the sum of a Kronecker product and a diagonal matrix. I found this answered here, but I don't see how the last step is valid.
Because of the fact that $C$ ...
1
vote
0answers
195 views
How to compare diagonal elements of precision matrix (the inverted correlation matrix)?
Let $$C=\begin{pmatrix}C_{11} & C_{12}\\ C_{21} & C_{22}\end{pmatrix}$$ be a $p\times p$ correlation matrix with positive entries, where $C_{11}$ is a $q\times q$ matrix. Define $D=C^{-1}=(d_{...
1
vote
0answers
137 views
when can I substitute an inverse with a pseudo-inverse in an estimator
Short Version: can I substitute the Moore-Penrose generalized inverse of a matrix (R function ginv()) for a matrix inverse (R function ...
1
vote
0answers
57 views
Generalised inverses - Solve normal equations
I am still not fully able to handle the concept of generalised inverses when applied to OLS.
Is there a way to show that
$\beta$ = $\ (X'X)^- X'y + (I_k−(X'X)^-X'X)z $
solves the normal equations:
...
2
votes
0answers
608 views
Efficient/feasible sparse matrix inversion in R
I am looking to perform a 2-stage least-squares estimation with sparse matrices in R, in the style of Bramoulle et al (J. Econometrics 2009). Specifically, let:
G be a very sparse block-diagonal ...
0
votes
1answer
533 views
Solving simple linear equation using Matrices [closed]
I need to find a matrix A whose dimensions will be 1 x n and I have input matrix X whose ...
3
votes
1answer
309 views
When is it appropriate to override the default reciprocal condition number tolerance for solve() in R?
I am estimating a GMM IV model, where I'm creating a weighting matrix by taking the inverse of Z'Z, where Z is a matrix of instruments. For certain combinations of instruments, when I try to compute ...
8
votes
1answer
2k views
What is the physical significance of inverse of a matrix? [closed]
I was asked this question in an interview. Though I tried my best to answer the question in whatever way I could (I was explaining in terms of mathematics), the professor looked upset.
Any idea?
The ...
1
vote
2answers
782 views
Residual Sum of squares in Weighted regression
The Residual Sum of squares (RSS) in Weighted regression is written as
$$(\mathbf{y-X\hat{\boldsymbol\beta}})^{'}\mathbf{C}^{-1}(\mathbf{y-X\hat{\boldsymbol\beta}})$$
Where $$\hat{\boldsymbol\beta}=(\...
0
votes
0answers
41 views
How to anticipate bad solutions to system of equations?
I know the identity that matrix $\mathbf{Y}$ is a product of $\mathbf{X}$ and $\mathbf{B}$ (all of which are integer matrices in my case):
\begin{equation}
\mathbf{Y} = \mathbf{X} \mathbf{B}
\end{...
8
votes
3answers
4k views
What is an example of perfect multicollinearity?
What is an example of perfect collinearity in terms of the design matrix $X$?
I would like an example where $\hat \beta = (X'X)^{-1}X'Y$ can't be estimated because $(X'X)$ is not invertible.
0
votes
1answer
1k views
Inverse of block covariance matrix
I have a positive definite symmetric covariance matrix which looks like this:
A, B, C, D and E, F, G are MATRICES, also positive definite symmetric covariance
What is the inverse of such a matrix? ...
1
vote
1answer
139 views
Evaluate high-dimensional Gaussian with variance matrix $\sigma^{2}I_{n_{t}\times n_{t}}+\boldsymbol{\Sigma}_{t}\boldsymbol{\Sigma}_{t}^{'}$
I need to compute the log-likelihood function in a high-dimensional
Gaussian time-series. I have the following model:
$\mathbf{y}_{t}\left|\mathcal{F}_{t-1}\sim\mathcal{N}\left(\mathbf{\boldsymbol{\...
0
votes
0answers
92 views
The Existance of Schur Complement Inverse
A block matrix $\mathbf{M}=\left[ \begin{array}{ccc}
\mathbf{A} & \mathbf{B} \\
\mathbf{B}^T & \mathbf{C} \end{array} \right]$ is invertible if $\mathbf{A}$ and $(\mathbf{C}-\mathbf{B}^T\...
1
vote
1answer
135 views
Efficient routines for a regression with orthogonal regressors?
I have a standard OLS regression setup, where (sets of) the regresors are orthogonal to each other. I am looking for a fast low-level way (using qr() instead of <...
1
vote
0answers
80 views
Application of Givens rotation to two matrices
I am reading this paper on Multiresolution Matrix Fatorization, http://arxiv.org/pdf/1507.04396v1.pdf, and have come across something that seems like an error to me. In Algorithm 2, the authors take $...
5
votes
3answers
583 views
Invert a sparse covariance matrix
I have a postive definite symmetric covariance matrix which looks like this:
Note that all A,B,C,D,E,F,G are also poitive definite symmetric covariance matrices I want to find an easy way were I can ...
5
votes
1answer
171 views
Interpretation of the cluster criterion $\operatorname{tr}(S_W^{-1}S_B)$
There is a cluster criterion defined as:
$$\mathcal{C} = \operatorname{tr}(S_W^{-1}S_B) = \sum_{i=1}^d \lambda_i,$$
where $\operatorname{tr}$ is the trace, $S_W$ is the pooled within-group scatter ...
9
votes
2answers
716 views
Lucid explanation for “numerical stability of matrix inversion” in ridge regression and its role in reducing overfit
I understand that we can employ regularization in a least squares regression problem as
$$\boldsymbol{w}^* = \operatorname*{argmin}_w \left[ (\mathbf y-\mathbf{Xw})^T(\boldsymbol{y}-\mathbf{Xw}) + \...
1
vote
0answers
52 views
Inverse of ordering function
Lets assume we have p by n matrix.We can generate an output matrix, w (p x p) such as w_ij represent how many times i_th rows number is bigger than j_th (can be at most n obviously).
My question is ...
7
votes
2answers
3k views
Prove that $(A^{-1} + B^{-1})^{-1}=A(A+B)^{-1}B$
I have this equality
$$(A^{-1} + B^{-1})^{-1}=A(A+B)^{-1}B$$ where $A$ and $B$ are square symmetric matrices.
I have done many test of R and Matlab that show that this holds, however I do not know ...
4
votes
3answers
2k views
Matrix inverse not able to be calculated while determinant is non-zero
I was attempting to calculate an OLS regression in R when I saw some strange things. The inverse of a square matrix does not exist if and only if the determinants is 0. Also, the matrix must be of ...
2
votes
1answer
348 views
What is the purpose of inverse distance?
In many econometric models, they use the inverse distance, instead of just distance. For example, if they are looking at the impact of distance to CBD on land values, they might use the inverse ...
1
vote
2answers
221 views
Efficient, stable inverse of a patterned covariance matrix for gridded data
I have computed a stationary covariance matrix defined for data on a grid. The data y are regularly spaced in 3D, lexicographically ordered in the covariance matrix, and I'm using a using a square ...
13
votes
1answer
287 views
Explain how `eigen` helps inverting a matrix
My question relates to a computation technique exploited in geoR:::.negloglik.GRF or geoR:::solve.geoR.
In a linear mixed model ...
7
votes
3answers
572 views
What does that statistically mean , if $(X'X)^{-1}$ does not exist?
It is not a real world case , but suppose that we have $n$ observations and $k$ variables , since $k= n - 1 $ , if $X$ is the design matrix,$(X'X)$ will be a square matrix , so What does that ...
2
votes
1answer
293 views
Notation for ordinary and ridge regression
I'm trying to obtain an estimator $f(x)=y$ where $x \in \mathbb{R}^{D_1}$ and $y \in \mathbb{R}^{D_2}$, both are column vectors.
So my training set $X$ and $Y$ are data matrices of size $D_1 \times ...
