Questions tagged [matrix-inverse]
The inverse of a given square matrix, $A$, is the matrix $A^{-1}$ such that $AA^{-1}$ is the identity matrix.
139
questions
0
votes
0answers
17 views
Singularity of a modelmatrix that consists of one predictor raised to multiple powers
I was playing around a little bit with linear regression models in R and wanted to try out whether I can get a perfect fit for a linear model where I have a response $$Y =
\left(
\begin{array}{c}
y_1\...
1
vote
1answer
58 views
Mercer's theorem and eigenfunctions
Consider a kernel $K$ which satisfies the conditions of Mercer's theorem. We know that $K(x,y) = \sum\limits_{i=1}^{+\infty}\lambda_ie_i(x)e_i(y)$ where $\lambda_i$ and $e_i$ are the eigenvalues/...
0
votes
0answers
45 views
Finding inverse matrix $(X'X)^{-1}$ with $X$ as design matrix
I'm relatively new to all this and I am trying to figure out how I can derive the matrix $(X'X)^{-1}$ when I have given $x_1, x_2, x_3$ and $y$. $X$ is the design matrix in that case but not sure how ...
0
votes
0answers
11 views
Updating the Matern Covariance inverse
I'm developing an MCMC, and within it, I use the Matern covariance function, provided by:
$$
{\displaystyle C_{\nu }(d)=\sigma ^{2}{\frac {2^{1-\nu }}{\Gamma (\nu )}}{\Bigg (}{\sqrt {2\nu }}{\frac {d}{...
3
votes
1answer
54 views
linear regression and obfuscation matrix - just not clicking for me?
Here is the basis of my question.
We have an exercise to take a data set of made up insurance claims and train a linear regression model to predict future claims; however, because it is sensitive data,...
2
votes
1answer
93 views
completing a square
If I have a density function of the form $p(x) \propto \exp(−q(x)/2)$ where $q(x)$ has
the following quadratic function
$$q(x)=x^Tx+y^Ty-[x^TA+y^TB][A^TA+B^TB+\beta\mathbb{I}]^{-1}[A^Tx+B^Ty]$$
where $...
1
vote
1answer
12 views
What is an example of a systems problem that can be solved by either 'matrix inversion' or an 'iterative procedure' to arrive at the same result?
I recently heard over a radio program (in French) that a given problem - can't recall it exactly, but it involved solving large systems - could be solved "through matrix inversion" or "...
15
votes
1answer
821 views
Relationship between Cholesky decomposition and matrix inversion?
I've been reviewing Gaussian Processes and, from what I can tell, there's some debate whether the "covariance matrix" (returned by the kernel), which needs to be inverted, should be done so ...
3
votes
1answer
60 views
What dimensionality reduction methods allow a lower dimensional reconstruction of the original data besides PCA via invertible transformations?
In eigenfaces, one used the inverse transformation PCA is capable of doing to reconstruct the low dimensional face image.
In tsne one may not reconstruct the original dataset to produce something akin ...
-2
votes
1answer
146 views
Why doesn't $(e^{A})^{-1} = e^{-A}$ hold for a symmetric matrix in Python?
$e^A$ is just the $A$ matrix with all of its elements exponentiated, called a matrix exponential.
It follows that the inverse $(e^{A})^{-1} = e^{-A}$ for square matrices, although I could find nothing ...
0
votes
0answers
21 views
Proof: how to prove that the inverse of the fundamental matrix N of an absorbing chain exists (i.e. $N^{-1}$ exists) and $Q=I-N^{-1}$?
I know that I-Q can not have a zero determinant, so it has an inverse, i.e. $N=(I-Q)^{-1}$ exists. I think I know how to prove part b of this question given that we assume $N^{-1}$ exists, my ...
0
votes
1answer
98 views
Proving that the hat matrix is unchanged even when the predictors are multiplied by constants
I know that the hat matrix $H = X(X^T X)^{-1} X^T$, and that $\hat{Y} = HY$. When we have some non-zero constants that we multiply each respective predictor by, which just multiplies every column in ...
2
votes
0answers
32 views
Why is it not possible to simplify $b=(X'X)^{-1}X'y$ [duplicate]
Why is the following not possible:
$$
b=(X'X)^{-1}X'y = X^{-1}(X')^{-1}X'y=X^{-1}y
$$
While this term $(AB)^{-1}=B^{-1}A^{-1}$ applies to any two matrices as long as both are of full rank and are $nxn$...
3
votes
1answer
353 views
Moore Penrose Pseudo-Inverse Fast Algorithm in R [closed]
I want to apply Moore Penrose Pseudo-Inverse on my matrix, which is a 20,000 * 20,000 symmetric matrix with rank 19,999. I found ginv() function from the ...
0
votes
1answer
174 views
How to do multiple linear regressions with overlapping predictors efficiently
I'm using covariance matrices to do a linear regression. I have the same number of predictors and dependent variables (3000 of each). For each $i$, $X_{i}$ and $Y_{i}$ have a kind of symmetry.
I ...
0
votes
0answers
38 views
NN type/architecture needed for inverse covariance matrix approximation
The idea is to construct a neural network (NN) that takes N series of financial returns as input and returns the (approximation of the) inverse of the sample covariance matrix (an N times N matrix). I ...
1
vote
0answers
228 views
Normalization for solving linear equations
Suppose I want to solve a linear equation system in the form of
$$A x = b$$
to get $x$, where $A$ are $n$ by $n$ matrix and $b$ is $n$ by 1 vector. Is there any normalization procedure necessary ...
4
votes
1answer
348 views
Updating the covariance matrix after deleting the i-th column and row
Suppose I have a covariance matrix $A_{n\times n}$ and $A^{-1}_{n\times n}$ is its inverse. Then I randomly exclude the $i$-th row and column, where $1\le i\le n \in \mathbb{N}$, obtaining the new ...
5
votes
2answers
293 views
Least Squares removing first $k$ observations Woodbury formula?
Given the matrix $X_{n,p}$ from the least squares problem $$ \mathbf{X} \cdot \mathbf{\beta} = z $$
Where the normal equation is:
$$ \mathbf{\hat{\beta}} = \left(\mathbf{X}^T \mathbf{X}\right)^{-1} ...
0
votes
0answers
207 views
Squared Multiple Correlation (SMC) of my correlation matrix tend towards 1. How to interpret this?
In order to be able to conduct exploratory factor analysis, I want to carry out parallel analysis to determine the number of factors to be extracted. To do so, I want to extract the eignevalues of the ...
0
votes
1answer
266 views
Upper bound trace of inverse of covariance matrix
Let C be the covariance matrix from any normal distribution. If the trace of C is upper-bounded by a constant k (i.e., tr(C)<=k), can I find an upper bound for the trace of the inverse of C (i.e., ...
0
votes
0answers
31 views
Fast inverse of positive definite matrix subtracted by its Nystrom approximation
Assume a positive definite symmetric covariance matrix $$C_{n,n}$$ and let its Nystrom approximation be $$\hat{C}_{n,n}=C_{n,q} C^{-1}_{q,q} C_{q,n}$$ for some $q<n$
Inverting $C_{n,n}$ is of $\...
5
votes
1answer
753 views
Generalized Least Squares using Moore Penrose pseudo inverse
I'm using GLS to fit a model where some independent variables are strongly correlated. Therefore my covariance matrix is singular. I have found that Moore-Penrose pseudo inverse can be used to find an ...
1
vote
1answer
337 views
Bypassing inverse matrix calculation and the comparison of Gradient Descent and Newton Descent
I am currently reading Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.
In the 5th chapter the Gradient Descent algorithm is introduced with the following notations :
$...
1
vote
0answers
175 views
Distribution of the Sum of an AR(1) Model Time Series
I have the following model for my model
$\Delta X_{t} = \mu \Delta t + \rho \Delta X_{t-1} + \sigma \sqrt{\Delta t} Z_t$
with the following initial conditions -
$\Delta X_{1} = \mu \Delta t + \...
3
votes
1answer
318 views
numerically stable sparse Gaussian process regression (matrix inversion)
In sparse approximations of GP for large data set $(X,\mathbf{y})$ with $n$ samples, usually $m$ inducing points are chosen such that the true covariance matrix is approximated by $K_{nn}\to K_{nm}K_{...
5
votes
1answer
376 views
Incremental solution for matrix inverse using Shermann-Morrison in $O(n^2)$
I have been reading a presentation on Value Function Approximation by David Silver (Introduction to Reinforcement Learning Course). On page 43 he finds a solution for linear least squares for an ...
5
votes
1answer
1k views
Which is more numerically stable for OLS: pinv vs QR
If I am doing standard OLS and want to calculate beta values (OLS estimators), which of the following is the more numerically stable method? And why?
Assuming that the columns of $X$ are already mean-...
3
votes
1answer
158 views
Use $X^{-1}Y$ instead of $(𝑋^T𝑋)^{−1}𝑋^T Y$ to calculate $\beta$ when $X$ is already a square matrix in the least square problem
In the least squares problem $X\beta = Y$, the solution is $\hat{\beta} = (𝑋^T𝑋)^{−1}𝑋^TY$. I learned that two facts:
$𝑋^T𝑋$ is square matrix so that the definition of matrix inversion is ...
3
votes
1answer
122 views
Proof of Pearson-Aitken selection formula
I am trying to understand the proof of the Pearson-Aitken selection formula, widely used in statistical genetics. A proof that the formula is general is given by Aitken (1936). However, I failed to ...
3
votes
1answer
93 views
Why the representation in the form of $Z'X(X'X)^{-1}X'Z$ can not be simplified into $Z'Z$
Representation similar to $Z'X(X'X)^{-1}X'Z$ frequently appear to e.g. 2SLS.
I think that $Z'X(X'X)^{-1}X'Z = Z'XX^{-1}X'^{-1}X'Z = Z'(XX^{-1})(X'^{-1}X')Z = Z'Z$. So why it seems that in the context ...
1
vote
1answer
34 views
matrix inequality related to finance
I'm trying to show that, for certain investment strategies, it pays to have more precise estimates of the covariance matrix of your returns. I have always took this for granted, but I've been having ...
0
votes
1answer
512 views
How are biases updated when 'batch size' > 1?
This is my network represented in matrices: (a dot represents an arbitrary number)
Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity)
Backpropagation
The ...
3
votes
1answer
3k views
Matrices: system that is “computationally singular” versus “exactly singular” [closed]
I would like to know the mathematical concepts behind singular matrices. Matrices that do not have inverses in R throw one of two errors. I have provided some examples of both errors below:
Error in ...
3
votes
1answer
364 views
Explanation of generalization of Newton's Method for multiple dimensions
I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
1
vote
0answers
23 views
Recursively expressing matrix inverse
Let $X$ be an $D \times N$ matrix. Let $I$ be a $D \times D$ identity matrix. Also let $y$ be a $N \times 1$ column vector. Suppose we are trying to solve $(X X ^T + k I) w = Xy$ for a $D$ dimensional ...
4
votes
1answer
2k views
How do we know $X'X$ is nonsingular in OLS?
I am currently working through understanding the mechanics of OLS estimates and the hat matrix. One thing I have been searching for without luck is how we know that the term $X'X$ is invertible where $...
10
votes
3answers
2k views
Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression
The closed form of w in Linear regression can be written as
$\hat{w}=(X^TX)^{-1}X^Ty$
How can we intuitively explain the role of $(X^TX)^{-1}$ in this equation?
0
votes
0answers
415 views
Why does higher variance of independent variables decrease standard errors of the estimator?
Some time ago, reading Advanced Data Analysis
from an Elementary Point of View by Cosma Rohilla Shalizi, I found this statement in a section about significant coefficients:
Moreover, at a fixed ...
2
votes
1answer
423 views
Use of Moore-Penrose Inverse in Likelihood Computation
I've been running MLE on a mixed-Gaussian model (let's just focus on the multivariate Gaussian case for now). In order to take the inverse of the covariance matrix, I've been using MATLAB's pinv() ...
0
votes
1answer
64 views
Newton-Raphson Error
According to Agresti(2013) pg 364-365, iterative methods such as Newton-Raphson methods, $
\begin{aligned}
\beta^\text{new} &= \beta^\text{old} + (X^{T}WX)^{-1}X^{T}(V)
\end{aligned}
$ help to ...
1
vote
0answers
75 views
Are the Inverses of two asymptotically equivalent matrices themselves asymptotically equivalent
Suppose $M_n = P_n + op(1)$.
Is it the case that $M_n^{-1} = P_n^{-1} + op(1)$, if both $M_n^{-1}$ and $P_n^{-1}$ exist with probability going to 1 as $n$ increases? Can the Continuous Mapping ...
3
votes
3answers
3k views
Obtaining the possible least squares solutions when $X^TX$ is not invertible
If $X^TX$ is not invertible, what is the set of solutions for the least squares estimator $\hat{\beta_1}$ in the below?
$Y_i = \beta_0 +\beta_1(x_i-\bar{x}) +\epsilon_i$
I got as far as writing out ...
1
vote
1answer
279 views
“Undefined real result” with inverse() function in WinBUGS
Using WinBUGS, I'd like to generate the posterior distribution for a function of several other variables (stochastic nodes). One step in building up this function involves taking the inverse of a ...
2
votes
0answers
133 views
Expectation of a Matrix Raised to a Power
I have been able to find little literature related to the topic online (e.g., A note on the Expected Value of an Inverse Matrix such that E($\frac{1}{X})$ $\geq$ ($\frac{1}{E(X)}$), though, vaguely, I ...
0
votes
1answer
87 views
Least Square Solution to Linear Regression Problem
Given two column-vectors, A = [0; 1.6818; 2.8284; 3.8337; 4.7568; 5.6234] and B = [0; 984.7; 1590.7; 2029.1; 2251.9; 2254.45], I ...
1
vote
0answers
322 views
Section on Moore-Penrose Pseudoinverse in the Deep Learning book
I am reading the following section in the Deep Learning book by Goodfellow, Bengio and Courville.
I have some questions.
If $A\in\mathbb{R}^{m\times n}$ and $Ax=y$, then we can have left inverse ...
7
votes
1answer
600 views
Why can't we cancel these two matrices in the OLS estimator?
When I have a multivariable linear regression model for sample element $i$:
$$y_i=\beta_1x_{1,i}+...+\beta_kx_{k,i}+\epsilon_i,$$
then the OLS estimator is determined by the following equation:
$$X^Ty=...
4
votes
1answer
250 views
Why forward and reverse transforms for PPCA are so different?
In probabilistic PCA (PPCA), they model dimensionality reduction as probabilistic model
$t = Wx + \mu + \epsilon$
where
$W$ is non-swuare matrix of size $(d \times q)$, $q < d$
$x$ is a vector ...
1
vote
1answer
84 views
Under-constrained models and invertibility of covariance matrix
In Goodfellow et al.'s Deep Learning, the authors write on page 232:
[$\mathbf{X^\top X}$] can be singular whenever the data-generating distribution truly has no variance in some direction, or when ...
