Questions tagged [matrix-inverse]
The inverse of a given square matrix, $A$, is the matrix $A^{-1}$ such that $AA^{-1}$ is the identity matrix.
169 questions
0
votes
0
answers
20
views
Fast covariance evaluation for sparse and large-scale non-linear least square problems
A non-linear least squares problem can be formulated as:
$$
\hat{x} = \underset{x}{\mathrm{argmin}}\, \frac{1}{2} ||\varepsilon(x)||^2
$$
where $x$ is the parameters of the problem, and $\varepsilon(x)...
- 556
1
vote
0
answers
14
views
in matlab, is a matrix of condition number of 1e20 definitely more ill-conditioned than a matrix of condition number of 1e19？
I am working with some ill-conditioned matrices, trying to find the relationship between the matrix's ill-conditioning and the results. However, I have noticed that the condition numbers of these ...
- 11
1
vote
1
answer
63
views
Pseudoinversion giving poor predictions, converges with smaller matrix
I'm having an issue where running pseudoinversion as a means of predicting the values of certain phenotypes using methylation data gives extremely large numbers, very pooly predicting said values, ...
- 25
1
vote
2
answers
68
views
Understanding the multicollinearity issue in relation to linear regression
There are 2 issues that multicollinearity in linear regression leads to
Interpretability goes for a toss
Parameter confidence intervals are wide and it is difficult to find a parameter significant
I ...
- 2,683
1
vote
0
answers
25
views
$E(X)*E(X'X)^{-1}$ simplifies when X vector contains a constant?
For self study, I've calculated $E(X)*E(X'X)^{-1}=[1,0]$ when $X=[1,X_2]$ for any scalar random variable $X_2$ (with finite mean and variance).
Proof:
$\begin{pmatrix}1 & E(X_2)\end{pmatrix}\begin{...
- 11
1
vote
0
answers
101
views
Efficient way to compute covariance matrix of Vector Autoregressive Process of order 1 (VAR)
For a VAR process
$$
X_t = A_1 X_{t-1} + \epsilon_t
$$
The covariance of $X_t$ can be computed in the following way:
$$
\text{vec}(\Sigma) = (I -(A \otimes A))^{-1} \text{vec}(\Sigma_{\epsilon})
$$
...
- 379
3
votes
0
answers
142
views
Random correlation matrices
Suppose that we simulate random $n\times n$ correlation matrices by assigning iid $U(-1,1)$ random variables to all off-diagonal entries and accept matrices $\boldsymbol\Sigma$ that are positive ...
- 11.6k
0
votes
0
answers
32
views
Series expansion for a single coefficient of OLS multiple linear regression?
The Matrix/vector formulation of OLS is given by
$$\vec \beta = \left( X^T X \right)^{-1} X^T \vec y$$
which is very nice as a matrix equation.
But what is the equivalent scalar equation for a single ...
- 9,660
0
votes
1
answer
94
views
Collapse of sampled Mahalanobis distance
Let $\{ x^{(1)}, \ldots, x^{(M)}\}$ be $M$ samples from a $n$-dimensional multivariate Gaussian distribution $\pi_{X} = \mathcal{N}(\mu, \Sigma)$.
We recall the definition of the squared Mahalanobis ...
5
votes
1
answer
1k
views
Why does a valid Kernel only have to be positive semi-definite instead of positive definite?
I'm currently concerned with the topic of Gaussian Processes. To compute the covariance matrix of the conditional distribution, we have to invert $(K_{XX})^{-1}$, where $K_{XX}$ is a matrix of a ...
- 53
3
votes
0
answers
420
views
Inverse of Matrix with normally distributed Elements?
Let's say we have some Matrix $X \in \mathbb{R}^{n_x \times n_x}$ whose elemets $x_{i,j} \sim \mathcal{N}(\mu, \sigma^2)$ are normally distributed. In other words: $vec(X) \sim \mathcal{N}(m, S)$, ...
- 307
2
votes
0
answers
63
views
The mean of Gaussian distribution subject to another Gaussian distribution, how to derive it？
I got a fomula
$$N(y;cx,R)N(x;\bar{x},\Sigma)=N(y;c\bar{x},S)N(x;g,F)$$
where:
\begin{align}S&=c\Sigma c^T+R\\g &=\bar{x}+\Sigma c^Ts^{-1}(y-c\bar{x})\\F &= \Sigma - \Sigma c^T s^{-1} c \...
- 21
3
votes
2
answers
677
views
Parameter estimate in linear regression model with singular matrix $X^TX$
I started to study linear regression this autumn and got stuck at the estimation of regression parameters (in matrix form).
In parameter estimation of the general linear model, the OLS method is used ...
- 31
6
votes
2
answers
1k
views
Why do large LMs use the transpose of the word embeddings matrix in the classification head?
All literature, guides and tutorials describing the construction of language models have used two separate matrices for the input and output projections:
To project one-hot token IDs into hidden ...
- 1,178
1
vote
0
answers
65
views
Updating the Posterior Psi parameter for an Inverse Wishart Distribution
I am fitting a Mixed Multivariate Normal Distribution where the mixing occurs over the mean $\mu_j$ and the covariance matrix $H$ with mixing parameter $B_j$. The number of mixing elements is denotes ...
- 21
1
vote
1
answer
159
views
Why the columns of design matrix for linear models with additive Gaussian noise linear independent?
This question comes from page 142 of the book "Pattern Recognition and Machine Learning" by Christopher M. Bishop. Since it takes pages to arrive at the result, I excerpt the major settings ...
- 333
3
votes
1
answer
143
views
compute Dirichlet distribution parameter from known mean distribution
For a particular Bayesian study I am going to apply Dirichlet distribution as my proposal random number generator. I am going to update the distribution parameter every trial based on a given ...
4
votes
1
answer
350
views
Natural gradients with Moore–Penrose inverse of the Fisher information matrix
I'd like to show you my rough sketch for scaling up natural gradients to deep neural networks that appears to be easy to automate just like automatic differentiation. I think there must be a flaw ...
5
votes
2
answers
146
views
What do you do in regression if $(X^\text{T} X)^{-1}$ does not exist?
Assume we want to solve $X z=b$ for $z$ where $X$ is a non-square matrix and $z$ and $b$ are column vectors. In case the system is overdetermined, we have no solution and can look for a solution ...
- 217
2
votes
1
answer
147
views
Invertibility of the Gram matrix of a convex combination
Let's assume two real valued matrices $A,B\in R^{w\times d}$ for which $d>w$ and they both have full (column) rank.
I am interested in the invertibility of the Gram matrix $$H(t):=(A+t(B-A))(A+t(B-...
- 61
2
votes
2
answers
384
views
How to obtain least squares when $X^TX$ cannot be inverted
This work is all theoretical and for school, so we were only provided this information to work with, no actual y values. I have a simple linear model I have been asked to translate into a matrix, ...
- 155
0
votes
0
answers
22
views
how can an inverse correlation matrix have <1 in diagonals. the correlations x1, x2 are about 0.5 [duplicate]
The inverse correlation matrix was calculated with the minverse function. The results show that in the diagonal the values are less than 1. how is this possible? the x1,x2 independent variables are ...
- 1
3
votes
1
answer
309
views
Inverse of the outer product of some vectors with their transpose
Assume i have $n$ 3D unit vectors $v_s$, with different values.
Then i define a matrix $T$ as:
$$
T = \frac{1}{n} \sum_{s=1}^{n} v_s \times v_s'
$$
where $v_s$ are $3\times 1$ vectors and therefore $...
- 53
1
vote
1
answer
402
views
Condition for covariance matrix to be non-invertible
Context: I'm working on a machine learning problem where I'm using multivariate normal likelihood which requires calculating determinant and inverting the covariance matrix. I'm trying to generate ...
- 831
1
vote
1
answer
1k
views
How to derive an inverse of Gaussian Kernel
As an example, say I have a function (Gaussian process kernel):
$$K(x_i, x_j) \equiv \exp(-\alpha |x_i-x_j|^2)+\beta \delta_{ij}$$
Is there a way to analytically express $K^{-1}(x_i,x_j)$, s.t. ...
- 281
3
votes
1
answer
164
views
How to prove $(P^{-1} + B^T R^{-1} B)^{-1} B^T R^{-1} = PB^T(BPB^T + R)^{-1}$
It is Equation C.5 from https://www.seas.upenn.edu/~cis520/papers/bishop_appendix_C.pdf
I tried right multiply both sides with $(BPB^T + R)$, but not sure how to continue from there.
- 1,176
1
vote
1
answer
380
views
How can nuisance parameters in Fisher matrix can deteriorate the useful constraints?
I have a Fisher matrix $F$ which has the matrix blocks form like this :
$$
F=\begin{bmatrix}
A & B\\
C & D
\end{bmatrix}
$$
The block $A$ is the most important block, in the sense the ...
user226073
0
votes
1
answer
40
views
Problem dealing with OLS estimator
I'm an econometrics student and I'm having a little trouble with lineal algebra.
I have seen that the OLS estimator, given the following regression in matrix form:
$$
y=X \beta+u
$$
Is:
$$
\hat{\beta}=...
- 3
1
vote
0
answers
14
views
How to simplify fallowing martrices' Expected value elements in the equation
I have a matrix equation with four variables inside, $x^1_{00}$, $x^1_{tt}$ and $x^2_{tt}$, $x^2_{tt}$. $x^1_{00}$, $x^1_{tt}$ come from the same distribution , they are only shifted by timelag $t$. ...
- 567
0
votes
0
answers
133
views
Distribution of solution to linear system
I have a random symmetric matrix $ A \in \mathbb{R}^{M \times M}$, and random vector $b \in \mathbb{R}^M$. I also have access to expressions for the mean and variance of each element of $A$ and $b$ (...
0
votes
0
answers
367
views
When is the Optimal weighting matrix in GMM singular?
currently I am trying to estimate a simple linear regression:
\begin{equation}
y_t = X \beta + \varepsilon_t,
\end{equation}
where I try to find 4 coefficients and one specific predictor is an ...
0
votes
0
answers
23
views
Singularity of a modelmatrix that consists of one predictor raised to multiple powers
I was playing around a little bit with linear regression models in R and wanted to try out whether I can get a perfect fit for a linear model where I have a response $$Y =
\left(
\begin{array}{c}
y_1\...
- 87
1
vote
1
answer
800
views
Mercer's theorem and eigenfunctions
Consider a kernel $K$ which satisfies the conditions of Mercer's theorem. We know that $K(x,y) = \sum\limits_{i=1}^{+\infty}\lambda_ie_i(x)e_i(y)$ where $\lambda_i$ and $e_i$ are the eigenvalues/...
- 341
0
votes
0
answers
67
views
Finding inverse matrix $(X'X)^{-1}$ with $X$ as design matrix [duplicate]
I'm relatively new to all this and I am trying to figure out how I can derive the matrix $(X'X)^{-1}$ when I have given $x_1, x_2, x_3$ and $y$. $X$ is the design matrix in that case but not sure how ...
3
votes
1
answer
307
views
linear regression and obfuscation matrix - just not clicking for me?
Here is the basis of my question.
We have an exercise to take a data set of made up insurance claims and train a linear regression model to predict future claims; however, because it is sensitive data,...
- 33
3
votes
1
answer
202
views
completing a square
If I have a density function of the form $p(x) \propto \exp(−q(x)/2)$ where $q(x)$ has
the following quadratic function
$$q(x)=x^Tx+y^Ty-[x^TA+y^TB][A^TA+B^TB+\beta\mathbb{I}]^{-1}[A^Tx+B^Ty]$$
where $...
- 117
2
votes
1
answer
16
views
What is an example of a systems problem that can be solved by either 'matrix inversion' or an 'iterative procedure' to arrive at the same result?
I recently heard over a radio program (in French) that a given problem - can't recall it exactly, but it involved solving large systems - could be solved "through matrix inversion" or "...
- 1,228
22
votes
1
answer
8k
views
Relationship between Cholesky decomposition and matrix inversion?
I've been reviewing Gaussian Processes and, from what I can tell, there's some debate whether the "covariance matrix" (returned by the kernel), which needs to be inverted, should be done so ...
- 3,520
3
votes
1
answer
343
views
What dimensionality reduction methods allow a lower dimensional reconstruction of the original data besides PCA via invertible transformations?
In eigenfaces, one used the inverse transformation PCA is capable of doing to reconstruct the low dimensional face image.
In tsne one may not reconstruct the original dataset to produce something akin ...
user294054
-2
votes
1
answer
208
views
Why doesn't $(e^{A})^{-1} = e^{-A}$ hold for a symmetric matrix in Python?
$e^A$ is just the $A$ matrix with all of its elements exponentiated, called a matrix exponential.
It follows that the inverse $(e^{A})^{-1} = e^{-A}$ for square matrices, although I could find nothing ...
- 4,049
0
votes
1
answer
901
views
Proving that the hat matrix is unchanged even when the predictors are multiplied by constants
I know that the hat matrix $H = X(X^T X)^{-1} X^T$, and that $\hat{Y} = HY$. When we have some non-zero constants that we multiply each respective predictor by, which just multiplies every column in ...
- 3
2
votes
0
answers
35
views
Why is it not possible to simplify $b=(X'X)^{-1}X'y$ [duplicate]
Why is the following not possible:
$$
b=(X'X)^{-1}X'y = X^{-1}(X')^{-1}X'y=X^{-1}y
$$
While this term $(AB)^{-1}=B^{-1}A^{-1}$ applies to any two matrices as long as both are of full rank and are $nxn$...
- 41
4
votes
1
answer
1k
views
Moore Penrose Pseudo-Inverse Fast Algorithm in R [closed]
I want to apply Moore Penrose Pseudo-Inverse on my matrix, which is a 20,000 * 20,000 symmetric matrix with rank 19,999. I found ginv() function from the ...
- 107
0
votes
1
answer
1k
views
How to do multiple linear regressions with overlapping predictors efficiently
I'm using covariance matrices to do a linear regression. I have the same number of predictors and dependent variables (3000 of each). For each $i$, $X_{i}$ and $Y_{i}$ have a kind of symmetry.
I ...
0
votes
1
answer
180
views
NN type/architecture needed for inverse covariance matrix approximation
The idea is to construct a neural network (NN) that takes N series of financial returns as input and returns the (approximation of the) inverse of the sample covariance matrix (an N times N matrix). I ...
- 111
1
vote
0
answers
995
views
Normalization for solving linear equations
Suppose I want to solve a linear equation system in the form of
$$A x = b$$
to get $x$, where $A$ are $n$ by $n$ matrix and $b$ is $n$ by 1 vector. Is there any normalization procedure necessary ...
- 2,083
5
votes
1
answer
2k
views
Updating the inverse covariance matrix after deleting the i-th column and row of the covariance matrix
Suppose I have a covariance matrix $A_{n\times n}$ and $A^{-1}_{n\times n}$ is its inverse. Then I randomly exclude the $i$-th row and column, where $1\le i\le n \in \mathbb{N}$, obtaining the new ...
- 352
7
votes
2
answers
1k
views
Least Squares removing first $k$ observations Woodbury formula?
Given the matrix $X_{n,p}$ from the least squares problem $$ \mathbf{X} \cdot \mathbf{\beta} = z $$
Where the normal equation is:
$$ \mathbf{\hat{\beta}} = \left(\mathbf{X}^T \mathbf{X}\right)^{-1} ...
- 205
0
votes
0
answers
545
views
Squared Multiple Correlation (SMC) of my correlation matrix tend towards 1. How to interpret this?
In order to be able to conduct exploratory factor analysis, I want to carry out parallel analysis to determine the number of factors to be extracted. To do so, I want to extract the eignevalues of the ...
- 561
1
vote
1
answer
541
views
Upper bound trace of inverse of covariance matrix
Let C be the covariance matrix from any normal distribution. If the trace of C is upper-bounded by a constant k (i.e., tr(C)<=k), can I find an upper bound for the trace of the inverse of C (i.e., ...
- 73