Questions tagged [linear-algebra]
A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.
673
questions
8
votes
2
answers
8k
views
How to show this matrix is positive semidefinite?
Let
$$K=\begin{pmatrix}
K_{11} & K_{12}\\
K_{21} & K_{22}
\end{pmatrix}$$
be a symmetric positive semidefinite real matrix (PSD) with $K_{12}=K_{21}^T$. Then, for $|r| \le 1$,
$$K^*=\...
- 129
2
votes
1
answer
1k
views
Deriving the maximum likelihood of REML for linear mixed model
Consider the linear model $Y = X \beta + e$, $e \sim N(0, V(\theta))$, where $Y$ is a $n \times 1$ vector, $X$ is the $n \times p$ full rank design matrix, $V(\theta)$ is the covariance matrix. I drop ...
- 88
6
votes
2
answers
2k
views
Why do the leading eigenvectors of $A$ maximize $\text{Tr}(D^TAD)$?
Given a matrix $X\in\mathbb{R}^{m\times n}$, I am trying to maximize $\text{Tr}(D^TX^TXD)$ over $D\in\mathbb{R}^{n\times l}$ ($n<l$) subject to $D^TD=I_l$, where $\text{Tr}$ denotes the trace, and $...
- 1,249
2
votes
0
answers
60
views
How to calculate a perfect separation in a d dimensional space between n class?
Assuming in a d-dimensional space, we have samples from n class.
The best way of separating samples of each class from each other is to have the samples from each class as far as possible from every ...
- 758
1
vote
0
answers
150
views
Is there a solution for Canonical Correlation Analysis on large sparse matrices?
I'm trying to run CCA over two views which are sparse matrices. The two views are very high dimensional (e.g. 300k, 400k) with 1m samples.
CCA needs the input views to be zero mean but I won't be ...
- 253
2
votes
0
answers
50
views
Matrix Decomposition $ B = B^* + \sum_{i>1}\lambda_i B_i$ [closed]
I recently encountered the following decomposition in a paper, may I know whether anyone on CV happened to know the name of the decomposition and a proof of it?
$B$ is a regular stochastic matrix and ...
- 29
1
vote
0
answers
79
views
Using Principal Component Analysis Based on Sample Covariance? [But really how to get these eigenvalues?] [closed]
I have the following sample covariance matrix;
$$
S=
\begin{pmatrix}
16 & 10 \\
10 & 25
\end{pmatrix}
$$
Now to use PCA I know I need to find the eigenvalues and ...
- 891
1
vote
1
answer
428
views
Prove $A(A+B)^{-1}B=B(A+B)^{-1}A$
I have this equality,
$A(A+B)^{-1}B=B(A+B)^{-1}A$
and the question specifically only states that $A+B$ is nonsingular.
I have looked at this many ways but the only I can see it working is if $A+B$ ...
52
votes
7
answers
32k
views
Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?
I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
- 663
1
vote
0
answers
207
views
Calculating Covariance Matrix from pooled sum of squares and products about the mean?
I am given the following (labeled as the "pooled sum of squares and products about the mean"
$$
\begin{matrix}
x_1 & x_1 & x_2 & x_3 & x_4\\
x_1 & ...
- 891
2
votes
0
answers
120
views
How can one design a polynomial function that really does require higher order terms to approximate it well?
My goal is to design an experiment such that only a high order polynomial function can approximate the target polynomial function.
I've been trying to approximate a polynomial function in 1D $f_{...
- 6,866
1
vote
0
answers
72
views
Prove that $f^{-1}$SSE in SSE = SST - SSTR is the pooled unbiased estimate of the common covariance $\Sigma$ where f = n-J
At the onset I know I need to prove that $E(f^{-1}) = \Sigma$
I just don't know how to go about doing this.
I know $\frac{SSE}{\sigma^2}$ follows a $\chi^2$ distribution with, I believe, n-J ...
- 891
1
vote
0
answers
621
views
How does evenly spaced X simplify the linear regression?
What effect does it have on the H-matrix or the regression line?
- 461
5
votes
1
answer
229
views
Confusion regarding the dimensions of matrices in the formula for SVD
I wanted to understand Singular Value Decomposition (SVD) hence consulted a few resources. In general I came across 2 different forms of SVD and hence got confused as to which one is correct or ...
- 167
3
votes
2
answers
977
views
Calculating the log-determinant of large, sparse covariance matrices
As part of calculating the log-density of a very large multivariate normal distribution, I need to evaluate the following log-determinant:
$$
f = \mathrm{ln}\left( \mathrm{det}(\mathbf{XAX}^\top + \...
- 613
0
votes
1
answer
618
views
What is wrong with my computation of projections on the first principal component?
Following the links in parenthesis (link1, link2) I wrote a bit of code in MATLAB to simulate a PCA.
After running the code and plotting the results I obtain this ...
- 101
4
votes
1
answer
236
views
matrix form of dummy expansion?
It's common in many applications to interact a set of numeric variables with a factor variable. The factor variable is a convenient abstraction for what is actually a matrix of dummy variables. For ...
- 13.3k
2
votes
0
answers
463
views
Linear Algebra and NumPy - How array broadcast translates to matrix operations?
I am familiar with Python's NumPy library and its broadcasting feature that allows to perform operations with vectors and matrices of different sizes. I might be missing the mathematical connection as ...
- 569
1
vote
0
answers
178
views
PRML: Why the elements of y(x) sums to 1? (Chapter 4.1.3)
I could not understand the reason why PRML (Pattern Recognition and Machine Learning by Christopher Michael Bishop) says:
An interesting property of least-squares solutions with multiple target ...
- 111
1
vote
0
answers
34
views
Random samples for particular cov(X,X), and cov(X,X.^2)
I have an empirical sample of multivariate data and I want to generate surrogate data X that matches certain characterizations of my data:
cov(X,X) = A, and
cov(X,X.*X) = B
where X.*X is the Hadamard ...
alexithematic
8
votes
2
answers
2k
views
Prove that $\mathrm{Cov}(x^TAx,x^TBx) = 2 \mathrm{Tr}(A \Sigma B \Sigma) + 4 \mu^TA \Sigma B \mu$
Suppose $\vec x \sim N(\vec \mu, \Sigma)$ is multivariate normal. I want to see that
$$\mathrm{Cov}(\vec x^TA\vec x,\vec x^TB\vec x) = 2 \mathrm{Tr}(A \Sigma B \Sigma) + 4 \vec \mu^TA \Sigma B \vec \...
- 821
2
votes
1
answer
390
views
Combined PCA and Quantization
I did some PCA on a dataset to reduce the size without compromising too much on their actual information.
However, I learnt that quantization is also an effective technique to do this. I guess it is ...
- 115
1
vote
0
answers
321
views
Are the Magnitudes of PCA Weights Reflective of their Importance?
I have a data matrix with N entries and n features.
I wanted to find which features are the most important in explaining the data.
So, I started with PCA, but PCA components are linear combination ...
- 1,355
1
vote
1
answer
1k
views
Conditional multivariate Gaussian distribution
I am trying to understand how 'completing the square' method is used to determine mean and co-variance of multivariate Gaussian distribution but I am not able to make much sense of the following step
$...
- 21
0
votes
0
answers
60
views
Solving weights in least squares to reduce distance to another vector
I'm using a procedure of WLS to find some values in a reduced space that actually generate a vector that is close to my target vector. That is, I have WLS:
$\beta=(X^T W X)^{-1} X^T W Y$, and I am ...
- 111
3
votes
1
answer
2k
views
Closed form and gradient calculation for linear regression
Given is a linear regression problem, where we have one training point, which is 1-dimensional: $x \in R_{>0}$ and the corresponding output, $y \in R$. We duplicate the feature, such that we have ...
user24544
3
votes
0
answers
4k
views
Proving that kernels are closed under addition and scalar multiplication
We have to show that given k_1 and k_2 are both kernels, the sum k_1 + 2*k_2 is a kernel as well.
I am attempting to show this by using Mercer's theorem:
$k(x,y) = <a(x),a(y)>$.
Showing that $...
- 1,511
2
votes
0
answers
546
views
Completing the Square for a Matrix Normal Distribution
Completing the square in the context of a multivariate gaussian distribution is pretty straightforward (here is a good explanation).
But what if we are talking about a matrix normal distribution?
...
- 21
2
votes
1
answer
91
views
Minimize weighted $l_1$ loss
It is known that the minimizer of $l_1$ loss function is the sample median, i.e.,
$$\operatorname{argmin}_{x_0}\sum_{i=1}^{n}|x_i-x_0| = \operatorname{median}(x_1,...,x_n)$$
assume that I have a ...
- 829
0
votes
0
answers
11
views
Relationships between sub-samples with limited information
I'm doing an analysis where I've been able to assemble the following information about a population, its overall conversion rate (so, the percent of the members of that population who go on to make a ...
- 173
2
votes
1
answer
627
views
Least Squares Derivation
I come from physics and would like to derive the chi-square function given by the Particle Data Group:
\begin{equation}
\chi^2 (\boldsymbol\theta) = (\boldsymbol y-\boldsymbol\mu(\boldsymbol \...
- 171
8
votes
1
answer
30k
views
Are 1-dimensional numpy arrays equivalent to vectors? [closed]
I'm new to both linear algebra and numpy, so please bear with me. I'm taking a course on linear regression, where I learned that we can express our hypothesis as $\theta^TX$ where $\theta$ is our ...
user153009
4
votes
1
answer
203
views
Identification restrictions in Structural VECMs
I am studying Structural VECM models using Lutkepohl's book. VECM process has the following Beveridge-Nelson MA representation:
$$y_t = \Xi\sum_{i=1}^{t}u_i + \sum_{j=0}^{\infty}\Xi_j^*u_{t-j} + y_0^*,...
- 1,159
0
votes
0
answers
2k
views
Confused about finding expectation of vector multiplied by matrix
I am little confused regarding finding expectation of vector multiplied matrix. X is vector having mean $m_x \in \mathbb{R}^{N\times 1}$ and variance vector $s_x \in \mathbb{R}^{N\times N}$. $\phi$ is ...
4
votes
1
answer
970
views
Know which variables are linear combinations of others from covariance (correlation) matrix
Suppose I have the covariance and correlation matrices for several variables. I know one of the variables is almost a linear combination of the others. Is there any characteristic about the ...
- 167
4
votes
0
answers
528
views
Linear transformation of a random vector with pseudo-inverse
If $$ \mathbf{X} = (X_1,..,X_n)^t$$ is a random variable drawn according to a probability density function (pdf) $$ f_{X_1,...,X_n}(x_1,...,x_n) $$ then $$ \mathbf{Y} = A\mathbf{X} = (Y_1,..,Y_n)^t$$ ...
- 426
2
votes
1
answer
379
views
Weight matrices computation in attention-based encoder of Deep Learning NLP
On page 4 of this research paper titled A Neural Attention Model for Sentence Summarization , it is mentioned the attention-based encoder is determined by the ...
- 143
2
votes
1
answer
55
views
Completing the Square (Bivariate)
I am running a bivariate regression with fitted equation given by
$$\hat{P} = b_0+b_1X+b_2Y+b_3X^2+b_4Y^2+b_5XY$$
It is easy to obtain the 6 coefficients given $X,Y$ and $P$. However, I want to ...
- 1,277
0
votes
1
answer
3k
views
How do PCA/SVD Decorrelate the variables [duplicate]
I understood the "technique" for doing SVD and PCA.
However, I couldn't understand these two claims:
PCA/SVD decorrelate the variables
They do so using orthogonal transformations
For 1, PCA does ...
- 1,355
0
votes
1
answer
28
views
How to get N of product into equation (or so)
Can somebody explain how this works or direct me to a rule that describes this simplification? I understand the whole process but how the 'getting the $n$ in front to $\lambda$' works is unclear to me....
- 155
1
vote
0
answers
41
views
What should be rank of matrix with compositional data variables?
I'm doing compositional data analysis on 'Baysite' dataset under 'compositions' package in r. Aim is to find nature of the relationship of its permeability to the mix of its four ingredients:
A: ...
- 51
5
votes
2
answers
629
views
How to determine if a matrix is close to being negative (semi-)definite?
Questions:
1. Is there a simple, interpretable way to
determine the distance/closeness of a matrix to being not
positive (semi-)definite?
2. Alternatively: how can I systematically create ...
- 684
5
votes
1
answer
202
views
What is the relationship between Mohr's Circle and Principal Component Analysis?
While I was studying PCA, I was told that it is related to Mohr's Circle. I don't know what that means. I don't know if they are related or not. I was just told, so I want to make sure here.
If they ...
- 1,673
3
votes
2
answers
3k
views
what does scaling the normal vector of a plane (/hyperplane) mean?
I understand that, scaling (multiplying or dividing by a constant) the normal vector of a plane, does not affect the plane itself.
But what happens when we do so? Are we zooming in or out of the space,...
3
votes
1
answer
14k
views
Why is the sum of eigenvalues of a PCA equal to the original variance of the data?
Can someone please give or point to a proof? Can't seem to find a post that address this directly.
- 330
3
votes
1
answer
132
views
Solving system of linear equations (to determine a boundry)
I'm puzzeled how to programmatically (in R) solve the following linear system:
Given $\mathbf{R} \in \mathbb{R}^{n \times n}$, $\mathbf{R}^{-1}$, and a constant $c$ what is the solution to $\mathbf{u}...
- 1,004
2
votes
0
answers
192
views
Algebraic derivation of canonical correlations
In a paper from 1936, Harold Hotelling (access on JSTOR) defined the concepts of canonical correlations and canonical variates for two sets of variates.
In pages 327 and 328, he precisely derives ...
- 477
1
vote
1
answer
67
views
Matrix operations in business
I have a chemical engineering background but I have decided to go into the corporate world instead of industry. I now work in payment streams and have been doing data science and optimizations. What I ...
- 41
3
votes
1
answer
3k
views
Help needed on algebraic steps for Maximum Likelihood Estimation of Multivariate Normal Distribution?
The negative loglikelihood is as follows:
$$\dfrac{nd}{2} \log 2\pi + \dfrac{n}{2} \log |\Sigma| + \dfrac{1}{2}\sum_{i=1}^n(x_i-\mu)^T\Sigma^{-1}(x_i-\mu) \tag{1}$$
If I take differentiation with ...
- 1,673
1
vote
2
answers
1k
views
How can I map data to lower dimension?
I am trying to learn data in higher space into lower space. To have a clue, I'd like to know how to transform the data in the image below into a lower dimension preserving the structure. Hope to hear ...
- 1,673