Questions tagged [linear-algebra]
A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.
657
questions
0
votes
1
answer
32
views
Covariance matrix square root
Consider a random variable $r_t$ which represents the return of an asset at time t. In the univariate case, we just consider $r_t$ to be the return of a single security at time t. Generally, we assume ...
- 5
0
votes
0
answers
10
views
trying to create a cache dataset in monai: LinAlgError: SVD did not converge [closed]
This is the error that I am getting:
...
- 1
7
votes
2
answers
673
views
Is the expectation of a random vector multiplied by its transpose equal to the product of the expectation of the vector and that of the transpose
I'm taking a course in advance statistics and we have to prove whether the following expression is true: $E[zz^T]=E[z]E[z^T]$. I am assuming it is not, since the formula of the covariante matrix is $...
- 101
0
votes
0
answers
21
views
Covariance matrix for data
Assume $n*p$ data matrix $X$, where n is the number of observations and p is the number of features. We are interested in the covariance among features.
I have seen notations where covariance matrix ...
- 31
0
votes
0
answers
7
views
Leverage and One-Out Model
I am currently trying to prove the following theorem, found within the course material from a Graduate Regression module. For context, here $X$ is the usual full-rank design matrix, estimators are ...
0
votes
0
answers
9
views
Heteroscedastic Asymptotic Variance Simple Transformation
Let's denote the asymptotic variance under heteroscedasticity as:
$$\hat{\text{Avar}}(\hat{\beta}) = 1/N * \left(\frac{1}{N} \sum_{i}{x_i x_i'}\right)^{-1} \left(\frac{1}{N} \sum_{i} \hat{u}_i x_i x_i'...
- 121
3
votes
1
answer
45
views
Spiked tensor decomposition vs canonical polyadic decomposition
What are the similarities and differences between Spiked tensor decomposition and canonical polyadic (CP) decomposition?
My understanding is that CP decomposition aims to find a low-rank approximation ...
- 43
0
votes
0
answers
17
views
How to adjust similarity scores by removing the influence of a common vector?
I have a similarity score function, $s(x,y)$. I know that I have two items that I'm trying to compare the similarity of, but both are based on the same template. How would I remove the template from ...
- 101
2
votes
1
answer
106
views
How to sample efficiently from an inverse Wishart distribution?
I am trying to understand the code from pybasicbayes, which defines the following function to sample from an inverse Wishart:
...
0
votes
0
answers
15
views
Estimate null hypothesis for correlation of linear combinations of variables?
Setting up the problem
Suppose I have a variable $x$ of length $n$ and I have another $p$ variables $y_1, y_2, \dots, y_p$, where $y_i$ is also of length $n$.
Based on the y's, I can make a linear ...
- 43
0
votes
1
answer
87
views
Difference between conducting PCA on $XX^\top$ vs $X^\top X$?
PCA: For a given set of centered data $\mathscr D =\{x_i\}_{i=1}^N \subset \mathbb R^d$, i.e. the data has $N$ examples with dimension $d$. Then the principal directions of PCA can be obtained from ...
- 1
0
votes
0
answers
11
views
How can a linear autoencoder with $h=1$ hidden unit reconstruct any rank 1 matrix?
I've had this as a homework problem as a true or false type of question and I'm trying to wrap my head around why this is true.
Is the reason simply represent each datapoint as a scaled version of a ...
- 290
1
vote
1
answer
75
views
Is it wrong to view convolution as template matching?
I am reading about the convolution operation but I can't see how it can be seen as template matching.
Suppose that we convolve the input $\mathbf{X}$:
$$
\begin{bmatrix}
1 & 0 & 0 & 0 \\
0 ...
- 452
1
vote
0
answers
27
views
Mediation study help
I have a simple order of operations question with sum products, I want to calculate the natural direct effect (NDE) for x and x prime on data similar to those below. My simple question is:
Do I first ...
0
votes
0
answers
28
views
Project Matrix Data onto PCA Space
Given a matrix A with dimension m by n which is a matrix of m samples and n features. Also SVD of Matrix A = U * Sigma * V^T, How to project matrix A onto its k principal components with only U and ...
- 31
0
votes
0
answers
38
views
"Dimensionality" of y vs the span of our Independent variables In regression
Question: I want to clarify my understanding of OLS regression using Matrix Algebra.
Let's assume we have 2 different independent variables $x_1$ and $x_2$.
Our 'model' will be the plane that lives in ...
- 121
0
votes
0
answers
16
views
Is it possible to apply matrix decomposition to a vector, injecting additional information to UV decomposition?
As I am reading about recommender systems in Machine Learning, UV decomposition caught my eye (click for an explanation or see below).
So I have two questions:
Question 1: what are the drawbacks of ...
- 1
3
votes
2
answers
68
views
What does the "direction of data" mean in the context of Principal Component Analysis?
I am reading the Introduction to Linear Algebra 5th edition, section 7.3: Principal Component Analysis. The section contains the following sentence
The first eigenvector $u_1$ of $S$ points in the ...
- 572
3
votes
0
answers
51
views
Relationship between SVD of Matrix and SVD of same Matrix with deleted entries (Matrix can be Adjacency Matrix of a Graph)
Could somebody direct to me to some literature dealing with this issue. So we have $X = U\Sigma V^{T}$ and we have $M \odot X = U^{'}\Sigma^{'}V^{'^{T}}$ with
\begin{equation}
M_{i,j} = \begin{cases}
...
8
votes
3
answers
1k
views
Is the sum of the diagonal elements of a covariance matrix always equal or larger than the sum of its off-diagonal elements?
For any given covariance matrix, will the sum of the diagonal elements always be bigger than the sum of the off-diagonal elements?
Let $\sigma_i$ be the standard deviation of the $i^\text{th}$ term of ...
- 175
3
votes
0
answers
40
views
Johansen test: why testing for the algebraic multiplicity of 0 and not for the nullity?
From what I already know about the Johansen test, it tests the rank of the VAR matrix (in error correction form) through steps testing whether every eigenvalue is signifincantly different from 0 (...
- 151
1
vote
0
answers
39
views
Is there a high probability bound of quadratic forms?
I am wondering about the following: For a symmetric matrix $A \in \mathbb{R}^{n \times n}$ and vector $x \in [-1,1]^n$, if $X$ is a random vector in $\mathbb{R}^n$ such that w.h.p. $X_i \not\in [-1,1] ...
- 33
2
votes
0
answers
35
views
Johansen test accepts first null hypothesis but would reject last one
Suppose that we perfrom a Johansen test over three I(1) variables that give us these results through the maximum eigenvalues statistic:
as you can see, we accept the null hypothesis in the first step ...
- 151
1
vote
0
answers
25
views
Least Squares Regression with Length and Density of input data
I have a collection of data that I expect to be linear but has a unknown amount of noise to the data. Initially I wanted to use the least squares regression line to determine if the slope, y axis, and ...
- 11
0
votes
1
answer
210
views
Prove that 2nd order polynomial kernel is positive semi-definite
I'm trying to prove that the 2nd order polynomial kernel, $K(x_i, x_j) = (x_i^Tx_j + 1)^2$ is a valid kernel which satisfies the following conditions:
K is symmetric, that is, $K(x_i, x_j) = K(x_j, ...
- 103
0
votes
0
answers
37
views
How to do column weighted truncated SVD?
I have an unusual case where I need to combine two vector spaces but weight one more than the other. Rather than discussing my specific use case, it's likely easier to imagine we trained two word2vec ...
- 298
0
votes
0
answers
23
views
Distribution of power two of a term
Assume we have matrices $A_{n\times n}$ and $\Delta_{n\times n}$ and know that:
$$A^{-1}_{i.}\Delta_{.j}\sim N(0,\,A^{-1}_{i.}\,\Sigma_j\,[A^{-1}_{i.}]^\top)\,,$$
where $\Delta\sim N(0,\,\Sigma_j)$, $...
- 601
1
vote
0
answers
83
views
Covariance matrix and vec operation
I have the following matrix:
$ U = \begin{bmatrix}u_{1} \\ u_{2} \\ \vdots \\u_{T}
\end{bmatrix}$
where $u_i$ are $k \times N $ matrices and
$E[vec (u_i) vec (u_i)'] = \Sigma_i \otimes I_k $
for $...
- 189
1
vote
1
answer
287
views
Calculate the output of a Neural Network
I have the following problem:
Here is my approach:
With the activation function: $F(x) = x^2 + 2x + 3$, we can calculate the activation of the two units of the second layer by: $a_1^2 = F(w_{13}\...
- 11
0
votes
0
answers
66
views
Calculating initial values of a Linear Regression Model
I have the following problem
Given the training data for a linear regresison problem as follow:
Input
Output
0
0
1
2
-1
-2
2
3
After the first iteration, the values of the two coefficients are ...
- 11
1
vote
0
answers
98
views
Deriving vectorized back propagation
I'm trying to derive vectorized backpropagation from mostly first principles, but I'm having
trouble marrying how
this
paper explains backpropagation with the derivative of a loss function with ...
- 11
2
votes
0
answers
103
views
Intuition behind rank of covariance matrix and testing hypotheses
I am trying to acquire some intuition about testing multivariate hypotheses where the test statistic involves inverse covariance matrix. As an example, suppose we have a $p$-variate random vector that ...
- 64.6k
1
vote
0
answers
47
views
Doubts about Gradient and Matrices derivatives
Studying NN and I wanted to grasp from scratch the theory of gradient descent and matrices derivatives, so I took a simple scenario and tried to apply gradient ascent and see if everything made sense.
...
- 317
2
votes
0
answers
34
views
Eigenvalues of a block matrix
Let $\bar{\lambda}$ be the smallest eigenvalue of $$M=\Omega^{-1/2}Y'Z(Z'Z)^{-1}Z'Y\Omega^{-1/2}$$, where $\Omega$, $Y$, and $Z$ are $(l\times l)$, $(N\times l)$, and $(N \times k)$ matrices, ...
- 191
2
votes
0
answers
53
views
What does $\times_i \Sigma_i$ mean?
I cannot for the life of me figure this out. The context is from game theory (source: Game Theory by Fudenberg, Tirole):
... the space of mixed strategy profiles is denoted $\Sigma = \times_i \...
- 21
1
vote
1
answer
150
views
How to calculate dimension of weight matrix for a vanilla RNN
Sorry if this is kind of a dumb question but I'm taking the NLP course from Andrew Ng on Coursera and I don't understand how to arrive at the correct answer for a question pertaining to calculating ...
- 11
3
votes
0
answers
105
views
Good math textbook on the intersection of linear algebra and probability
I've been puzzling over some linear algebra problems and going through some old textbooks, but they rarely talk about prob stats things like expected value, std, or mean.
Is there a good textbook / ...
5
votes
1
answer
217
views
Procrustes Problem for rank-deficient input
It is well known that the solution to the orthogonal Procrustes problem
$$
\textrm{arg min}_{\Omega \in \text{SO}(n)} ||Y - \Omega X||^2_2,
$$
can be expressed in terms of an SVD of the covariance ...
- 247
2
votes
1
answer
133
views
Generate two random correlation matrices which share equal correlations
My setting is, I want to simulate a data set in two conditions, e.g. control and disease. I want them to share mostly the same correlations except some should be different to simulate a "signal&...
- 21
1
vote
0
answers
66
views
How to reconstruct Cartesian coordinates from a Gram matrix?
I read this article and am wondering why we can reconstruct Cartesian coordinates from a Gram matrix generated by taking dot product of the distance from the origin.
They had a Euclidean distance ...
- 11
0
votes
0
answers
114
views
Show by derivation why both Order and Rank conditions are needed for identification with instrumental variables
Why are both Order and Rank conditions needed for identification with instrumental variables?
- 23
1
vote
1
answer
48
views
QR Iteration convergence
Assuming $\mathbf x_1$ and $\mathbf x_2$ are eigenvectors of matrix $\mathbf A \in \mathbb{R}^{2\times 2}$, there is $$\left(\mathbf I-\mathbf x_1\mathbf x_1^\top\right)\mathbf A\left(\mathbf I-\...
- 11
2
votes
1
answer
139
views
How does the scale of the target value effect the Gaussian Process performance?
I am currently working with Gaussian Process to make a surrogate model for temperature prediction. A question arose when I was thinking to make result plots in Celsius instead of Kelvin. I suspect if ...
- 43
1
vote
0
answers
48
views
SVD to find the common factors
Let's say I have a time series $R\in \mathbb{R}^{T\times M}$ where $T$ shows the dates and $M$ is the number of variables. I do an SVD on it and obtain $R=U\Sigma V^\top$, where $U\in \mathbb{R}^{T\...
- 601
1
vote
1
answer
44
views
Picturing ellipse plot $(x,y) = \left(\cos(\theta+\frac{d}{2}),\cos(\theta-\frac{d}{2})\right)$
The following paragraph taken from: A Graphical Display of Large Correlation Matrices
I am trying to produce a plot with the provided equation above for the ellipse.
For example, given I have random ...
- 23
1
vote
1
answer
28
views
How can I use matrix operation to represent the following result?
I need to derive a pairwise offset matrix from the two known data array $X_{n \times p}$ and $Y_{m \times p}$, in which the offset are derived from values at corresponding locations $x_{ij}$ and $y_{...
- 511
2
votes
1
answer
54
views
Source needed: Why vector representation are ideal for statistical models
I am just looking for a scientific source, either a paper or book, that I can refer to when talking about the fact that the ability to represent data in vectors is a desired property.
Reasons can ...
- 882
1
vote
0
answers
74
views
Solution of Ax - b (A is square, symmetric, positive-definite )with iterative methods
I was hoping someone could help me verify my understanding about solving for a system of equations iteratively.
I have a square symmetric, positive-definite matrix $A$ and a vector $b$ and I want to ...
- 963
1
vote
1
answer
52
views
Proof of linear independence of dummy encoding
I understand one-hot encoding is linearly dependent and if I drop one column, it would become linear independent, but I don't know how to prove it, can someone give me a mathematical proof.
- 11
3
votes
0
answers
127
views
Is there an outer product counterpart for the Covariance?
Covariance
The covariance of two quantities $X$ and $Y$ within a population, $Cov(X,Y)$, is symmetric and bilinear. It is also true that $Cov(X,X) \ge 0$. So, clearly $Cov(X,Y)$ qualifies as an inner ...
- 31