Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Understanding a particular rotation strategy

I am trying to replicate the rotation strategy (for factors) in a paper. I will try to make as clear as possible what I don't understand. Let the model be: $X = F\Lambda + \epsilon$, where $F$ is Tx3 ...
1
vote
0answers
19 views

Decomposing Gradient Decent Error in Eigenvector Space

I'm going through Why Momentum Really Works and am unable to understand the following line in the article. "By writing the contributions of each eigenspace’s error to the loss $$f(w^{k})-f(w^{\star})=...
0
votes
0answers
47 views

The second principal component of PCA [closed]

Anyone have an idea of how to prove this question (the last sentence above)?
3
votes
1answer
34 views

Triangular Markov chain question

A triathlon consists of $3$ disciplines: swimming, cycling and running. A triathlete does a training session every day. However he doesn’t want to pay for professional coaching advice so instead his ...
2
votes
0answers
78 views

Row similarity in matrix vs in different factorizations

Suppose an arbitrary $m \times n$ matrix $M$ and the factorizations: Arbitrary: $M = U_a V_a^T$, where $U_a$ is $m \times k$, $V_a$ is $n \times k$ ($k < m,n$), and $rank(U_a)=rank(V_a)=k$. SVD: $...
0
votes
0answers
6 views

Equivalence of gaussian process and bayesian linear regression by inspecting the covariance matrix

I'm aware that a gaussian process is equivalent to bayesian linear regression for the kernel $K(x_i,x_j) = x_i x_j$ (assume scalar $x$ here). However, the proof itself didn't lend much intuition to me....
5
votes
1answer
89 views

Parameter estimation in the linear mixed effects model

In Parameter estimation and inference in the linear mixed effects model, page 1923, the variance \begin{equation} \begin{aligned} \text{var}(\tilde{u} - u) & = \sigma^2G - \text{var}(\tilde{u}) \...
0
votes
0answers
13 views

covariance in 2 dimensional field

I want to know is the covariance equation which is shown below is how valid when we are in 2-dimensional fields (not 2-dimensional data)? $$Cov(x,y)=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-y)}{N-1}$$ ...
0
votes
1answer
22 views

Why do we get a new co-ordinate system when we dot product the transpose of eigen vectors with the transpose of a matrix

I am working on implementing PCA on the MNIST dataset and have calculated the eigen vector and eig Values from the co-variance matrix. Now I want to have a new co-ordinate system represented by PC1 ...
0
votes
1answer
18 views

Why does the variance of the estimate of coefficients blow up when (XTX)-1 is singular?

I have been getting my hands on linear regression and multicolinearity problem, mainly trying to approach it within a sense of linear algebra. I found this pdf from CMU as to colinearity. It says the ...
0
votes
0answers
26 views

Reducing linear dependency among the columns of a matrix [duplicate]

I have a matrix whose columns are highly correlated, hence using this matrix in compressed sensing algorithms is not giving satisfactory results. So is there any way in which we can reduce the linear ...
0
votes
0answers
8 views

Binary Matrix Low Rank Matrix Factorization

Low Rank Matrix Factorization is a pretty popular problem in data mining. We need to find 2 matrices, $W, H$ such as $F = W \cdot H$. I know that this approximation is NPC problem, so we won't find ...
8
votes
1answer
151 views

Mixed Model Equations

In this paper on page 1924 it is stated that \begin{equation} \text{Var}(u \mid y) = \sigma^2[G - GZ^\top H^{-1}ZG] \end{equation} can be written as \begin{equation} \text{Var}(u \mid y) = \sigma^...
0
votes
0answers
18 views

R - factor model - R-squared calculation

I am trying to recalculate in R an example of a factor model presented in the Zivot, Wang book (Modeling Financial Time Series with S-PLUS) p.548 (link) I am looking for an explanation of the ...
1
vote
0answers
29 views

Compact/Vectorized Multiclass Logistic Regression Hessian

I know that the Hessian of the categorical cross entropy w.r.t the weights is given by $$\frac{\partial^2 L}{\partial w^2} = \sum_{i=1}^{m} (Diag(\hat{y}_i)-\hat{y}_i^T \hat{y}_i) \otimes x_i^T x_i$$ ...
4
votes
0answers
76 views

Nystrom approximation with inexact/stochastic kernel evaluation

Suppose we have several data points $x_1,\ldots,x_m\in\mathbb R^n$ as well as a positive definite kernel $K(x,y):\mathbb R^n\times\mathbb R^n\to\mathbb R$ that can be written in the form $$K(x,y)=\...
1
vote
1answer
120 views

On the properties of covariance and kernel matrices

I'm stumbling upon an example of a mixed model or a Gaussian Process, say: $Z \in\mathbb{R}^{n \times m}, m \ge n$ ie random effect $X \in\mathbb{R}^{n \times p}, p \ge 1$ ie fixed effects $K \in\...
0
votes
0answers
9 views

Numpy corrcoef giving different result from manual calculation

I'm trying to calculate a partial correlation matrix for a high dimensional problem. I'm using this paper as a guide. I'm also referencing this function from Pingouin. Starting from the inverse ...
0
votes
0answers
23 views

Derive the inference equation for Bernoulli-distributed two-class classification problem

Consider the following probabilistic model: We have a training set of X = x1.....xn, where each sample consists of m binary features. tn=1 corresponds to C1 and tn=0 to C2. I already derived the ...
2
votes
0answers
16 views

Can't Recreate Values for U, S, V from SVD in numpy [duplicate]

To better understand SVD, I'm trying to recreate the values for U, S, and V using straight ...
2
votes
1answer
27 views

covariance matrix derivation

How to derive from the first step to the second step, where y is k by 1 vector and A is k by p matrix. Thanks for any help in advance
2
votes
1answer
29 views

QR Factorization to Solve Least Squares Without Using an Inverse

I'm playing around with different ways to solve least squares, and am using numpy to derive values for $\beta$ in a regression problem. I know that if you do a $QR$ factorization of $X$ such that $...
0
votes
0answers
21 views

If 2 linear combinations of 2 predictor variables (x1, x2) produce 2 outcome variables (y1, y2), what is cov(x2, y2)?

Assume that cor(x1, x2) means the correlation (Pearson's r) between x1 and x2, and cov(x1, x2) means the covariance between x1 and x2. Given the below assumptions: N = 1000 Both mean(x1) and mean(x2)...
3
votes
2answers
170 views

R: linear algebra representation of the prediction operator for a mixed effects model

(See edit at the bottom for the bounty) I am trying to learn how to simulate LMM data with matrix linear algebra. So far I've managed to simulate a simple model with a random intercept: ...
7
votes
3answers
557 views

Proof that if covariance is zero then there is no linear relationship

I get that a zero covariance doesn´t imply independence, but everybody says that if there is dependence and the covariance is zero then it is a non linear dependence. People base their interpretation ...
0
votes
0answers
12 views

Intuition about how the formula for “variance of axis of angle $\alpha$ with horizontal axis” works (multiple correspondents analysis)

From the text : Multiple Correspondents Analysis by Brigette LeRoux the following is given (page 32). For the purposes of this post I'm just considering there to be two dimensions that point clouds ...
0
votes
0answers
22 views

prove for Eckart-young in Frobenius norm

On page 74, linear algebra and learning from data. P74 the prove for Eckart-young in the frobenius norm. I couldnot understand why G = 0 in the proof, anybody can help me? Thank you!
0
votes
1answer
17 views

(SVMs) Do the specific higher dimensional mappings of attributes not matter when calculating a kernel?

From what I know, one of the strategies employed by an SVM is to increase dimensionality of your data until they are linearly separable. (I guess there's some mathematical proof that your data will ...
0
votes
0answers
19 views

Assessing the size of a cone by the singular values of $M$

Suppose I work with vectors from a high dimensional space with $100<N<1000$, e.g. word-embeddings. Say I have, already selected $R$ vectors, with $R\simeq10$, which form a matrix $M \in \mathbb{...
0
votes
0answers
34 views

Two layer network working in PyTorch, but cannot reproduce using pure linear algebra

Rewording the original question to comply with the CV topics. I am trying to build a simple 2 layer network, and was going through the backprop mathematics of it, when I got stuck. I am defining my ...
0
votes
0answers
40 views

Why are the eigenvalues of $X'X$ equal to that of $XX'$ when $X$ is a design matrix? [duplicate]

The title says it all. If $X$ is a design matrix (columns containing variables, rows containing observations), I have observed that eigs($X'X$)=eigs($XX'$). I actually found this by accident when I ...
2
votes
0answers
42 views

Geometric interpretation of Cholesky Decomposition

I understand that a square matrix, say $A$, can be thought of as a linear transformation within the same space. I could be as simple as basis change or some other transformation. In this way of ...
0
votes
1answer
39 views

Distance between 2 hyperplanes in SVM formulation

During the SVM formulation, the 2 hyperplanes is given by the equations: wᵀx + b = 1 ---------(1) wᵀx + b = -1 ---------(2) Now, the margin between these 2 hyperplanes is given by: 2/||w|| ...
2
votes
1answer
41 views

Why is the following choice of factor loadings optimal in two-state MLE for factor analysis?

Suppose we have $n$, $p$-dimensional, samples $\overrightarrow{X_i} \sim \mathcal{N}(\mu, \Psi+\mathbf{w^Tw})$. $\Psi$ is a diagonal matrix of specific variances, while $\mathbf{w^Tw}$ composes the ...
0
votes
1answer
43 views

Matrix representation of PCA

In both Wikipedia and this medium post, I see the succinct principle components decomposition of X represented as $$T=XW$$ However, it seems to me that it should be $T = WX$ instead, if according ...
1
vote
1answer
37 views

Determinants of the sum of two matrices and their eigenvalues

I know some basic property of determinant. I read an article and see this formula: \begin{equation} |(\delta-p-1)D+S|=\bigg(\prod_{i=1}^{p}\lambda_{i}(D)\bigg) \bigg(\prod_{i=1}^{p}(\delta-p-1)\...
1
vote
1answer
57 views

How to merge two different sum of squared errors?

Given two datasets $ U \in R$ and $P \in R$. $n_1$ and $n_2$ are respectively the number of points contained in U and P. The sum of squared errors of U and P are as follow: $$ SSE_U=\sum_{i=1}^{n_1}||...
0
votes
0answers
42 views

Why does Alternating Least Squares (ALS) give us good results for missing values?

I was reading about the alternating least squares algorithm and could follow the math but somehow it didn't click for me. We start with random values for $U$ and $V$ and run the algorithm until we ...
0
votes
0answers
27 views

eigen value decomposition of co-variance a series generated by factor model

Let's assume $N\times T$ series $Y_t$ is generated by the following equation. $$ Y_t = \begin{bmatrix}A_x & A_m\end{bmatrix}\begin{bmatrix}x_t \\ m_t \end{bmatrix}$$ Where $A_x$ and $A_m$ are $N\...
2
votes
1answer
33 views

Random matrix theory and research - a lot like doing linear algebra?

I've been searching for a subfield of research to get into and wonder whether random matrix theory could suit me well; it seems like it does, because the stuff I read, and the seminars that I watch ...
3
votes
1answer
110 views

If I have only $(\pmb{X}'\pmb{X})^{-1}$, how can I find $\pmb{X}$ (the design matrix)?

My teacher gave me a problem, but he only give me the $(\pmb{X}'\pmb{X})^{-1}$ matrix. If I have only $(\pmb{X}'\pmb{X})^{-1}$, how can I find $\pmb{X}$ (the design matrix)? I think this is an ...
1
vote
1answer
20 views

A derivation regarding kernel regression for the support vector machine

THis is from the Elements of Statistical Learning book page 437 in the section of support vector machine. Can anyone give me some hint for the missing derivation steps for why 12.49 is true (as seen ...
0
votes
1answer
41 views

Expansion of inner product for polynomial kernel for SVMs

On page 424 in "The Elements of Statistical Learning" by Hastie et al (2013) (https://web.stanford.edu/~hastie/Papers/ESLII.pdf), we see the following expansion of a polynomial kernel with degree 2: ...
1
vote
1answer
96 views

Proof $E[Z'TZ]^2=\operatorname{tr}^2(T)+\operatorname{tr}(T^2)$ [duplicate]

How to prove second moment of a quadratic form where $Z$ has normal distribution with mean zero and covariance matrix identical?
1
vote
0answers
28 views

Difference between two dimensions sampled from Dirichlet distribution

Say I'm doing Bayesian inference on a Dirichlet-Multinomial model: $$ x \in [1,2,3]; \\ x \sim Multinomial(p_1, p_2, p_3); \\ p_1, p_2, p_3 \sim Dirichlet(\alpha_1, \alpha_2, \alpha_3); \\ \alpha_n =...
2
votes
0answers
38 views

How to characterize the effect of $(\textrm{Diag}(\Sigma^{-1}))^{-1}$ badly approximating $\textrm{Diag}(\Sigma)$

I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's. If I were to try to approximate ...
0
votes
1answer
40 views

Gradient descent derivation in Eigenspace [duplicate]

I am trying to decode article on https://distill.pub/2017/momentum/ I was able to follow everything until the part with a change of basis x$^k=Q^T(w^k−w^⋆)$ to eigenspace... I conceptually understand ...
0
votes
0answers
44 views

Variables estimation with Cholesky decomposition

I have the covariance matrix between log-returns of n variables. I suppose the distribution of the log-returns is normal for all the variables with average=0 but standard deviation in general $\neq$ 1....
1
vote
0answers
30 views

Understanding PCA from a linear transformation perspective

I've came across several great questions and answers here regarding PCA, but I would like to have a look at it from a linear transformation perspective. Let's say I have a (demeaned) data matrix $X$ ...
0
votes
0answers
28 views

The least squares estimator for beta in matrix notation in connection to normal equations and orthogonal projection in linear algebra [duplicate]

Somebody posted this link here about DERIVATION OF THE LEAST SQUARES ESTIMATOR FOR BETA IN MATRIX NOTATION-(https://economictheoryblog.com/2015/02/19/ols_estimator/). My question is 'How does THE ...