Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
1
vote
1answer
76 views

Minimizing expected loss in Regression with Rademacher random variables

I am trying to prove the following equality. I am able to solve the terms inside the expectation but I am stuck because of the expectation with respect to $x,y$. I might be wrong in the whole process; ...
1
vote
1answer
126 views

Implementation of predictive variance in Gaussian process regression of scikit-learn

I'm studying the implementation of Gaussian Process Regression in scikit-learn to get a better understanding of the topic. There I've stumbled upon the following snippet: ...
5
votes
2answers
282 views

Least Squares removing first $k$ observations Woodbury formula?

Given the matrix $X_{n,p}$ from the least squares problem $$ \mathbf{X} \cdot \mathbf{\beta} = z $$ Where the normal equation is: $$ \mathbf{\hat{\beta}} = \left(\mathbf{X}^T \mathbf{X}\right)^{-1} ...
0
votes
0answers
45 views

Variance of Least Squares Estimate

Universally the literature seems to make a jump in the proof of variance of the least squares estimator and I'm hoping you can fill in the gaps for me. In matrix form, the least squares estimate is: ...
1
vote
0answers
82 views

How do we know adding an uncorrelated variable to a regression will not change existing coefficients?

Say I have a regression with 3 independent variables and I decide to introduce a 4th variable and rerun the regression. A previous post states that the coefficient on an original variable will change ...
1
vote
1answer
53 views

How do I solve this system of equations?

I am doing something that is commmon practice in economics to uniquely identify matrices. After deriving 3 unrotated factors from PCA, I then want to rotate them to be able to interpret them in ...
2
votes
1answer
72 views

Decomposing Gradient Decent Error in Eigenvector Space

I'm going through Why Momentum Really Works and am unable to understand the following line in the article. "By writing the contributions of each eigenspace’s error to the loss $$f(w^{k})-f(w^{\star})=...
4
votes
1answer
50 views

Triangular Markov chain question

A triathlon consists of $3$ disciplines: swimming, cycling and running. A triathlete does a training session every day. However he doesn’t want to pay for professional coaching advice so instead his ...
2
votes
0answers
108 views

Row similarity in matrix vs in different factorizations

Suppose an arbitrary $m \times n$ matrix $M$ and the factorizations: Arbitrary: $M = U_a V_a^T$, where $U_a$ is $m \times k$, $V_a$ is $n \times k$ ($k < m,n$), and $rank(U_a)=rank(V_a)=k$. SVD: $...
1
vote
1answer
489 views

Equivalence of gaussian process and bayesian linear regression by inspecting the covariance matrix

I'm aware that a gaussian process is equivalent to bayesian linear regression for the kernel $K(x_i,x_j) = x_i x_j$ (assume scalar $x$ here). However, the proof itself didn't lend much intuition to me....
5
votes
1answer
147 views

Parameter estimation in the linear mixed effects model

In Parameter estimation and inference in the linear mixed effects model, page 1923, the variance \begin{equation} \begin{aligned} \text{var}(\tilde{u} - u) & = \sigma^2G - \text{var}(\tilde{u}) \...
0
votes
0answers
17 views

covariance in 2 dimensional field

I want to know is the covariance equation which is shown below is how valid when we are in 2-dimensional fields (not 2-dimensional data)? $$Cov(x,y)=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-y)}{N-1}$$ ...
0
votes
1answer
27 views

Why do we get a new co-ordinate system when we dot product the transpose of eigen vectors with the transpose of a matrix

I am working on implementing PCA on the MNIST dataset and have calculated the eigen vector and eig Values from the co-variance matrix. Now I want to have a new co-ordinate system represented by PC1 ...
0
votes
1answer
161 views

Why does the variance of the estimate of coefficients blow up when (XTX)-1 is singular?

I have been getting my hands on linear regression and multicolinearity problem, mainly trying to approach it within a sense of linear algebra. I found this pdf from CMU as to colinearity. It says the ...
0
votes
0answers
26 views

Reducing linear dependency among the columns of a matrix [duplicate]

I have a matrix whose columns are highly correlated, hence using this matrix in compressed sensing algorithms is not giving satisfactory results. So is there any way in which we can reduce the linear ...
8
votes
1answer
160 views

Mixed Model Equations

In this paper on page 1924 it is stated that \begin{equation} \text{Var}(u \mid y) = \sigma^2[G - GZ^\top H^{-1}ZG] \end{equation} can be written as \begin{equation} \text{Var}(u \mid y) = \sigma^...
1
vote
0answers
221 views

Compact/Vectorized Multiclass Logistic Regression Hessian

I know that the Hessian of the categorical cross entropy w.r.t the weights is given by $$\frac{\partial^2 L}{\partial w^2} = \sum_{i=1}^{m} (Diag(\hat{y}_i)-\hat{y}_i^T \hat{y}_i) \otimes x_i^T x_i$$ ...
5
votes
0answers
120 views

Nystrom approximation with inexact/stochastic kernel evaluation

Suppose we have several data points $x_1,\ldots,x_m\in\mathbb R^n$ as well as a positive definite kernel $K(x,y):\mathbb R^n\times\mathbb R^n\to\mathbb R$ that can be written in the form $$K(x,y)=\...
2
votes
1answer
354 views

On the properties of covariance and kernel matrices

I'm stumbling upon an example of a mixed model or a Gaussian Process, say: $Z \in\mathbb{R}^{n \times m}, m \ge n$ ie random effect $X \in\mathbb{R}^{n \times p}, p \ge 1$ ie fixed effects $K \in\...
2
votes
0answers
17 views

Can't Recreate Values for U, S, V from SVD in numpy [duplicate]

To better understand SVD, I'm trying to recreate the values for U, S, and V using straight ...
4
votes
1answer
108 views

covariance matrix derivation

How to derive from the first step to the second step, where y is k by 1 vector and A is k by p matrix. Thanks for any help in advance
4
votes
1answer
563 views

QR Factorization to Solve Least Squares Without Using an Inverse

I'm playing around with different ways to solve least squares, and am using numpy to derive values for $\beta$ in a regression problem. I know that if you do a $QR$ factorization of $X$ such that $...
4
votes
2answers
245 views

R: linear algebra representation of the prediction operator for a mixed effects model

(See edit at the bottom for the bounty) I am trying to learn how to simulate LMM data with matrix linear algebra. So far I've managed to simulate a simple model with a random intercept: ...
7
votes
3answers
1k views

Proof that if covariance is zero then there is no linear relationship

I get that a zero covariance doesn´t imply independence, but everybody says that if there is dependence and the covariance is zero then it is a non linear dependence. People base their interpretation ...
1
vote
1answer
27 views

(SVMs) Do the specific higher dimensional mappings of attributes not matter when calculating a kernel?

From what I know, one of the strategies employed by an SVM is to increase dimensionality of your data until they are linearly separable. (I guess there's some mathematical proof that your data will ...
0
votes
0answers
37 views

Two layer network working in PyTorch, but cannot reproduce using pure linear algebra

Rewording the original question to comply with the CV topics. I am trying to build a simple 2 layer network, and was going through the backprop mathematics of it, when I got stuck. I am defining my ...
0
votes
0answers
151 views

Why are the eigenvalues of $X'X$ equal to that of $XX'$ when $X$ is a design matrix? [duplicate]

The title says it all. If $X$ is a design matrix (columns containing variables, rows containing observations), I have observed that eigs($X'X$)=eigs($XX'$). I actually found this by accident when I ...
4
votes
0answers
175 views

Geometric interpretation of Cholesky Decomposition

I understand that a square matrix, say $A$, can be thought of as a linear transformation within the same space. I could be as simple as basis change or some other transformation. In this way of ...
0
votes
2answers
414 views

Distance between 2 hyperplanes in SVM formulation

During the SVM formulation, the 2 hyperplanes is given by the equations: wᵀx + b = 1 ---------(1) wᵀx + b = -1 ---------(2) Now, the margin between these 2 hyperplanes is given by: 2/||w|| ...
3
votes
1answer
49 views

Why is the following choice of factor loadings optimal in two-state MLE for factor analysis?

Suppose we have $n$, $p$-dimensional, samples $\overrightarrow{X_i} \sim \mathcal{N}(\mu, \Psi+\mathbf{w^Tw})$. $\Psi$ is a diagonal matrix of specific variances, while $\mathbf{w^Tw}$ composes the ...
0
votes
1answer
58 views

Matrix representation of PCA

In both Wikipedia and this medium post, I see the succinct principle components decomposition of X represented as $$T=XW$$ However, it seems to me that it should be $T = WX$ instead, if according ...
1
vote
1answer
43 views

Determinants of the sum of two matrices and their eigenvalues

I know some basic property of determinant. I read an article and see this formula: \begin{equation} |(\delta-p-1)D+S|=\bigg(\prod_{i=1}^{p}\lambda_{i}(D)\bigg) \bigg(\prod_{i=1}^{p}(\delta-p-1)\...
1
vote
1answer
102 views

How to merge two different sum of squared errors?

Given two datasets $ U \in R$ and $P \in R$. $n_1$ and $n_2$ are respectively the number of points contained in U and P. The sum of squared errors of U and P are as follow: $$ SSE_U=\sum_{i=1}^{n_1}||...
0
votes
0answers
32 views

eigen value decomposition of co-variance a series generated by factor model

Let's assume $N\times T$ series $Y_t$ is generated by the following equation. $$ Y_t = \begin{bmatrix}A_x & A_m\end{bmatrix}\begin{bmatrix}x_t \\ m_t \end{bmatrix}$$ Where $A_x$ and $A_m$ are $N\...
2
votes
1answer
44 views

Random matrix theory and research - a lot like doing linear algebra?

I've been searching for a subfield of research to get into and wonder whether random matrix theory could suit me well; it seems like it does, because the stuff I read, and the seminars that I watch ...
4
votes
1answer
123 views

If I have only $(\pmb{X}'\pmb{X})^{-1}$, how can I find $\pmb{X}$ (the design matrix)?

My teacher gave me a problem, but he only give me the $(\pmb{X}'\pmb{X})^{-1}$ matrix. If I have only $(\pmb{X}'\pmb{X})^{-1}$, how can I find $\pmb{X}$ (the design matrix)? I think this is an ...
1
vote
1answer
58 views

A derivation regarding kernel regression for the support vector machine

THis is from the Elements of Statistical Learning book page 437 in the section of support vector machine. Can anyone give me some hint for the missing derivation steps for why 12.49 is true (as seen ...
0
votes
1answer
165 views

Expansion of inner product for polynomial kernel for SVMs

On page 424 in "The Elements of Statistical Learning" by Hastie et al (2013) (https://web.stanford.edu/~hastie/Papers/ESLII.pdf), we see the following expansion of a polynomial kernel with degree 2: ...
6
votes
3answers
274 views

Intuition for nonmonotonicity of coefficient paths in ridge regression

Intuitively, why may some of the slope coefficients in ridge regression increase in magnitude when the penalty parameter $\lambda$ is increased? Or in other words, why are the coefficient paths ...
1
vote
1answer
180 views

Proof $E[Z'TZ]^2=\operatorname{tr}^2(T)+\operatorname{tr}(T^2)$ [duplicate]

How to prove second moment of a quadratic form where $Z$ has normal distribution with mean zero and covariance matrix identical?
2
votes
0answers
56 views

Difference between two dimensions sampled from Dirichlet distribution

Say I'm doing Bayesian inference on a Dirichlet-Multinomial model: $$ x \in [1,2,3]; \\ x \sim Multinomial(p_1, p_2, p_3); \\ p_1, p_2, p_3 \sim Dirichlet(\alpha_1, \alpha_2, \alpha_3); \\ \alpha_n =...
3
votes
0answers
43 views

How to characterize the effect of $(\textrm{Diag}(\Sigma^{-1}))^{-1}$ badly approximating $\textrm{Diag}(\Sigma)$

I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's. If I were to try to approximate ...
0
votes
1answer
105 views

Gradient descent derivation in Eigenspace [duplicate]

I am trying to decode article on https://distill.pub/2017/momentum/ I was able to follow everything until the part with a change of basis x$^k=Q^T(w^k−w^⋆)$ to eigenspace... I conceptually understand ...
2
votes
0answers
79 views

Understanding PCA from a linear transformation perspective

I've came across several great questions and answers here regarding PCA, but I would like to have a look at it from a linear transformation perspective. Let's say I have a (demeaned) data matrix $X$ ...
2
votes
1answer
271 views

Question about the gradient of weight normalization

In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$ \mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v $$ Then they ...
0
votes
0answers
80 views

Alternating Least Squares for Baseline Predictor

I am trying to figure out how ALS works when minimizing the following formula: $\\ \\$ $\text{min}_{\lbrace b_u,b_i \rbrace} \sum_{(u,i)\in \mathcal{K}} (r_{ui} - \bar{r} - b_u - b_i )^2 + \lambda_{...
2
votes
1answer
62 views

Struggling on second to last part deriving linear discriminant function

From this post here I am struggling with the matrix multiplication to get from: $\log \pi _{k} - \frac{1}{2}(x-\mu _k)^T{\sum }^{-1}(x-\mu _k)$ to $\log \pi _{k} - \frac{1}{2}[x^{T}{\sum }^{-1}x +\...
4
votes
1answer
529 views

Chebyshev inequality in terms of RMS

I'm self studying the book Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares In page 48, the author write: "It says,for example, that no more than 1/25 = 4% of the entries ...
0
votes
0answers
41 views

what is a vector perpendicular to a plane of vectors

I have 3 or 4 vectors connected that forms a plane. How can I find the vector that is perpendicular to this plane? it can be a unit vector as long as it preserves this direction. each vector is on ...
3
votes
0answers
29 views

Are there extant deep learning analogs to random coefficient (aka mixed) models?

Random coef models, applied to longitudinal data, capture response heterogeneity by cross-sectional unit. I've got a longitudinal prediction problem, in which I know that some "features" (or ...

1 2
3
4 5
11