Skip to main content

Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
1 vote
0 answers
36 views

how to approximate the eigendecomposition of a correlation matrix when the data have been standardized?

Context I am working to develop a penalized regression framework that will scale up to analyzing high dimensional data with a certain correlation structure. Let $X$ represent an $n \times p$ matrix of ...
Tabitha Peter's user avatar
0 votes
0 answers
13 views

Prove that if eigenvalues are all different then eigenvectors are linearly independent [migrated]

I was trying to work out a proof of the fact that if eigenvalues $\lambda_1$, $\lambda_2$,...,$\lambda_n$ are all different, then the eigenvectors $\mathbf{v}_1$, $\mathbf{v}_2$,...,$\mathbf{v}_n$ are ...
Tirthankar's user avatar
4 votes
1 answer
47 views

Why do OLS libraries fit models using the MP Pseudoinverse of the design matrix?

For the linear model $y = X\beta$ for design matrix $X$, it's well known that the optimal solution is $\hat{\beta} = (X'X)^{-1}X'y$. Some statistical libraries (such as Python's statsmodels) estimate ...
user1993951's user avatar
0 votes
0 answers
20 views

Constrained Cholesky Decomposition

Suppose that I have an $(n\times 1)$ vector of random variables, $\varepsilon$. However, I know that $k$ linear combinations of $\varepsilon$ are 0. Specifically, I know that for a $(k\times n)$ ...
Leland's user avatar
  • 1
5 votes
1 answer
182 views

Rasmussen Equation 5.9

Can any one add the steps showing how Rasmussen (Gaussian Processes for Machine Learning, the MIT Press, 2006) got from line 1 to line 2 of equation 5.9. (pg 114)? It is calculating the gradient of ...
Snowy Baboon's user avatar
10 votes
2 answers
282 views

Does the conditional expectation operator have an interpretable decomposition like the projection matrix does in linear algebra?

I'm trying to draw a parallel between the concept of projections in a finite linear space to an infinite linear space. Here is the set-up, first in the finite dimensional case, and then second in the ...
absolutelyzeroEQ's user avatar
3 votes
1 answer
70 views

The Math Behind the Conditional Probability of a Probabilistic PCA

I am trying to understand how to calculate the conditional distribution of probabilistic principal component analysis. This is explained in the book "Pattern Recognition and Machine Learning"...
CAM_etal's user avatar
1 vote
0 answers
13 views

Positive distance weighting

I have an overdetermined linear system of equations that's solved with least squares. I'd like to weight the equations to penalize a bunch of inputs clumped up together. Ideally if two (or more) ...
Marsupilami's user avatar
1 vote
0 answers
25 views

What exactely is "the part of the interaction orthogonal to factors $A$ and $B$" in a two-way ANOVA?

Consider a two-way ANOVA with factors $A$ and $B$ and the interaction $A\times B$. The author of this answer answer https://stats.stackexchange.com/a/608301/359647 (@svendvn) explains that the Type ...
Quertiopler's user avatar
0 votes
0 answers
17 views

Error term in SGD with momentum

I am reading the article "How Momentum really works" (https://distill.pub/2017/momentum/), and i am confused in one point: I am trying to derive the convergence rate for momentum from the ...
Patricio's user avatar
  • 121
0 votes
0 answers
10 views

Decomposing model volatility with respect to factor contributions

Consider a linear model $\textbf{y} = \textbf{x}\pmb{\beta} + \pmb{\varepsilon}$ with $\textbf{y}$ a $T \times 1$ vector of random variables, $\pmb{\beta}$ a $K \times 1$ vector and $\textbf{x}$ a $T \...
user9875321__'s user avatar
5 votes
1 answer
227 views

Why does this matrix form of weighted least squares not match sklearn's weight?

I coded up the answer to this question and it turned out not to match: https://math.stackexchange.com/questions/1021812/matrix-form-for-weighted-least-squares The solutions are close, and I'm ...
ron burgundy's user avatar
0 votes
2 answers
71 views

simple ANN as a set of linear transformations

We cannot classify the points of the XOR problem with a single perceptron in the hidden layer. However, we can achieve this by using two perceptrons in the hidden layer and one for the output layer, ...
Mag's user avatar
  • 1
1 vote
1 answer
21 views

Finding a design matrix

I am trying to understand how a design matrix was obtained in this problem below. Consider the one sample problem: $Y_i \sim N(\mu, \sigma^2), 1 \le i \le n$. with the $Y_i's$ i.i.d. The MLE is: $\hat\...
Harry Lofi's user avatar
3 votes
1 answer
96 views

Geometric understanding of linear regression

I am reading up on linear regression from mit 16.850 Here is how the lecture goes: Given: $Y_{n,1}$ (targets), $X_{n, p}$ (data), $t_{p, 1}$ (the parameters I'm optimizing over), True model: $Y = \...
figs_and_nuts's user avatar
0 votes
1 answer
30 views

Principal Component Analysis and Relation to the SVD of a matrix [duplicate]

We are learning about Principal Component analysis in our class, and I having trouble understanding how to compute the principal component given a matrix. For example, here is the matrix we were given....
Harry Lofi's user avatar
5 votes
1 answer
104 views

Linear algebra properties of a confusion matrix (eigenvalues, eigenvectors, and determinants)

This answer to a question on Math Stack Exchange got me thinking about a confusion matrix as more than just a rectangular array of numbers. We don’t talk about a confusion matrix as a linear ...
Dave's user avatar
  • 64.7k
0 votes
0 answers
24 views

Proving how scaling of predictor variable in linear regression, affects the fitted coefficient [duplicate]

In linear regression the OLS solution is given by: $$ \hat{\beta} = (X^TX)^{-1}X^TY $$ I want to show that if you scale the $i$th predictor variable by a constant, then the corresponding $i$th ...
Dylan Dijk's user avatar
0 votes
0 answers
28 views

Simple matrix calculus but I am struggling to understand [duplicate]

Here is my problem: We have $\mathbf{D} \in \Re^{m n}$, $\mathbf{W} \in \Re^{m q}$, and $\mathbf{X} \in \Re^{q n}$. Furthermore, $\mathbf{D} = \mathbf{W}\mathbf{X}$. (NOT an element wise ...
wrek's user avatar
  • 195
3 votes
1 answer
31 views

Converting Adjusted R²

I just examined the $R^2_\text{adj}$ Formula on Wikipedia and found two ways to calculate the adjusted $R^2$. Firstly as $$R^2_\text{adj}=1-\frac{\frac{SS_\text{res}}{(n-p-1)}}{\frac{SS_\text{tot}}{(n-...
Linus's user avatar
  • 53
0 votes
2 answers
73 views

The Impact of Vector Magnitudes in Recommendation Systems Matrix Factorization Models

I'm currently exploring latent factor models in recommendation systems, specifically focusing on the interaction between vector magnitudes and the angles between these vectors. While it's clear that ...
Amit S's user avatar
  • 57
3 votes
0 answers
42 views

How much is the data energy loss in PCA?

Recently in a slide in about PCA (Principal Component Analysis) I saw a question: "How much is the data energy loss in PCA?&...
hasanghaforian's user avatar
1 vote
0 answers
24 views

Product of Two t-distribution Formulas

Does the product of two t-distribution formulas with same degrees of freedom simplify? $T_v(x; \mu_1, \Sigma_1)T_v(x; \mu_2, \Sigma_2) =\ ?...$ In the normal case it simplifies to: $\mathcal{N}(x; \...
Snowy Baboon's user avatar
3 votes
1 answer
104 views

Dual form of the least square solution (ridge rigression)

I was reading this introductory material and on the 5th page, it describes the dual form of the least-square solution (with ridge regression) as $$A(aI + A^\top A)^{-1} = (aI + AA^\top)^{-1}A$$ for a $...
Alemu's user avatar
  • 125
0 votes
0 answers
18 views

Calculating the Orthogonal Distance to Kernel PCA subspace (with a new data)

I am studying Kernel PCA methods and now I'm trying to calculate orthogonal distances (OD) on the feature space. What I've found is, you can calculate ODs with a kernel trick if you are interested in ...
cccanhakan's user avatar
1 vote
0 answers
20 views

How to compare different clusters of different size, rotation, scale and translation?

Assume that you have a matrix $X$ that contains the data inside the left image. The data inside $X$ is not classified. The matrix $X$ also contains outliers/noise. On the right, we can se the template ...
euraad's user avatar
  • 425
0 votes
1 answer
47 views

What does $(x_i - \xi_k)_+$ mean in this regression spline formula? [duplicate]

I have seen regression models with a continuous predictor fitted as a spline written like this: What is the meaning of the little "addition symbol" subscript that I have circled in red? Is ...
user167591's user avatar
5 votes
2 answers
175 views

Covariance matrix square root

Consider a random variable $r_t$ which represents the return of an asset at time t. In the univariate case, we just consider $r_t$ to be the return of a single security at time t. Generally, we assume ...
rudinable's user avatar
7 votes
2 answers
795 views

Is the expectation of a random vector multiplied by its transpose equal to the product of the expectation of the vector and that of the transpose

I'm taking a course in advance statistics and we have to prove whether the following expression is true: $E[zz^T]=E[z]E[z^T]$. I am assuming it is not, since the formula of the covariante matrix is $...
ghost wizard's user avatar
0 votes
0 answers
25 views

Covariance matrix for data

Assume $n*p$ data matrix $X$, where n is the number of observations and p is the number of features. We are interested in the covariance among features. I have seen notations where covariance matrix ...
Kaiwen Wang's user avatar
0 votes
0 answers
7 views

Leverage and One-Out Model

I am currently trying to prove the following theorem, found within the course material from a Graduate Regression module. For context, here $X$ is the usual full-rank design matrix, estimators are ...
CarloPatti's user avatar
1 vote
0 answers
22 views

Heteroscedastic Asymptotic Variance Simple Transformation

Let's denote the asymptotic variance under heteroscedasticity as: $$\hat{\text{Avar}}(\hat{\beta}) = 1/N * \left(\frac{1}{N} \sum_{i}{x_i x_i'}\right)^{-1} \left(\frac{1}{N} \sum_{i} \hat{u}^2_i x_i ...
Marlon Brando's user avatar
3 votes
1 answer
70 views

Spiked tensor decomposition vs canonical polyadic decomposition

What are the similarities and differences between Spiked tensor decomposition and canonical polyadic (CP) decomposition? My understanding is that CP decomposition aims to find a low-rank approximation ...
Omar Shehab's user avatar
0 votes
0 answers
21 views

How to adjust similarity scores by removing the influence of a common vector?

I have a similarity score function, $s(x,y)$. I know that I have two items that I'm trying to compare the similarity of, but both are based on the same template. How would I remove the template from ...
user760900's user avatar
2 votes
1 answer
342 views

How to sample efficiently from an inverse Wishart distribution?

I am trying to understand the code from pybasicbayes, which defines the following function to sample from an inverse Wishart: ...
seeker_after_truth's user avatar
0 votes
0 answers
17 views

Estimate null hypothesis for correlation of linear combinations of variables?

Setting up the problem Suppose I have a variable $x$ of length $n$ and I have another $p$ variables $y_1, y_2, \dots, y_p$, where $y_i$ is also of length $n$. Based on the y's, I can make a linear ...
tbolind's user avatar
  • 43
0 votes
1 answer
185 views

Difference between conducting PCA on $XX^\top$ vs $X^\top X$?

PCA: For a given set of centered data $\mathscr D =\{x_i\}_{i=1}^N \subset \mathbb R^d$, i.e. the data has $N$ examples with dimension $d$. Then the principal directions of PCA can be obtained from ...
Fong Lam's user avatar
0 votes
0 answers
12 views

How can a linear autoencoder with $h=1$ hidden unit reconstruct any rank 1 matrix?

I've had this as a homework problem as a true or false type of question and I'm trying to wrap my head around why this is true. Is the reason simply represent each datapoint as a scaled version of a ...
Oliver's user avatar
  • 310
1 vote
1 answer
173 views

Is it wrong to view convolution as template matching?

I am reading about the convolution operation but I can't see how it can be seen as template matching. Suppose that we convolve the input $\mathbf{X}$: $$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 ...
ado sar's user avatar
  • 477
1 vote
0 answers
27 views

Mediation study help

I have a simple order of operations question with sum products, I want to calculate the natural direct effect (NDE) for x and x prime on data similar to those below. My simple question is: Do I first ...
I_like_insights's user avatar
0 votes
0 answers
75 views

"Dimensionality" of y vs the span of our Independent variables In regression

Question: I want to clarify my understanding of OLS regression using Matrix Algebra. Let's assume we have 2 different independent variables $x_1$ and $x_2$. Our 'model' will be the plane that lives in ...
CormJack's user avatar
  • 161
3 votes
2 answers
77 views

What does the "direction of data" mean in the context of Principal Component Analysis?

I am reading the Introduction to Linear Algebra 5th edition, section 7.3: Principal Component Analysis. The section contains the following sentence The first eigenvector $u_1$ of $S$ points in the ...
desert_ranger's user avatar
3 votes
0 answers
66 views

Relationship between SVD of Matrix and SVD of same Matrix with deleted entries (Matrix can be Adjacency Matrix of a Graph)

Could somebody direct to me to some literature dealing with this issue. So we have $X = U\Sigma V^{T}$ and we have $M \odot X = U^{'}\Sigma^{'}V^{'^{T}}$ with \begin{equation} M_{i,j} = \begin{cases} ...
Yunus Cobanoglu's user avatar
8 votes
3 answers
1k views

Is the sum of the diagonal elements of a covariance matrix always equal or larger than the sum of its off-diagonal elements?

For any given covariance matrix, will the sum of the diagonal elements always be bigger than the sum of the off-diagonal elements? Let $\sigma_i$ be the standard deviation of the $i^\text{th}$ term of ...
krenova's user avatar
  • 175
3 votes
0 answers
45 views

Johansen test: why testing for the algebraic multiplicity of 0 and not for the nullity?

From what I already know about the Johansen test, it tests the rank of the VAR matrix (in error correction form) through steps testing whether every eigenvalue is signifincantly different from 0 (...
Mauro's user avatar
  • 151
2 votes
0 answers
41 views

Is there a high probability bound of quadratic forms?

I am wondering about the following: For a symmetric matrix $A \in \mathbb{R}^{n \times n}$ and vector $x \in [-1,1]^n$, if $X$ is a random vector in $\mathbb{R}^n$ such that w.h.p. $X_i \not\in [-1,1] ...
swuk's user avatar
  • 43
2 votes
0 answers
44 views

Johansen test accepts first null hypothesis but would reject last one

Suppose that we perfrom a Johansen test over three I(1) variables that give us these results through the maximum eigenvalues statistic: as you can see, we accept the null hypothesis in the first step ...
Mauro's user avatar
  • 151
1 vote
0 answers
26 views

Least Squares Regression with Length and Density of input data

I have a collection of data that I expect to be linear but has a unknown amount of noise to the data. Initially I wanted to use the least squares regression line to determine if the slope, y axis, and ...
Caleb Laws's user avatar
0 votes
1 answer
500 views

Prove that 2nd order polynomial kernel is positive semi-definite

I'm trying to prove that the 2nd order polynomial kernel, $K(x_i, x_j) = (x_i^Tx_j + 1)^2$ is a valid kernel which satisfies the following conditions: K is symmetric, that is, $K(x_i, x_j) = K(x_j, ...
Muhteva's user avatar
  • 103
0 votes
0 answers
24 views

Distribution of power two of a term

Assume we have matrices $A_{n\times n}$ and $\Delta_{n\times n}$ and know that: $$A^{-1}_{i.}\Delta_{.j}\sim N(0,\,A^{-1}_{i.}\,\Sigma_j\,[A^{-1}_{i.}]^\top)\,,$$ where $\Delta\sim N(0,\,\Sigma_j)$, $...
statwoman's user avatar
  • 703

1
2 3 4 5
14