Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
0
votes
0answers
33 views

Good concise, big picture, linear algebra book?

I have looked at this answer and am not satisfied with the results. Reference book for linear algebra applied to statistics? I have briefly looked at two of the books suggested by the answer, the one ...
1
vote
1answer
9 views

Intuitive explanation of Minimum Covariance Determinant (MCD)

I am an undergrad working on Anomaly Detection on an 8 dimensional dataset, with PYOD, which relies on the MCD in the sklearn's MinCovDet. I tried reading Minimum Covariance Determinant and Extensions,...
0
votes
0answers
28 views

This is how I understand why the denominator of the root mean square is raised to the power of 1/2. Am I on the right track?

I've been going through some threads (see links below) and also a lot of introductory statistics textbooks to try to understand why in the formula for the root mean square, the denominator is also ...
0
votes
0answers
3 views

Algebra for Intervention Effect in Interrupted Time Series (with delayed effect)

I have run an interrupted time series analysis using GLS regression model in R. My data consists of 48 observations [time 1:48], with the intervention implemented at time 20, but it's effect not ...
0
votes
0answers
18 views

PCA dimension reduction on correlation matrix for invertability

I have a non-singular (correlation) matrix $C$ of dimension $N{\times}N$, this is a modified version of another correlation matrix, and therefore I don't think I am able to apply any calculations on ...
0
votes
0answers
17 views

Published source for D-dimensional behaviour of Dot-Product

I am currently studying the behaviour of the dot product between two random vectors in $R^d$. Specifically I wanted to start with the case of uniform random vectors on $\mathcal{S}^{d-1}$. I found ...
1
vote
1answer
34 views

Is the variance of an estimator equal to the variance of error and the SSE of a regression?

Is the variance of an estimator equal to the variance of error? $Var(β) = σ^2 $ Since $Var(β)=E[εε']$ and $E[εε'] = Var(σ) = σ^2$ ? Additionally why is it that the expression for $Var(β) = E[εε']$ ...
1
vote
1answer
39 views

Rank of sample covariance matrix when $p = n$

Suppose we have a $p$-dimensional Gaussian distribution, and we take $n$ observations from that distribution. This answer states that when $p > n$, then the sample variance covariance matrix is ...
2
votes
1answer
55 views

Is $tr(B(B^TWB + D)^{-1}B^TW) = tr((I + D(B^TWB)^{-1})^{-1})$?

I am reading Eilers and Marx (1996) and at the beginning of page 94 they write, for $Q = B^TWB$, $D$ a symmetric positive definite matrix and $W$ a diagonal matrix, \begin{align} tr\left(B(Q + D)^{-1}...
0
votes
0answers
14 views

Connection between samples and dimensions of a matrix with the covariance matrix in PCA

In PCA, for a given matrix $M_{S\times D}$ where s = samples and d = dimensions, computing covariance matrix of dimension vector and then an eigen decomposition on it leads to eigenvectors which can ...
1
vote
0answers
24 views

how to formulate this constraint?

Assume I want to solve a regression problem $AX=B$ , the matrix A is a thin matrix and rank deficient i.e, the nullity of $A$ is non-empty , i want a solution for which the block entries of $x$ are in ...
0
votes
0answers
26 views

Must an interpolator exist with linearly independent basis functions?

I have data $(x_1,y_1), \dots, (x_n, y_n) \in \mathbb R^2$ and the $x_i$ are distinct and increasing. I want to interpolate the $y_i$ with a function $$ f(x) = \sum_{i=1}^Na_i h_i(x) $$ where the $...
0
votes
0answers
13 views

What is the difference between np.linalg.norm(x-y,axis=1) and np.linalg.norm(x-y)?

I'm creating a K-Medoids algorithm from scratch in Python using numpy, and I'm in the process of using a distance function to determine the cluster center. I want the center to be the point in the ...
0
votes
0answers
10 views

Regularizing the difference in the norms of two independent weight matrices in a neural network

Say, there are two neural network layers with weights $W_1$ and $W_2$. These two layers are part of a larger network but their inputs are completely independent of each other and their outputs could ...
1
vote
0answers
38 views

Angle between PCA vector spaces?

I have two datasets of the same shape, one for condition A, the other for condition B. I would like to test if the major axes of variance of condition A are different than those of B. Here is my idea. ...
1
vote
0answers
27 views

Factor Analysis - restricting Loadings

I´m new to Multivariable Statistics and just started to learn about Factor Analysis. I do understand that for p=k (there are as many factors as dimensions in the data) L (loadings) is unique. But ...
2
votes
1answer
37 views

What is this projection matrix doing?

Let’s say we have a $m\times d$ zero mean multivariate Gaussian matrix $X$. Its covariance matrix is $X^{T}X$. Let $V$ be the $d\times d$ matrix of eigenvectors of $X^{T}X$, with the columns sorted in ...
1
vote
1answer
40 views

Is this statement about linear-activation neural network in the PRML in error?

I found the following statement in "Pattern Recognition and Machine Learning" (C. M. Bishop, 2016) p.229. If the activation functions of all the hidden units in a network are taken to be linear, ...
0
votes
1answer
31 views

What can you say about spread of data by looking at singular values and clusters?

I have dataset 250X5 and its singular values are [200 50 25.2 2.3 0.35]. Singular values are directly related to variance. Can you say something about the clustering of data and how much is the spread ...
2
votes
1answer
43 views

Can anyone show how the concept of Identifiability is geometrically/intuitively presented?

The motivation for this question comes from the following: When I was studying statistics for the first time long ago, no one presented the mathematical concepts behind linear regression, like the one ...
1
vote
0answers
76 views

Prove variance of locally weighted regression increases with degree

I am interested in proving the following fact for locally weighted polynomial regression from The Elements of Statistical Learning by Hastie et. al. It can be shown that $||l(x_0)||$ increases with ...
0
votes
1answer
28 views

How to do multiple linear regressions with overlapping predictors efficiently

I'm using covariance matrices to do a linear regression. I have the same number of predictors and dependent variables (3000 of each). For each $i$, $X_{i}$ and $Y_{i}$ have a kind of symmetry. I ...
1
vote
1answer
22 views

Finding $G$ in a solved least squares $d = Gm$?

This may be odd, but say someone solved $d = Gm$ using the least squares method and calculated the parameters vector $m$. I have the vector of data $d$, how could I solve for $G$ to see the matrix ...
0
votes
0answers
8 views

Kalman Filter with Non-Linear State-Transition but No Process Noise

In the regular Kalman Filter, one of the requirements is that the observation model, state-transition model, and control-input model are all linear. If any are non-linear, then you would need to use ...
0
votes
0answers
31 views

Matrix formula for the correlation matrix, page 72 from Mathematical Tools for Applied Multivariate Analysis

Book information The book is titled: Mathematical Tools for Applied Multivariate Analysis By Paul E. Green and J. Douglas Carroll The print that I have is from 1976 The ISBN number is ...
1
vote
1answer
34 views

Matrix Formulation of Equations

I am trying to learn statistical learning and machine learning independently. One of the main challenges I am facing is that many resources/books use matrix formulation when expressing equations. I am ...
0
votes
1answer
35 views

Why is my A.T*A (A transpose A) matrix singular?

I'm running into an wall on my intuition when using least squares. I'm trying to simulate some data, for fun, and I'm getting a result that says my (A.T * A) matrix is singular. In order to condense ...
0
votes
1answer
73 views

Why does logistic regression's likelihood function have no closed form?

The derivative of Log likelihood function of logistic regression with respect to theta is Why can't we equate it to zero and solve for theta so that we can obtain a 'closed form solution' for theta? ...
0
votes
0answers
21 views

Generalized singular value dicomposition

For a given I × J matrix A, generalizing the singular value decomposition, involves using two positive definite square matrices with size I × I and J × J respectively. These two matrices express ...
0
votes
0answers
28 views

What is the first order derivative of linear regression's cost function using matrix calculus?

For linear regression's cost function J(b), where X is a n*m matrix, b is a m*1 vector and y is n*1 vector: First order derivative with respect to vector b (coefficients) is shown to be Using the ...
0
votes
1answer
38 views

In most machine learning books, how are matrices of data organized?

I'm trying understand linear algebra but am still a bit confused. I think I am missing some of the conventions. Let's say there is a supervised learning problem. We have 100 observations, 7 ...
4
votes
1answer
78 views

In a linear regression, trying to show $R^2 = r_{xy}^2$ using projections / geometric intuition

In a linear regression $$ Y = X\beta + \varepsilon, $$ I define two (standard) projection matrices. The projection matrix into subspace spanned by columns of the design matrix $X$: $$ H := X(X^\top ...
0
votes
1answer
59 views

Interpretation of $\mathbf{w^{\top}Cw}$ [closed]

I'm reading a piece on portfolio optimization where it is stated that $\mathbf{w^{\top}Cw}$ is the variance of the expected return, where $\mathbf{C}$ is a covariance matrix and $\mathbf{w}$ is a ...
2
votes
0answers
12 views

How to approximate a Hermitian matrix with a transposed cross product of a single matrix?

I have a complex square matrix, and wish to learn latent factors (equally weighted latent factors, so not SVD) from this matrix. Given a Hermitian matrix A of ...
1
vote
1answer
104 views

Cholesky decomposition or alternative for negatively correlated data simulations

I want to generate some signals that have a correlation distribution around a specific pre-defined correlation value (i.e., the distribution of the values of their correlation matrix is around a ...
1
vote
0answers
51 views

how does VAT algorithm works

I want to know how the VAT algorithm for cluster tendency works in detail. As you can see in the picture below R is the dissimilarity matrix and R-tilda is the ordered dissimilarity matrix. What is ...
1
vote
1answer
59 views

Minimizing expected loss in Regression with Rademacher random variables

I am trying to prove the following equality. I am able to solve the terms inside the expectation but I am stuck because of the expectation with respect to $x,y$. I might be wrong in the whole process; ...
1
vote
1answer
76 views

Implementation of predictive variance in Gaussian process regression of scikit-learn

I'm studying the implementation of Gaussian Process Regression in scikit-learn to get a better understanding of the topic. There I've stumbled upon the following snippet: ...
3
votes
1answer
103 views

Least Squares removing first $k$ observations Woodbury formula?

Given the matrix $X_{n,p}$ from the least squares problem $$ \mathbf{X} \cdot \mathbf{\beta} = z $$ Where the normal equation is: $$ \mathbf{\hat{\beta}} = \left(\mathbf{X}^T \mathbf{X}\right)^{-1} ...
0
votes
0answers
33 views

Variance of Least Squares Estimate

Universally the literature seems to make a jump in the proof of variance of the least squares estimator and I'm hoping you can fill in the gaps for me. In matrix form, the least squares estimate is: ...
0
votes
0answers
14 views

How can I run CCA with a Y matrix that only has one observation per row?

I have two matrices I want to run Canonical Correlation Analysis on. The X matrix is fully filled in. The Y matrix has multiple columns, but only one column filled in per row, the rest are missing. ...
1
vote
0answers
53 views

How do we know adding an uncorrelated variable to a regression will not change existing coefficients?

Say I have a regression with 3 independent variables and I decide to introduce a 4th variable and rerun the regression. A previous post states that the coefficient on an original variable will change ...
1
vote
1answer
50 views

How do I solve this system of equations?

I am doing something that is commmon practice in economics to uniquely identify matrices. After deriving 3 unrotated factors from PCA, I then want to rotate them to be able to interpret them in ...
0
votes
0answers
10 views

Understanding a particular rotation strategy

I am trying to replicate the rotation strategy (for factors) in a paper. I will try to make as clear as possible what I don't understand. Let the model be: $X = F\Lambda + \epsilon$, where $F$ is Tx3 ...
2
votes
1answer
51 views

Decomposing Gradient Decent Error in Eigenvector Space

I'm going through Why Momentum Really Works and am unable to understand the following line in the article. "By writing the contributions of each eigenspace’s error to the loss $$f(w^{k})-f(w^{\star})=...
4
votes
1answer
46 views

Triangular Markov chain question

A triathlon consists of $3$ disciplines: swimming, cycling and running. A triathlete does a training session every day. However he doesn’t want to pay for professional coaching advice so instead his ...
2
votes
0answers
91 views

Row similarity in matrix vs in different factorizations

Suppose an arbitrary $m \times n$ matrix $M$ and the factorizations: Arbitrary: $M = U_a V_a^T$, where $U_a$ is $m \times k$, $V_a$ is $n \times k$ ($k < m,n$), and $rank(U_a)=rank(V_a)=k$. SVD: $...
0
votes
0answers
29 views

Equivalence of gaussian process and bayesian linear regression by inspecting the covariance matrix

I'm aware that a gaussian process is equivalent to bayesian linear regression for the kernel $K(x_i,x_j) = x_i x_j$ (assume scalar $x$ here). However, the proof itself didn't lend much intuition to me....
5
votes
1answer
107 views

Parameter estimation in the linear mixed effects model

In Parameter estimation and inference in the linear mixed effects model, page 1923, the variance \begin{equation} \begin{aligned} \text{var}(\tilde{u} - u) & = \sigma^2G - \text{var}(\tilde{u}) \...
0
votes
0answers
16 views

covariance in 2 dimensional field

I want to know is the covariance equation which is shown below is how valid when we are in 2-dimensional fields (not 2-dimensional data)? $$Cov(x,y)=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-y)}{N-1}$$ ...

1
2 3 4 5
10