Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
2
votes
0answers
82 views

Derivation of skewness and kurtosis algebra of random variables

In algebra of random variables, the symbolic rule for computing variance of random variable $X\in\mathbb{R}^{n\times p}$ multiplied by a coefficent vector, $a\in\mathbb{R}^p$, is $$\text{Var}(X\cdot a)...
1
vote
0answers
172 views

What's the importance of parallel eigenvectors?

I'm studying eigenvectors. I read that if a matrix is symmetric and if the eigenvalues are real numbers, the eigenvectors will be perpendicular. However, I have no idea what it means (if anything) ...
1
vote
0answers
35 views

PCA with $\mathcal{l}1$-regularization

For any matrix, $X$, $\lambda>0$ and $j$ vectors, $v_1$, $v_2$, ..., $v_k$. We want to solve the following optimization problem. $$\textrm{max}_vv^TX^TXv\textrm{ s.t. }\|v\|_2\leq1, \|v\|_1\leq \...
0
votes
1answer
23 views

how to normalize data such that an estimated OLS regression vector has pre-specified length (= L_2 norm)

I have the following data: $n$ observations on $d$ variables $X$ and one outcome variable $Y$; i.e. $X$ is a $n \times d$ matrix and $Y$ an $n \times 1$ vector. I consider the following Ordinary Least ...
2
votes
1answer
44 views

Why can't same-size layers of a neural network be combined or compressed into a single layer?

Probably a dumb question, obviously not seen in practice, but after reviewing linear algebra, I can't pinpoint the misunderstanding in this logic: We can represent input data to the network as a ...
1
vote
0answers
117 views

Adjoint relationship in Neural ODEs

The Chen et. al paper Neural ODE (https://arxiv.org/pdf/1806.07366.pdf) uses the adjoint method to take derivatives of solutions generated by an ODE solver with respect to neural network parameters θ. ...
3
votes
1answer
158 views

Human intuition behind SVD in case of recommendation system

This does not answer my question. I struggled very hard to understand the SVD from a linear-algebra point of view. But in some cases I failed to connect the dots. So, I started to see all the ...
0
votes
0answers
30 views

How vector projection works behind SVD?

I was reading a blog on mathematical intuition behind SVD. Here, author pointed out three information we get after vector decomposition. The directions of projection — the unit vectors (v₁ and v₂) ...
1
vote
0answers
44 views

Relationship between the SVD and correlation matrices

I'm reading Data Driven Science and Engineering by Kutz and Brunton to understand more about the SVD. Consider $X = U\Sigma V$, $XX^*$, and $X^*X$ where $X \in \mathbb{R}^{m\times n} $ In particular, ...
3
votes
0answers
146 views

Does training loss go to zero in kernel regression?

Edit Have left the original post in tact, scroll to bottom for updated thinking High Level Problem Statement While studying kernel regression, after playing around with some linear algebra, I appear ...
5
votes
2answers
203 views

Distribution of the inner product between a noise-free and a noisy signal

I am working on a problem where we have a noisy measured signal, which is stored as an $N$-dimensional vector $\mathbf{Y},$ and a set of $n_s$ simulated noise-free signals $\{\mathbf{X}_i\}_{i=1}^{n_s}...
1
vote
0answers
32 views

How to maximize the steady state transition probability for a state in a Markov chain by altering that state's outgoing transition probabilities?

Let's say we have a transition matrix of which can be solved to come up with steady state transition probabilities of Alpha: 34.9% Beta: 24.0% Gamma: 16.9% Delta: 24.2% Now, imagine Alpha, Beta, ...
0
votes
0answers
24 views

Condition numbers, invertibility and multicollinearity

The following is an excerpt from Greene's Regression Analysis (Seventh Edition): a) What does it mean to be "difficult" to invert a matrix accurately? Shouldn't all matrixes be either ...
2
votes
0answers
69 views

What is the significance of “orthogonal” vectors in statistics?

So I am reading What does orthogonal mean in the context of statistics?, and there are contradictory answers. The most upvoted answer says that "Therefore, orthogonality does not imply ...
2
votes
0answers
20 views

Relationship between a linear difference equation and the hyperbolic functions [closed]

Considering a linear difference equation \begin{equation} \underbrace{\begin{bmatrix} -p & 1 & 0 & 0 & 0 & \cdots & 0\\ 1 &-p & 1 & 0 & 0 & \cdots & 0\\...
0
votes
0answers
74 views

Good concise, big picture, linear algebra book?

I have looked at this answer and am not satisfied with the results. Reference book for linear algebra applied to statistics? I have briefly looked at two of the books suggested by the answer, the one ...
1
vote
1answer
561 views

Intuitive explanation of Minimum Covariance Determinant (MCD)

I am an undergrad working on Anomaly Detection on an 8 dimensional dataset, with PYOD, which relies on the MCD in the sklearn's MinCovDet. I tried reading Minimum Covariance Determinant and Extensions,...
0
votes
0answers
12 views

Algebra for Intervention Effect in Interrupted Time Series (with delayed effect)

I have run an interrupted time series analysis using GLS regression model in R. My data consists of 48 observations [time 1:48], with the intervention implemented at time 20, but it's effect not ...
0
votes
0answers
72 views

PCA dimension reduction on correlation matrix for invertability

I have a non-singular (correlation) matrix $C$ of dimension $N{\times}N$, this is a modified version of another correlation matrix, and therefore I don't think I am able to apply any calculations on ...
0
votes
0answers
18 views

Published source for D-dimensional behaviour of Dot-Product

I am currently studying the behaviour of the dot product between two random vectors in $R^d$. Specifically I wanted to start with the case of uniform random vectors on $\mathcal{S}^{d-1}$. I found ...
1
vote
1answer
71 views

Is the variance of an estimator equal to the variance of error and the SSE of a regression?

Is the variance of an estimator equal to the variance of error? $Var(β) = σ^2 $ Since $Var(β)=E[εε']$ and $E[εε'] = Var(σ) = σ^2$ ? Additionally why is it that the expression for $Var(β) = E[εε']$ ...
1
vote
1answer
153 views

Rank of sample covariance matrix when $p = n$

Suppose we have a $p$-dimensional Gaussian distribution, and we take $n$ observations from that distribution. This answer states that when $p > n$, then the sample variance covariance matrix is ...
2
votes
1answer
61 views

Is $tr(B(B^TWB + D)^{-1}B^TW) = tr((I + D(B^TWB)^{-1})^{-1})$?

I am reading Eilers and Marx (1996) and at the beginning of page 94 they write, for $Q = B^TWB$, $D$ a symmetric positive definite matrix and $W$ a diagonal matrix, \begin{align} tr\left(B(Q + D)^{-1}...
0
votes
0answers
34 views

Connection between samples and dimensions of a matrix with the covariance matrix in PCA

In PCA, for a given matrix $M_{S\times D}$ where s = samples and d = dimensions, computing covariance matrix of dimension vector and then an eigen decomposition on it leads to eigenvectors which can ...
1
vote
0answers
25 views

how to formulate this constraint?

Assume I want to solve a regression problem $AX=B$ , the matrix A is a thin matrix and rank deficient i.e, the nullity of $A$ is non-empty , i want a solution for which the block entries of $x$ are in ...
0
votes
0answers
29 views

Must an interpolator exist with linearly independent basis functions?

I have data $(x_1,y_1), \dots, (x_n, y_n) \in \mathbb R^2$ and the $x_i$ are distinct and increasing. I want to interpolate the $y_i$ with a function $$ f(x) = \sum_{i=1}^Na_i h_i(x) $$ where the $...
0
votes
0answers
86 views

What is the difference between np.linalg.norm(x-y,axis=1) and np.linalg.norm(x-y)?

I'm creating a K-Medoids algorithm from scratch in Python using numpy, and I'm in the process of using a distance function to determine the cluster center. I want the center to be the point in the ...
0
votes
0answers
28 views

Regularizing the difference in the norms of two independent weight matrices in a neural network

Say, there are two neural network layers with weights $W_1$ and $W_2$. These two layers are part of a larger network but their inputs are completely independent of each other and their outputs could ...
1
vote
0answers
198 views

Angle between PCA vector spaces?

I have two datasets of the same shape, one for condition A, the other for condition B. I would like to test if the major axes of variance of condition A are different than those of B. Here is my idea. ...
1
vote
0answers
29 views

Factor Analysis - restricting Loadings

I´m new to Multivariable Statistics and just started to learn about Factor Analysis. I do understand that for p=k (there are as many factors as dimensions in the data) L (loadings) is unique. But ...
2
votes
1answer
63 views

What is this projection matrix doing?

Let’s say we have a $m\times d$ zero mean multivariate Gaussian matrix $X$. Its covariance matrix is $X^{T}X$. Let $V$ be the $d\times d$ matrix of eigenvectors of $X^{T}X$, with the columns sorted in ...
1
vote
1answer
55 views

Is this statement about linear-activation neural network in the PRML in error?

I found the following statement in "Pattern Recognition and Machine Learning" (C. M. Bishop, 2016) p.229. If the activation functions of all the hidden units in a network are taken to be linear, ...
0
votes
1answer
46 views

What can you say about spread of data by looking at singular values and clusters?

I have dataset 250X5 and its singular values are [200 50 25.2 2.3 0.35]. Singular values are directly related to variance. Can you say something about the clustering of data and how much is the spread ...
2
votes
1answer
49 views

Can anyone show how the concept of Identifiability is geometrically/intuitively presented?

The motivation for this question comes from the following: When I was studying statistics for the first time long ago, no one presented the mathematical concepts behind linear regression, like the one ...
2
votes
0answers
101 views

Prove variance of locally weighted regression increases with degree

I am interested in proving the following fact for locally weighted polynomial regression from The Elements of Statistical Learning by Hastie et. al. It can be shown that $||l(x_0)||$ increases with ...
0
votes
1answer
163 views

How to do multiple linear regressions with overlapping predictors efficiently

I'm using covariance matrices to do a linear regression. I have the same number of predictors and dependent variables (3000 of each). For each $i$, $X_{i}$ and $Y_{i}$ have a kind of symmetry. I ...
1
vote
1answer
26 views

Finding $G$ in a solved least squares $d = Gm$?

This may be odd, but say someone solved $d = Gm$ using the least squares method and calculated the parameters vector $m$. I have the vector of data $d$, how could I solve for $G$ to see the matrix ...
1
vote
1answer
28 views

Why is it necessary to “ignore” a level when applying sum contrasts?

I am confused about how sum contrasts are set up. As I understand, if I have some $K$-leveled factor, I can use sum contrasts to compare each level to the grand mean ($M_G$), effectively testing ...
0
votes
0answers
36 views

Matrix formula for the correlation matrix, page 72 from Mathematical Tools for Applied Multivariate Analysis

Book information The book is titled: Mathematical Tools for Applied Multivariate Analysis By Paul E. Green and J. Douglas Carroll The print that I have is from 1976 The ISBN number is ...
1
vote
1answer
37 views

Matrix Formulation of Equations

I am trying to learn statistical learning and machine learning independently. One of the main challenges I am facing is that many resources/books use matrix formulation when expressing equations. I am ...
0
votes
1answer
112 views

Why is my A.T*A (A transpose A) matrix singular?

I'm running into an wall on my intuition when using least squares. I'm trying to simulate some data, for fun, and I'm getting a result that says my (A.T * A) matrix is singular. In order to condense ...
0
votes
0answers
38 views

Principal Component Analysis for a Vector with Mean 0

If we are deriving the first principal component for a vector $X$ with mean 0 and covariance matrix $\Sigma$, can we find it in any other form than $Y^{T}X$ where $Y^{T}$ is the eigenvector ...
1
vote
1answer
842 views

Why does logistic regression's likelihood function have no closed form?

The derivative of Log likelihood function of logistic regression with respect to theta is Why can't we equate it to zero and solve for theta so that we can obtain a 'closed form solution' for theta? ...
0
votes
0answers
94 views

What is the first order derivative of linear regression's cost function using matrix calculus?

For linear regression's cost function J(b), where X is a n*m matrix, b is a m*1 vector and y is n*1 vector: First order derivative with respect to vector b (coefficients) is shown to be Using the ...
0
votes
1answer
51 views

In most machine learning books, how are matrices of data organized?

I'm trying understand linear algebra but am still a bit confused. I think I am missing some of the conventions. Let's say there is a supervised learning problem. We have 100 observations, 7 ...
4
votes
1answer
158 views

In a linear regression, trying to show $R^2 = r_{xy}^2$ using projections / geometric intuition

In a linear regression $$ Y = X\beta + \varepsilon, $$ I define two (standard) projection matrices. The projection matrix into subspace spanned by columns of the design matrix $X$: $$ H := X(X^\top ...
0
votes
1answer
61 views

Interpretation of $\mathbf{w^{\top}Cw}$ [closed]

I'm reading a piece on portfolio optimization where it is stated that $\mathbf{w^{\top}Cw}$ is the variance of the expected return, where $\mathbf{C}$ is a covariance matrix and $\mathbf{w}$ is a ...
2
votes
0answers
16 views

How to approximate a Hermitian matrix with a transposed cross product of a single matrix?

I have a complex square matrix, and wish to learn latent factors (equally weighted latent factors, so not SVD) from this matrix. Given a Hermitian matrix A of ...
1
vote
1answer
285 views

Cholesky decomposition or alternative for negatively correlated data simulations

I want to generate some signals that have a correlation distribution around a specific pre-defined correlation value (i.e., the distribution of the values of their correlation matrix is around a ...
2
votes
0answers
241 views

How does the VAT algorithm work?

I want to know how the VAT algorithm for cluster tendency works in detail. As you can see in the picture below, R is the dissimilarity matrix and R-tilda is the ordered dissimilarity matrix. What is ...

1
2
3 4 5
11