Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

206 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
0answers
117 views

Is multivariate Cauchy stable?

I am trying to prove (if possible), for given $A_{n\times n}$ and $B_{n\times n}$, there exists a $C_{n\times n}$ satisfying $$A\pmb{X}_1 + B\pmb{X}_2 \stackrel{D}{=} C\pmb{X},$$ where $X_1, ~X_2$, ...
7
votes
2answers
117 views

Random rotation of a set of distinct points in $R^n$

Consider a set $\{\mathbf{X}_1,\cdots , \mathbf{X}_M\}$ of distinct points in $\mathbb{R}^n$ with $M$ finite. The $M$ values of the $i$-th coordinate do not all have to be dinstinct. For example, in $\...
7
votes
0answers
586 views

QR decomposition of normally distributed matrices

Assume $M$ is an $N \times k$ Gaussian matrix, i.e., its entries are i.i.d. standard normal random variables, with $N>>k$. Take $D=\text{diag}(\lambda_1, \dotsc ,\lambda_N)$ for some fixed real ...
6
votes
0answers
466 views

Rank of kernel Gram matrix and classifier performance

In kernel machines we have some kernel function $k$ and we compute the $n \times n$ Gram matrix $K$ where $K_{ij} = k(x_i, x_j)$ for observations $x_i, x_j \in \mathbb R^p$. I'm letting $n$ denote the ...
5
votes
0answers
126 views

Nystrom approximation with inexact/stochastic kernel evaluation

Suppose we have several data points $x_1,\ldots,x_m\in\mathbb R^n$ as well as a positive definite kernel $K(x,y):\mathbb R^n\times\mathbb R^n\to\mathbb R$ that can be written in the form $$K(x,y)=\...
5
votes
1answer
84 views

Fitting a curve OVER OR UNDER a set of points

I want to fit a curve $f(x) = mx+b$ on my data points $x_1, \ldots, x_N$ using linear regression with a single predictor. However, the cost function is not even, rather, it has different weights on ...
4
votes
0answers
87 views

Physical interpretation of $U$ and $V$ matrices in SVD

I have a question about the physical interpretation of $U$ and $V$ matrices in SVD. I collect measurements at multiple devices across time are collected into an $m$ × $T$ matrix $M$, where m is the ...
4
votes
0answers
200 views

Geometric interpretation of Cholesky Decomposition

I understand that a square matrix, say $A$, can be thought of as a linear transformation within the same space. I could be as simple as basis change or some other transformation. In this way of ...
4
votes
0answers
75 views

Show that regression coefficients on group-level variables are unaffected by individual-level variation

My question I have a linear regression that contains some regressors that vary only at a group level and some that vary at the individual level. Slide 8 of this suggests that the coefficients on the ...
4
votes
0answers
214 views

Are null space of matrix and kernel function same?

I have recently started learning about machine learning and have come across kernels and null spaces. I understand that null space is the set of all vectors that satisfy the equation A.v = 0 (Where A ...
4
votes
1answer
99 views

What is the relationship between Mohr's Circle and Principal Component Analysis?

While I was studying PCA, I was told that it is related to Mohr's Circle. I don't know what that means. I don't know if they are related or not. I was just told, so I want to make sure here. If they ...
4
votes
0answers
824 views

Ill-conditioned covariance matrices in EM

I am currently working with the Expectation-Maximization algorithm. I have some pre-clustered sets of 3D points and am trying to run the algorithm. However I've seen that most of my covariance ...
4
votes
0answers
547 views

Caclulating standard errors for generalized least squares without raw data (just sample means and covariance)

I have a question very similar to the question asked here: is it possible to calculate standard errors (specifically, the standard error of the intercept) for generalized least squares regression ...
4
votes
0answers
259 views

Techniques for scaling data matrix to avoid rank deficiency issues

I have a $n \times p$ matrix $A$ where $n$ is the number of observables and $p$ is the number of observations. $n \gg p$ In my code, I have done $[E,V] \,=\, eig(A)$ and doing a least squares ...
4
votes
0answers
1k views

Principal component analysis, bootstrap and probability of eigenvalue collision?

This is really a side project of mine ... while writing on a paper on something totally different! I read (part of ) the excellent paper "FINITE SAMPLE APPROXIMATION RESULTS FOR PRINCIPAL COMPONENT ...
3
votes
0answers
41 views

Do someone understand what the authors mean? - a very strange notation

I'm reading the paper Estimation of (near) low-rank matrices with noise and high-dimensional scaling and came across a very very odd notation. I'll quote the entire passage: Any matrix $\Theta^*\in \...
3
votes
0answers
70 views

Decorrelation, PCA and rotation

I am not a PCA expert, nor do I have a good knowledge of linear algebra, so bear with me and my ignorance. I am trying to understand how the authors of some papers I have been reading decorrelate two ...
3
votes
0answers
169 views

Does training loss go to zero in kernel regression?

Edit Have left the original post in tact, scroll to bottom for updated thinking High Level Problem Statement While studying kernel regression, after playing around with some linear algebra, I appear ...
3
votes
0answers
44 views

How to characterize the effect of $(\textrm{Diag}(\Sigma^{-1}))^{-1}$ badly approximating $\textrm{Diag}(\Sigma)$

I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's. If I were to try to approximate ...
3
votes
0answers
30 views

Are there extant deep learning analogs to random coefficient (aka mixed) models?

Random coef models, applied to longitudinal data, capture response heterogeneity by cross-sectional unit. I've got a longitudinal prediction problem, in which I know that some "features" (or ...
3
votes
0answers
613 views

Obtain within-group Gram matrix out of distance matrix

Gram matrix Let $\bf X$ be a n x p dataset with columns (variables) centered. Then p x p $\bf X'X$ is the total scatter matrix ...
3
votes
0answers
391 views

Linear transformation of a random vector with pseudo-inverse

If $$ \mathbf{X} = (X_1,..,X_n)^t$$ is a random variable drawn according to a probability density function (pdf) $$ f_{X_1,...,X_n}(x_1,...,x_n) $$ then $$ \mathbf{Y} = A\mathbf{X} = (Y_1,..,Y_n)^t$$ ...
3
votes
0answers
1k views

PCA loadings of correlation matrix

I know similar questions have been asked before already but I'm still a bit confused about the understanding of 'loadings'. Eigenvectors are unit-scaled loadings; and they are the coefficients (...
3
votes
0answers
2k views

Simulating data from a given multivariate covariance matrix - workarounds for a non positive definite covariance matrix?

As part of a simulation study, I would like to create multivariate data that follow a specific covariance matrix. In this study I would like to be able to show that my algorithm is able to find highly ...
3
votes
0answers
702 views

When is the scatter matrix in linear discriminant analysis singular?

In linear discriminant analysis (LDA), when there are fewer data instances than the number of dimensions (i.e., when the data matrix is of order $n \times m$ where $n$ is less than $m$), the within-...
3
votes
0answers
87 views

Back-prop question: can this gradient be decomposed?

So, I was going over the lectures for the Oxford 2015 deep learning course, and in the lectures, they introduce back-propagation as a recursive procedure which involves two key formulas: The ...
3
votes
0answers
382 views

What is the theory behind using eigenvectors for spatial filtering

As a programmer I have used the spdep package successfully for spatial filtering. But would appreciate it if someone could offer a description (preferably with supporting references) of how this ...
3
votes
1answer
384 views

Finding optimal hyperplane

I have a set of vectors $\{V_i\}$ in $n$-dimensional space. There is a number corresponded to each vector $\alpha_i = f(V_i)$ ($\alpha_i$ can be negative). I want to find a hyperplane which would ...
2
votes
0answers
43 views

Does the Cholesky decomposition of a covariance matrix lead to a lower triangular matrix with positive diagonals?

We know that an $N\times N$ covariance matrix $\Sigma$ is symmetric positive definite, and can be factorized using Cholesky decomposition as follows \begin{equation} \Sigma=LL' \end{equation} where $L$...
2
votes
0answers
38 views

How to calculate the eigenvector?

I've been struggling to solve this math problem for two days... So I calculated the mean of all samples (0,0). Put it into the equation and got V as \begin{array} {rrr} 4 & 2 \\ 2 & 2 \end{...
2
votes
0answers
86 views

Derivation of skewness and kurtosis algebra of random variables

In algebra of random variables, the symbolic rule for computing variance of random variable $X\in\mathbb{R}^{n\times p}$ multiplied by a coefficent vector, $a\in\mathbb{R}^p$, is $$\text{Var}(X\cdot a)...
2
votes
0answers
77 views

What is the significance of “orthogonal” vectors in statistics?

So I am reading What does orthogonal mean in the context of statistics?, and there are contradictory answers. The most upvoted answer says that "Therefore, orthogonality does not imply ...
2
votes
1answer
49 views

Can anyone show how the concept of Identifiability is geometrically/intuitively presented?

The motivation for this question comes from the following: When I was studying statistics for the first time long ago, no one presented the mathematical concepts behind linear regression, like the one ...
2
votes
0answers
111 views

Prove variance of locally weighted regression increases with degree

I am interested in proving the following fact for locally weighted polynomial regression from The Elements of Statistical Learning by Hastie et. al. It can be shown that $||l(x_0)||$ increases with ...
2
votes
0answers
16 views

How to approximate a Hermitian matrix with a transposed cross product of a single matrix?

I have a complex square matrix, and wish to learn latent factors (equally weighted latent factors, so not SVD) from this matrix. Given a Hermitian matrix A of ...
2
votes
0answers
287 views

How does the VAT algorithm work?

I want to know how the VAT algorithm for cluster tendency works in detail. As you can see in the picture below, R is the dissimilarity matrix and R-tilda is the ordered dissimilarity matrix. What is ...
2
votes
0answers
108 views

Row similarity in matrix vs in different factorizations

Suppose an arbitrary $m \times n$ matrix $M$ and the factorizations: Arbitrary: $M = U_a V_a^T$, where $U_a$ is $m \times k$, $V_a$ is $n \times k$ ($k < m,n$), and $rank(U_a)=rank(V_a)=k$. SVD: $...
2
votes
1answer
57 views

Random matrix theory and research - a lot like doing linear algebra?

I've been searching for a subfield of research to get into and wonder whether random matrix theory could suit me well; it seems like it does, because the stuff I read, and the seminars that I watch ...
2
votes
0answers
63 views

Difference between two dimensions sampled from Dirichlet distribution

Say I'm doing Bayesian inference on a Dirichlet-Multinomial model: $$ x \in [1,2,3]; \\ x \sim Multinomial(p_1, p_2, p_3); \\ p_1, p_2, p_3 \sim Dirichlet(\alpha_1, \alpha_2, \alpha_3); \\ \alpha_n =...
2
votes
0answers
94 views

Understanding PCA from a linear transformation perspective

I've came across several great questions and answers here regarding PCA, but I would like to have a look at it from a linear transformation perspective. Let's say I have a (demeaned) data matrix $X$ ...
2
votes
0answers
187 views

Does anyone know the rank of the Netflix Prize dataset?

I'm looking into the Netflix Prize at the moment. We model the dataset as an $n \times m$ matrix, where $n$ is the number of users and $m$ is the number of movies. Does anyone know the rank of the ...
2
votes
0answers
55 views

Finding Lag Polynomial Roots = Cayley-Hamilton?

In my time series class we defined the lag polynomial by $\phi(L) = \sum_{i=0}^N\phi_i L^i$. It is well known that this polynomial can be factored as by $\prod_{i=0}^N(I - \lambda_i L)$ (note that $I$ ...
2
votes
0answers
28 views

Integration of the product of Two multivariable Gaussain pdfs

I want to calculate \begin{align} \int_{-\infty}^{\infty} G(x-v_i, \Sigma_i) G(x-v_j, \Sigma_j) dx \end{align} where \begin{align} G(x-v_i, \Sigma_i) = \frac{1}{(2\pi)^{d/2} |\Sigma_i|^{1/2}} \...
2
votes
0answers
88 views

how to calculate curvature of multivariate function?

EDIT1: tried to clarify the question Context In the context of an MCMC investigation of non-linear interaction effects in dichotomous models, I am creating data generating processes based on the ...
2
votes
0answers
23 views

Regression with 0-1 features

Is there a fast strategy for estimating regression when all your features are either 0 or 1? The OLS solution would be $(X^TX)^{-1} X ^T y$. Given X is a binary matrix, is there a fast way to ...
2
votes
0answers
309 views

Does minimum norm solution guarantee generalization in the underconstrained case (in the statistical learning sense)?

Recall that pseudo-inverse can be characterized as follows: Solve $$ \| w \|^2 $$ subject to: $$ Xw = y $$ thus it is plausible since its a constrained optimization problem that the solution ...
2
votes
0answers
52 views

How to calculate a perfect separation in a d dimensional space between n class?

Assuming in a d-dimensional space, we have samples from n class. The best way of separating samples of each class from each other is to have the samples from each class as far as possible from every ...
2
votes
0answers
47 views

Matrix Decomposition $ B = B^* + \sum_{i>1}\lambda_i B_i$

I recently encountered the following decomposition in a paper, may I know whether anyone on CV happened to know the name of the decomposition and a proof of it? $B$ is a regular stochastic matrix and ...
2
votes
0answers
83 views

How can one design a polynomial function that really does require higher order terms to approximate it well?

My goal is to design an experiment such that only a high order polynomial function can approximate the target polynomial function. I've been trying to approximate a polynomial function in 1D $f_{...
2
votes
0answers
376 views

Linear Algebra and NumPy - How array broadcast translates to matrix operations?

I am familiar with Python's NumPy library and its broadcasting feature that allows to perform operations with vectors and matrices of different sizes. I might be missing the mathematical connection as ...

1
2 3 4 5