Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

3
votes
2answers
38 views

Steps of Matrix Multiplication

It may seem kind of silly, but can anyone please show me the intermediate steps implied by the second equality in this derivation? $$e^\prime e = \left(y - Xb\right)^\prime\left(y - Xb\right) = y^\...
2
votes
0answers
31 views

Show positive semi-definiteness of co-variance differences

I am having difficulty checking if $$ (A'D^{-1}A)(A'D^{-1}BD^{-1}A)^{-1}(A'D^{-1}A) - (A'A)(A'BA)^{-1}(A'A) \succeq 0 $$ where $B\succ 0$ is a square co-variance matrix and $D$ is a diagonal matrix ...
3
votes
1answer
40 views

The form of the Log-Likelihood Function in Mixed Linear Models

Let us assume the following mixed effects model: $y = X\beta+Zu+e$ where $y$ is a vector of n observable random variables, $\beta$ is a vector of $p$ fixed effects, $X$ and $Z$ are known matrices, ...
0
votes
0answers
10 views

Given m n-dimensional vectors, how to create a vector perpendicular to all of them?

Given $m$ vectors, $x_1$, $x_2$, ... $x_m$ with all $x_i \,\, \epsilon \,\, \mathcal{R}^n$, $i=1,2... m$ and $m < n$. How to sample a vector $x_{m+1}$ perpendicular to all the vectors $x_1$, $...
0
votes
0answers
9 views

Least squares and identificability condition

Let a discrete-time system (which is minimal) with input $u \in \mathbb{R}^m$ and state $x \in \mathbb{R}^n$ be $ x_{k+1} = [x^T_k \quad u^T_k]\begin{bmatrix} A^T \\ B^T \end{bmatrix} + v^T_k $ ...
0
votes
0answers
18 views

Additional Property of Singular Value Decomposition

I am new to SVD so forgive me if the question is trivial. Following is my question. If I have two sets of linear equations, Y1 = T1.X Y2 = T2.X where T1 and T2 are mxn rectangular matrices. Now let'...
3
votes
2answers
51 views

What is the Joint Density Function of a Three-Level Mixed-Effects Model?

This is a follow-up question to a question I posted earlier. Obviously, maximum-likelihood estimation of mixed-effects models requires the joint density function. Let us assume a two-level mixed ...
0
votes
0answers
27 views

Idea behind change of basis and how it relates to projecting your points onto principal components

I would like to clarify if my understanding is correct. In the traditional X-Y coordinate system, our choice of basis vectors are $\vec{i} = (1, 0)$ and $\vec{j} = (0, 1)$ and when you I have a point $...
1
vote
1answer
52 views

Is this dataset lineary seperable? How can I find it out using (linear) algebra?

I have this dataset: I want to know if it is linearly separable (fully separable). I want to use this rule, but I'm not sure if it's correct: Make $X'$ - matrix with d+1 column of all 1's. Then ...
4
votes
1answer
69 views

Mixed Models: How to derive the mixed-model equations?

In the context of best linear unbiased predictors (BLUP), Henderson specified the mixed-model equations (see Henderson (1950): Estimation of Genetic Parameters. Annals of Mathematical Statistics, 21, ...
0
votes
0answers
17 views

Linear Algebra: Dimension of kernel [migrated]

Suppose that we have the vector space $V=\{f/f:R\rightarrow R, \text{every class derivative is defined in R}\}$ and $φ: V\rightarrow W$ with $φ(f)=f+f'$ is linear. I want to find the dimension of ...
1
vote
1answer
13 views

Problem with Softmax decision boundary

While reading this paper: sphere face on page 2, it explains that original softmax boundary is given by: $$(W_1 −W_2)x+b_1 −b_2 = 0$$ While trying to obtain the boundary on a toy generated 2D dataset ...
7
votes
2answers
673 views

What is the problem with $p > n$?

I know that this is the solving system of linear equation problem. But my question is why it is a problem the number of observation is lower than the number of predictors how can that thing happen? ...
1
vote
0answers
53 views

Multilevel Modeling: Minimization Problem when A = B + TCT'

I currently study multilevel model using Leeuw & Meijer (2008) Handbook of multilevel analysis. On page 65, they state the following theorem: If $A = B + TCT'$ with $A$ and $B$ positive definite, ...
-1
votes
1answer
15 views

How to cluster a (directional) dissimilarity matrix with both positive and negative values?

I may be thinking of this incorrectly but what would be the best way to cluster a dissimilarity measure that has direction? For example, if someone had condition A ...
0
votes
0answers
21 views

Is there a problem with the “low-rank matrix approximation”?

I know that "rank" is the number or independent rows in the matrix and I know that the resulting matrix of the "low-rank matrix approximation" algorithm has lower rank than the original matrix. 1- But ...
-1
votes
1answer
31 views

PCA with zero and high correlation in data

How would the eigen values look like when we apply PCA to a dataset with zero correlation between variables and when there is very high correlation between variables
8
votes
4answers
252 views

Is “random projection” strictly speaking not a projection?

Current implementations of the Random Projection algorithm reduce the dimensionality of data samples by mapping them from $\mathbb R^d$ to $\mathbb R^k$ using a $d\times k$ projection matrix $R$ whose ...
0
votes
1answer
16 views

Equation for solving for a variable value of a GLMM probability equation

I am trying to solve for a value of x given all the values of the other x's and at a set value for πij. For example .5 = e^(20 + 10(1) + 15(1) + 20(x3) + 3 + 2)/ 1+ e^(20 + 10(1) + 15(1) + 20(x3) + ...
1
vote
0answers
44 views

Change in eigenvalues due to perturbation to a correlation matrix

Let $A$ be a $m \times n$ matrix defined as $ A = \Big[\frac{a_1}{\|a_1\|} \cdots \frac{a_n}{\|a_n\|}\Big]$ and $a_k \in \mathbb{R}^{m\times 1}$ where $k \in [1,\dots,n]$. Now, we define a ...
0
votes
1answer
31 views

Concept of square in multivariate statistics

This might be more of a linear algebra question, but here we go. I have always been confused about how the concept of squares in $\mathbb{R}^1$ sometimes corresponds to a matrix product $A^{T}A$ and ...
0
votes
0answers
19 views

Effect of PCA on Bias and Variance tradeoff of a model?

My naïve understanding is that PCA are the eignenvectors with the highest eigenvalues. Say that I have 5 predictors for a target variable. I then have 5 pairs of Eigenvectors and Eigenvalues. Say ...
1
vote
1answer
32 views

A proof of within-cluster sum of squares?

Anyone can provide a proof of the following equation as in @cardinal 's answer? $x_i$ and $x_j$ are vectors from the same clusters。 $\sum_{i,j} ||x_i - x_j||^2 = \sum_{i \neq j} ||(x_i - \bar{x}) - (...
0
votes
1answer
80 views

Relation between t-value and correlation coefficient r

Page 6 of this article shows that following holds for an "independent samples t-test". I was wondering how the equation to the Right of equal sign would change if we consider a "paired samples t-...
0
votes
0answers
15 views

singular within scatter matrix and non singular total scatter matrix

When is the scatter matrix in linear discriminant analysis singular although total scatter matrix is non singular ? on which conditions this happens? Or can you introduce me a book or paper to read ...
0
votes
0answers
23 views

when do the principal components of PCA form a basis for the dataset?

Suppose I do a PCA on a data set and get $k$ principal components that explain 100% of the total variance of the data set. We can say any observation from the data set can be reconstructed by the ...
0
votes
0answers
16 views

The miracle of the Lanczos/conjugate gradient algorithm

Lanczos/Arnoldi/Rietz/CG-like algorithm share the same core strategy... In each, a little miracle appears, most of the Gram-Schmidt inner products are zeroes ! In others words, new direction need only ...
1
vote
0answers
30 views

How to understand Jacobian Matrix from the geometric perspective?

I found a good lecture about Jacobian Matrix which was part of a statistics course. However, it was published 20 years ago and lack of explanation. As a beginner of statistics, I'm not able to find ...
0
votes
0answers
36 views

Decomposition of a diagonal matrix

I want to decompose a diagonal matrix $\Lambda \in R^{n \times n}$ such that $$ \Lambda \approx A\Sigma A^T $$ where $\Sigma \in R^{k \times k}$ is a diagonal matrix and $A \in R^{n \times k}$ is a ...
0
votes
1answer
63 views

Distribution of the dot product of a multivariate gaussian random variable and a fixed vector

If $a$ is a multivariate normal random variable, and $x$ is a plain old vector (of the same shape as $a$), then the inner product $x \cdot a$ is a random variable. This post on math exchange suggests ...
0
votes
0answers
21 views

If $X_1,\cdots,X_n \sim \mathcal{N}(\mu,1)$, project unbiased estimator $X_1$ on the span of score function and a constant

Question If $X_1,\cdots,X_n \sim \mathcal{N}(\mu,1)$, then project unbiased estimator $X_1$ on the span of the score function and a constant. Partial attempt: It is straightforward to show that ...
1
vote
0answers
78 views

What is the meaning of expression $(X_i−\bar{X})′$?

I came across an expression ($X_i−\bar{X}$)′ while going through covariance matrix calculation. I know $X_i$ is a random variable and $\bar{X}$ is the mean of all variables, but I cant figure out ...
0
votes
0answers
32 views

Does Column ordering matter in QR decomposition?

I am trying to understand if the ordering of columns matters in QR decompsoition. In general it seems that column ordering won't matter. I guess for SVD or any matrix factorization the way columns ...
4
votes
1answer
85 views

Compute the mean of normalized norms of linear transformations of Gaussian random vectors

if $M$ is a $m\times n$ constant matrix and $\eta\sim\mathcal{N}(0,I)$, then does $$\mathbf{E}_{\eta\sim\mathcal{N}}\left[\frac{\lVert M\eta\rVert}{\lVert\eta\rVert}\right]$$ exist? Also, let $x\in \...
0
votes
0answers
95 views

How are functional margin and geometric margin used in SVM?

I believe, geometric margin is euclidean distance between the point and hyperplane, whereas the functional margin just gives the confidence. At which stage is geometric margin and functional margin ...
1
vote
1answer
24 views

Solving an equation to find two unknown weights given an unbiased estimate

Apologies if this is a simple question; I am reviewing out of Seber and Lee's book on regression and I am pretty rusty in my linear algebra Suppose that $X_1, ..., X_n$ have a common mean $\mu$ and ...
1
vote
0answers
35 views

How to obtain the rotation angles from two successive PCA (3D space)

I have a rigid body moves in 3D space over 30 timesteps. I have computed the PCAs of the rigid body over the timescale, so I have now 30 PCA (x, y, and z vectors). Each PCA represents the body at a ...
3
votes
1answer
74 views

Explanation of generalization of Newton's Method for multiple dimensions

I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
4
votes
0answers
22 views

Show that regression coefficients on group-level variables are unaffected by individual-level variation

My question I have a linear regression that contains some regressors that vary only at a group level and some that vary at the individual level. Slide 8 of this suggests that the coefficients on the ...
1
vote
1answer
173 views

Training error remain unchanged when more feature vectors are added

Given a vector of training data y and a corresponding matrix of features X (the ith row of X containing the feature vector associated with observation yi), why the training error (ie the mean squared ...
1
vote
0answers
47 views

PCA = Eigen decomposition of Covariance Matrix is Not True? [closed]

I have a dataset with 400 features. What I did: ...
2
votes
2answers
43 views

Bilinear and K-Mapping Basis in Linear Algebra

I am reading a machine learning paper which has some mathematical terminologies that are proving a little hard for me to understand. I am going to write the lines from the papers here. The policy ...
1
vote
1answer
31 views

Decomposition of vector into product of a function on a matrix and a function on a vector - Possible? [closed]

Say I have access to $N$-dim vector $Y$, $N \times p$ matrix $X$, and $q$-dim vector $Z$. Ultimately, I would like to learn the functions $g,f$ in: $\underset{N\times1}{\underbrace{Y}}=\underset{N\...
1
vote
0answers
27 views

Role of rank condition in identification of 2SLS - matrix algebra

I could write down all the steps for identification of the 2SLS estimator but my question is really a matrix algebra question which is required in the last step for finding out what the beta vector is ...
1
vote
0answers
18 views

Finding Lag Polynomial Roots = Cayley-Hamilton?

In my time series class we defined the lag polynomial by $\phi(L) = \sum_{i=0}^N\phi_i L^i$. It is well known that this polynomial can be factored as by $\prod_{i=0}^N(I - \lambda_i L)$ (note that $I$ ...
4
votes
2answers
94 views

Eckart-Young-Mirsky theorem: rank $≤k$ or rank $=k$

The Eckart-Young-Mirsky theorem is sometimes stated with rank $\le k$ and sometimes with rank $= k$. Why? More specifically, given a matrix $X \in \mathbb{R}^{n \times d}$, and a natural number $k \...
5
votes
1answer
88 views

Interpreting a matrix calculation

I recently came across this problem, although trivial to compute by hand - is a little challenging for me to interpret. Notably, we have three matrices: $$\vec{c}= \begin{bmatrix} 0.5 \\ 0....
1
vote
0answers
21 views

Integration of the product of Two multivariable Gaussain pdfs

I want to calculate \begin{align} \int_{-\infty}^{\infty} G(x-v_i, \Sigma_i) G(x-v_j, \Sigma_j) dx \end{align} where \begin{align} G(x-v_i, \Sigma_i) = \frac{1}{(2\pi)^{d/2} |\Sigma_i|^{1/2}} \...
0
votes
0answers
16 views

On expected values of triple Kronecker Products

Consider a random vector $\boldsymbol{x} \in \mathbb{R}^N$ and the identity matrix $\boldsymbol{I}_N \in \mathbb{R}^{N\times N}$. I have to compute the expected value of the following Kronecker ...
1
vote
0answers
30 views

Extracting latent vectors from autoencoder similar to SVD

I have read that there is an equivalency between a linear autoencoder and performing SVD. SVD can be used in collaborative filtering, for example, factorization of a user-movies matrix $\mathbf{M}$ ...