Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

1
vote
0answers
60 views

Why high dimensional data is multi-collinear? [duplicate]

If the number of predictors (P) is greater than the number of observations (N) (P > N), ...
2
votes
0answers
98 views

Does anyone know the rank of the Netflix Prize dataset?

I'm looking into the Netflix Prize at the moment. We model the dataset as an $n \times m$ matrix, where $n$ is the number of users and $m$ is the number of movies. Does anyone know the rank of the ...
1
vote
0answers
46 views

Distribution of dot products of two random independent unit vectors in $D$ dimensions

Duplicate of the stats stack exchange question here; however, I need some help with some of the steps in the accepted answer. A uniform distribution on the unit sphere $\mathbb{S}^{D-1}$ is ...
0
votes
2answers
32 views

Is there a mistake in the expression of this variance?

I'm busy reading through an econometrics textbook (page 147), and I don't understand the step $$\mathrm {Var}\left(n^{\frac 12}\left(\hat\beta - \beta\right)\right) = \boldsymbol{A^{-1}}\sigma^2\...
0
votes
1answer
36 views

Want to make sense of array dimensions in logistic regression algorithms

I am trying to implement a simple logistic regression algorithm from scratch in python (for learning purposes). Every article I've seen online so far presents the following expression for $z$ (...
4
votes
2answers
415 views

Inverse of the covariance matrix of a multivariate normal distribution

Is the covariance matrix of a multivariate normal distribution always invertible?
4
votes
1answer
46 views

Incremental solution for matrix inverse using Shermann-Morrison in $O(n^2)$

I have been reading a presentation on Value Function Approximation by David Silver (Introduction to Reinforcement Learning Course). On page 43 he finds a solution for linear least squares for an ...
0
votes
1answer
43 views

What is matrix norm in (Collins, 2001)

I am studying "A generalization of PCA to the exponential family" (Collins et al., 2001) and I don't understand some notations. What is the meaning of the matrix squared norm on page 6 ? Is it a ...
0
votes
0answers
36 views

Write Lack-of-fit Sum of Squares in Quadratic Form

Let \begin{equation} SSLF = \sum_{i=1}^{m}n_{i}(\bar{y_{i}} - \hat{y_{i}})^{2} \end{equation} then \begin{equation} \sum_{i=1}^{m}n_{i}(\bar{y_{i}} - \hat{y_{i}})^{2} = n(\bar{\overrightarrow{y}} -...
2
votes
1answer
120 views

Mean and Variance of SSE

First, let \begin{equation} SSE = \overrightarrow{y}'(I - H) \overrightarrow{y} \end{equation} where \begin{equation} \overrightarrow{y} \sim MN(\textbf{X} \overrightarrow{\beta}, \sigma^{2}I) \\ H ...
0
votes
0answers
5 views

Is sketching a method for dimensionality reduction and its relation to random projection

I want to know if sketching can be categorized as a method of dimensionality reduction and more specifically feature extraction. Also, i want to understand if its related to random projection.
0
votes
0answers
20 views

Cosine similarity matrix of linearly transformed inputs

Given a matrix $\mathbf{C}$ which contains pairwise cosine similarities between rows of a matrix $\mathbf{A}$, linearly transformed by matrix $\mathbf{U}$: $$ \mathbf{C} = K(\mathbf{UA}, \mathbf{UA}) $...
1
vote
1answer
28 views

Reference point in projection axis of SVD (singular value decomposition)

I am watching a YouTube video on SVD, and attempting to recreate some of its examples to better understand the internal machinery of the algorithm. In one of the slides, the instructor mentions that ...
1
vote
0answers
30 views

Higher moments of linear regression residuals?

I previously asked this on Math StackExchange, with no success, but this post will add to that with some simulations. Background In the following linear regression with i.i.d $\epsilon_i$ $(i = 1, \...
0
votes
0answers
23 views

Solving for variable in a multivariate adaptive regression spline model with logistic link function

I am working on solving for a variable within a MARS model where the probability is set at .5 and so far I have the following equation: log(0.5/0.5) = -1.31 + -2.81(var1 - -0.81) + 4.47(.04 – var2) + ...
4
votes
2answers
74 views

Steps of Matrix Multiplication

It may seem kind of silly, but can anyone please show me the intermediate steps implied by the second equality in this derivation? $$e^\prime e = \left(y - Xb\right)^\prime\left(y - Xb\right) = y^\...
3
votes
1answer
65 views

The form of the Log-Likelihood Function in Mixed Linear Models

Let us assume the following mixed effects model: $y = X\beta+Zu+e$ where $y$ is a vector of n observable random variables, $\beta$ is a vector of $p$ fixed effects, $X$ and $Z$ are known matrices, ...
1
vote
1answer
27 views

Given $m$ $n$-dimensional vectors, how to create a vector perpendicular to all of them?

Given $m$ vectors, $x_1$, $x_2$, ... $x_m$ with all $x_i \,\, \epsilon \,\, \mathcal{R}^n$, $i=1,2... m$ and $m < n$. How to sample a vector $x_{m+1}$ perpendicular to all the vectors $x_1$, $...
0
votes
0answers
23 views

Additional Property of Singular Value Decomposition

I am new to SVD so forgive me if the question is trivial. Following is my question. If I have two sets of linear equations, Y1 = T1.X Y2 = T2.X where T1 and T2 are mxn rectangular matrices. Now let'...
3
votes
2answers
79 views

What is the Joint Density Function of a Three-Level Mixed-Effects Model?

This is a follow-up question to a question I posted earlier. Obviously, maximum-likelihood estimation of mixed-effects models requires the joint density function. Let us assume a two-level mixed ...
0
votes
0answers
30 views

Idea behind change of basis and how it relates to projecting your points onto principal components

I would like to clarify if my understanding is correct. In the traditional X-Y coordinate system, our choice of basis vectors are $\vec{i} = (1, 0)$ and $\vec{j} = (0, 1)$ and when you I have a point $...
1
vote
1answer
52 views

Is this dataset lineary seperable? How can I find it out using (linear) algebra?

I have this dataset: I want to know if it is linearly separable (fully separable). I want to use this rule, but I'm not sure if it's correct: Make $X'$ - matrix with d+1 column of all 1's. Then ...
5
votes
1answer
111 views

Mixed Models: How to derive Henderson's mixed-model equations?

In the context of best linear unbiased predictors (BLUP), Henderson specified the mixed-model equations (see Henderson (1950): Estimation of Genetic Parameters. Annals of Mathematical Statistics, 21, ...
1
vote
1answer
25 views

Problem with Softmax decision boundary

While reading this paper: sphere face on page 2, it explains that original softmax boundary is given by: $$(W_1 −W_2)x+b_1 −b_2 = 0$$ While trying to obtain the boundary on a toy generated 2D dataset ...
7
votes
2answers
734 views

What is the problem with $p > n$?

I know that this is the solving system of linear equation problem. But my question is why it is a problem the number of observation is lower than the number of predictors how can that thing happen? ...
1
vote
0answers
55 views

Multilevel Modeling: Minimization Problem when A = B + TCT'

I currently study multilevel model using Leeuw & Meijer (2008) Handbook of multilevel analysis. On page 65, they state the following theorem: If $A = B + TCT'$ with $A$ and $B$ positive definite, ...
-1
votes
1answer
25 views

How to cluster a (directional) dissimilarity matrix with both positive and negative values?

I may be thinking of this incorrectly but what would be the best way to cluster a dissimilarity measure that has direction? For example, if someone had condition A ...
0
votes
0answers
28 views

Is there a problem with the “low-rank matrix approximation”?

I know that "rank" is the number or independent rows in the matrix and I know that the resulting matrix of the "low-rank matrix approximation" algorithm has lower rank than the original matrix. 1- But ...
-1
votes
1answer
32 views

PCA with zero and high correlation in data

How would the eigen values look like when we apply PCA to a dataset with zero correlation between variables and when there is very high correlation between variables
9
votes
4answers
448 views

Is “random projection” strictly speaking not a projection?

Current implementations of the Random Projection algorithm reduce the dimensionality of data samples by mapping them from $\mathbb R^d$ to $\mathbb R^k$ using a $d\times k$ projection matrix $R$ whose ...
0
votes
1answer
24 views

Equation for solving for a variable value of a GLMM probability equation

I am trying to solve for a value of x given all the values of the other x's and at a set value for πij. For example .5 = e^(20 + 10(1) + 15(1) + 20(x3) + 3 + 2)/ 1+ e^(20 + 10(1) + 15(1) + 20(x3) + ...
1
vote
0answers
52 views

Change in eigenvalues due to perturbation to a correlation matrix

Let $A$ be a $m \times n$ matrix defined as $ A = \Big[\frac{a_1}{\|a_1\|} \cdots \frac{a_n}{\|a_n\|}\Big]$ and $a_k \in \mathbb{R}^{m\times 1}$ where $k \in [1,\dots,n]$. Now, we define a ...
0
votes
1answer
37 views

Concept of square in multivariate statistics

This might be more of a linear algebra question, but here we go. I have always been confused about how the concept of squares in $\mathbb{R}^1$ sometimes corresponds to a matrix product $A^{T}A$ and ...
0
votes
0answers
30 views

Effect of PCA on Bias and Variance tradeoff of a model?

My naïve understanding is that PCA are the eignenvectors with the highest eigenvalues. Say that I have 5 predictors for a target variable. I then have 5 pairs of Eigenvectors and Eigenvalues. Say ...
1
vote
1answer
50 views

A proof of within-cluster sum of squares?

Anyone can provide a proof of the following equation as in @cardinal 's answer? $x_i$ and $x_j$ are vectors from the same clusters。 $\sum_{i,j} ||x_i - x_j||^2 = \sum_{i \neq j} ||(x_i - \bar{x}) - (...
0
votes
1answer
147 views

Relation between t-value and correlation coefficient r

Page 6 of this article shows that following holds for an "independent samples t-test". I was wondering how the equation to the Right of equal sign would change if we consider a "paired samples t-...
0
votes
0answers
23 views

singular within scatter matrix and non singular total scatter matrix

When is the scatter matrix in linear discriminant analysis singular although total scatter matrix is non singular ? on which conditions this happens? Or can you introduce me a book or paper to read ...
0
votes
0answers
18 views

The miracle of the Lanczos/conjugate gradient algorithm

Lanczos/Arnoldi/Rietz/CG-like algorithm share the same core strategy... In each, a little miracle appears, most of the Gram-Schmidt inner products are zeroes ! In others words, new direction need only ...
1
vote
0answers
64 views

How to understand Jacobian Matrix from the geometric perspective?

I found a good lecture about Jacobian Matrix which was part of a statistics course. However, it was published 20 years ago and lack of explanation. As a beginner of statistics, I'm not able to find ...
0
votes
0answers
39 views

Decomposition of a diagonal matrix

I want to decompose a diagonal matrix $\Lambda \in R^{n \times n}$ such that $$ \Lambda \approx A\Sigma A^T $$ where $\Sigma \in R^{k \times k}$ is a diagonal matrix and $A \in R^{n \times k}$ is a ...
0
votes
1answer
288 views

Distribution of the dot product of a multivariate gaussian random variable and a fixed vector

If $a$ is a multivariate normal random variable, and $x$ is a plain old vector (of the same shape as $a$), then the inner product $x \cdot a$ is a random variable. This post on math exchange suggests ...
1
vote
0answers
79 views

What is the meaning of expression $(X_i−\bar{X})′$?

I came across an expression ($X_i−\bar{X}$)′ while going through covariance matrix calculation. I know $X_i$ is a random variable and $\bar{X}$ is the mean of all variables, but I cant figure out ...
0
votes
0answers
50 views

Does Column ordering matter in QR decomposition?

I am trying to understand if the ordering of columns matters in QR decompsoition. In general it seems that column ordering won't matter. I guess for SVD or any matrix factorization the way columns ...
4
votes
1answer
87 views

Compute the mean of normalized norms of linear transformations of Gaussian random vectors

if $M$ is a $m\times n$ constant matrix and $\eta\sim\mathcal{N}(0,I)$, then does $$\mathbf{E}_{\eta\sim\mathcal{N}}\left[\frac{\lVert M\eta\rVert}{\lVert\eta\rVert}\right]$$ exist? Also, let $x\in \...
0
votes
0answers
109 views

How are functional margin and geometric margin used in SVM?

I believe, geometric margin is euclidean distance between the point and hyperplane, whereas the functional margin just gives the confidence. At which stage is geometric margin and functional margin ...
1
vote
1answer
25 views

Solving an equation to find two unknown weights given an unbiased estimate

Apologies if this is a simple question; I am reviewing out of Seber and Lee's book on regression and I am pretty rusty in my linear algebra Suppose that $X_1, ..., X_n$ have a common mean $\mu$ and ...
1
vote
0answers
50 views

How to obtain the rotation angles from two successive PCA (3D space)

I have a rigid body moves in 3D space over 30 timesteps. I have computed the PCAs of the rigid body over the timescale, so I have now 30 PCA (x, y, and z vectors). Each PCA represents the body at a ...
3
votes
1answer
96 views

Explanation of generalization of Newton's Method for multiple dimensions

I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
4
votes
0answers
25 views

Show that regression coefficients on group-level variables are unaffected by individual-level variation

My question I have a linear regression that contains some regressors that vary only at a group level and some that vary at the individual level. Slide 8 of this suggests that the coefficients on the ...
1
vote
1answer
188 views

Training error remain unchanged when more feature vectors are added

Given a vector of training data y and a corresponding matrix of features X (the ith row of X containing the feature vector associated with observation yi), why the training error (ie the mean squared ...