Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

1
vote
0answers
12 views

Understanding PCA from a linear transformation perspective

I've came across several great questions and answers here regarding PCA, but I would like to have a look at it from a linear transformation perspective. Let's say I have a (demeaned) data matrix $X$ ...
0
votes
0answers
21 views

The least squares estimator for beta in matrix notation in connection to normal equations and orthogonal projection in linear algebra [duplicate]

Somebody posted this link here about DERIVATION OF THE LEAST SQUARES ESTIMATOR FOR BETA IN MATRIX NOTATION-(https://economictheoryblog.com/2015/02/19/ols_estimator/). My question is 'How does THE ...
0
votes
0answers
12 views

Is the prediction with and without mean normalisation different in Collaborative Filtering?

In case of Collaborative Filtering: Given an output matrix I wish to learn parameters $\Theta$ (Parameter Vector) and X (Feature Vector). Now if I mean normalise the output matrix the values of $\...
2
votes
1answer
24 views

Question about the gradient of weight normalization

In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$ \mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v $$ Then they ...
0
votes
0answers
29 views

Alternating Least Squares for Baseline Predictor

I am trying to figure out how ALS works when minimizing the following formula: $\\ \\$ $\text{min}_{\lbrace b_u,b_i \rbrace} \sum_{(u,i)\in \mathcal{K}} (r_{ui} - \bar{r} - b_u - b_i )^2 + \lambda_{...
1
vote
0answers
26 views

Understanding the proof of independence-dimension inequality [migrated]

I'm self studying the book Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares Below is an excerpt from the book: Independence-dimension inequality. If the $n$-vectors $\...
2
votes
1answer
24 views

Struggling on second to last part deriving linear discriminant function

From this post here I am struggling with the matrix multiplication to get from: $\log \pi _{k} - \frac{1}{2}(x-\mu _k)^T{\sum }^{-1}(x-\mu _k)$ to $\log \pi _{k} - \frac{1}{2}[x^{T}{\sum }^{-1}x +\...
0
votes
0answers
16 views

PCA whitened data

Let $X$ be a feature matrix of size $m\times d$. I understand that the standard PCA whitening process follows three steps. (Centerization) ${X} \to \hat{X}:=({X} - \mu)$, where $\mu$ is a matrix of ...
3
votes
1answer
154 views

Chebyshev inequality in terms of RMS

I'm self studying the book Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares In page 48, the author write: "It says,for example, that no more than 1/25 = 4% of the entries ...
0
votes
0answers
26 views

what is a vector perpendicular to a plane of vectors

I have 3 or 4 vectors connected that forms a plane. How can I find the vector that is perpendicular to this plane? it can be a unit vector as long as it preserves this direction. each vector is on ...
0
votes
0answers
6 views

Find the fitting lines in a data with multiple general distribution axis

I have the following data which the breaking point can be varying. How I can find 2 fitting lines? Obviously I can't use minimum square error or RANSAC on raw data, because the work on one major axis.
3
votes
0answers
24 views

Are there extant deep learning analogs to random coefficient (aka mixed) models?

Random coef models, applied to longitudinal data, capture response heterogeneity by cross-sectional unit. I've got a longitudinal prediction problem, in which I know that some "features" (or ...
0
votes
1answer
20 views

how this transform 2d to 3d is work in mathmatically?

this is the 2d->3d projection for svm. they used the kernel trick to change the dimension of the vector for easier classification. I want to understand the detail math behind this projectin which ...
1
vote
2answers
139 views

Proving efficiency of OLS over GLS

I'm trying to prove the efficiency of OLS over GLS when the covariance matrix of the error $\varepsilon$ is mistakenly assumed to be $\sigma^2\Sigma$ instead of $\sigma^2 I$. After deriving the ...
0
votes
1answer
25 views

Matrix Orthogonal to Vector: why take transpose?

In econometrics, we often have n observations (in a column vector $y$) which we want to explain with k$<$n regressors (the observations are in an nxk matrix $X$). In this case we use least squares ...
2
votes
0answers
105 views

Does anyone know the rank of the Netflix Prize dataset?

I'm looking into the Netflix Prize at the moment. We model the dataset as an $n \times m$ matrix, where $n$ is the number of users and $m$ is the number of movies. Does anyone know the rank of the ...
1
vote
0answers
60 views

Distribution of dot products of two random independent unit vectors in $D$ dimensions

Duplicate of the stats stack exchange question here; however, I need some help with some of the steps in the accepted answer. A uniform distribution on the unit sphere $\mathbb{S}^{D-1}$ is ...
0
votes
2answers
36 views

Is there a mistake in the expression of this variance?

I'm busy reading through an econometrics textbook (page 147), and I don't understand the step $$\mathrm {Var}\left(n^{\frac 12}\left(\hat\beta - \beta\right)\right) = \boldsymbol{A^{-1}}\sigma^2\...
0
votes
1answer
36 views

Want to make sense of array dimensions in logistic regression algorithms

I am trying to implement a simple logistic regression algorithm from scratch in python (for learning purposes). Every article I've seen online so far presents the following expression for $z$ (...
4
votes
2answers
536 views

Inverse of the covariance matrix of a multivariate normal distribution

Is the covariance matrix of a multivariate normal distribution always invertible?
4
votes
1answer
57 views

Incremental solution for matrix inverse using Shermann-Morrison in $O(n^2)$

I have been reading a presentation on Value Function Approximation by David Silver (Introduction to Reinforcement Learning Course). On page 43 he finds a solution for linear least squares for an ...
0
votes
1answer
45 views

What is matrix norm in (Collins, 2001)

I am studying "A generalization of PCA to the exponential family" (Collins et al., 2001) and I don't understand some notations. What is the meaning of the matrix squared norm on page 6 ? Is it a ...
0
votes
0answers
42 views

Write Lack-of-fit Sum of Squares in Quadratic Form

Let \begin{equation} SSLF = \sum_{i=1}^{m}n_{i}(\bar{y_{i}} - \hat{y_{i}})^{2} \end{equation} then \begin{equation} \sum_{i=1}^{m}n_{i}(\bar{y_{i}} - \hat{y_{i}})^{2} = n(\bar{\overrightarrow{y}} -...
2
votes
1answer
219 views

Mean and Variance of SSE

First, let \begin{equation} SSE = \overrightarrow{y}'(I - H) \overrightarrow{y} \end{equation} where \begin{equation} \overrightarrow{y} \sim MN(\textbf{X} \overrightarrow{\beta}, \sigma^{2}I) \\ H ...
0
votes
0answers
5 views

Is sketching a method for dimensionality reduction and its relation to random projection

I want to know if sketching can be categorized as a method of dimensionality reduction and more specifically feature extraction. Also, i want to understand if its related to random projection.
0
votes
0answers
20 views

Cosine similarity matrix of linearly transformed inputs

Given a matrix $\mathbf{C}$ which contains pairwise cosine similarities between rows of a matrix $\mathbf{A}$, linearly transformed by matrix $\mathbf{U}$: $$ \mathbf{C} = K(\mathbf{UA}, \mathbf{UA}) $...
2
votes
1answer
47 views

Reference point in projection axis of SVD (singular value decomposition)

I am watching a YouTube video on SVD, and attempting to recreate some of its examples to better understand the internal machinery of the algorithm. In one of the slides, the instructor mentions that ...
1
vote
0answers
39 views

Higher moments of linear regression residuals?

I previously asked this on Math StackExchange, with no success, but this post will add to that with some simulations. Background In the following linear regression with i.i.d $\epsilon_i$ $(i = 1, \...
0
votes
0answers
28 views

Solving for variable in a multivariate adaptive regression spline model with logistic link function

I am working on solving for a variable within a MARS model where the probability is set at .5 and so far I have the following equation: log(0.5/0.5) = -1.31 + -2.81(var1 - -0.81) + 4.47(.04 – var2) + ...
4
votes
2answers
74 views

Steps of Matrix Multiplication

It may seem kind of silly, but can anyone please show me the intermediate steps implied by the second equality in this derivation? $$e^\prime e = \left(y - Xb\right)^\prime\left(y - Xb\right) = y^\...
3
votes
1answer
79 views

The form of the Log-Likelihood Function in Mixed Linear Models

Let us assume the following mixed effects model: $y = X\beta+Zu+e$ where $y$ is a vector of n observable random variables, $\beta$ is a vector of $p$ fixed effects, $X$ and $Z$ are known matrices, ...
1
vote
1answer
28 views

Given $m$ $n$-dimensional vectors, how to create a vector perpendicular to all of them?

Given $m$ vectors, $x_1$, $x_2$, ... $x_m$ with all $x_i \,\, \epsilon \,\, \mathcal{R}^n$, $i=1,2... m$ and $m < n$. How to sample a vector $x_{m+1}$ perpendicular to all the vectors $x_1$, $...
0
votes
0answers
26 views

Additional Property of Singular Value Decomposition

I am new to SVD so forgive me if the question is trivial. Following is my question. If I have two sets of linear equations, Y1 = T1.X Y2 = T2.X where T1 and T2 are mxn rectangular matrices. Now let'...
3
votes
2answers
98 views

What is the Joint Density Function of a Three-Level Mixed-Effects Model?

This is a follow-up question to a question I posted earlier. Obviously, maximum-likelihood estimation of mixed-effects models requires the joint density function. Let us assume a two-level mixed ...
0
votes
0answers
33 views

Idea behind change of basis and how it relates to projecting your points onto principal components

I would like to clarify if my understanding is correct. In the traditional X-Y coordinate system, our choice of basis vectors are $\vec{i} = (1, 0)$ and $\vec{j} = (0, 1)$ and when you I have a point $...
1
vote
1answer
55 views

Is this dataset lineary seperable? How can I find it out using (linear) algebra?

I have this dataset: I want to know if it is linearly separable (fully separable). I want to use this rule, but I'm not sure if it's correct: Make $X'$ - matrix with d+1 column of all 1's. Then ...
5
votes
1answer
193 views

Mixed Models: How to derive Henderson's mixed-model equations?

In the context of best linear unbiased predictors (BLUP), Henderson specified the mixed-model equations (see Henderson (1950): Estimation of Genetic Parameters. Annals of Mathematical Statistics, 21, ...
1
vote
1answer
53 views

Problem with Softmax decision boundary

While reading this paper: sphere face on page 2, it explains that original softmax boundary is given by: $$(W_1 −W_2)x+b_1 −b_2 = 0$$ While trying to obtain the boundary on a toy generated 2D dataset ...
7
votes
2answers
745 views

What is the problem with $p > n$?

I know that this is the solving system of linear equation problem. But my question is why it is a problem the number of observation is lower than the number of predictors how can that thing happen? ...
1
vote
0answers
55 views

Multilevel Modeling: Minimization Problem when A = B + TCT'

I currently study multilevel model using Leeuw & Meijer (2008) Handbook of multilevel analysis. On page 65, they state the following theorem: If $A = B + TCT'$ with $A$ and $B$ positive definite, ...
-1
votes
1answer
42 views

How to cluster a (directional) dissimilarity matrix with both positive and negative values?

I may be thinking of this incorrectly but what would be the best way to cluster a dissimilarity measure that has direction? For example, if someone had condition A ...
0
votes
0answers
33 views

Is there a problem with the “low-rank matrix approximation”?

I know that "rank" is the number or independent rows in the matrix and I know that the resulting matrix of the "low-rank matrix approximation" algorithm has lower rank than the original matrix. 1- But ...
-1
votes
1answer
37 views

PCA with zero and high correlation in data

How would the eigen values look like when we apply PCA to a dataset with zero correlation between variables and when there is very high correlation between variables
10
votes
4answers
538 views

Is “random projection” strictly speaking not a projection?

Current implementations of the Random Projection algorithm reduce the dimensionality of data samples by mapping them from $\mathbb R^d$ to $\mathbb R^k$ using a $d\times k$ projection matrix $R$ whose ...
0
votes
1answer
24 views

Equation for solving for a variable value of a GLMM probability equation

I am trying to solve for a value of x given all the values of the other x's and at a set value for πij. For example .5 = e^(20 + 10(1) + 15(1) + 20(x3) + 3 + 2)/ 1+ e^(20 + 10(1) + 15(1) + 20(x3) + ...
1
vote
0answers
55 views

Change in eigenvalues due to perturbation to a correlation matrix

Let $A$ be a $m \times n$ matrix defined as $ A = \Big[\frac{a_1}{\|a_1\|} \cdots \frac{a_n}{\|a_n\|}\Big]$ and $a_k \in \mathbb{R}^{m\times 1}$ where $k \in [1,\dots,n]$. Now, we define a ...
0
votes
1answer
44 views

Concept of square in multivariate statistics

This might be more of a linear algebra question, but here we go. I have always been confused about how the concept of squares in $\mathbb{R}^1$ sometimes corresponds to a matrix product $A^{T}A$ and ...
0
votes
0answers
40 views

Effect of PCA on Bias and Variance tradeoff of a model?

My naïve understanding is that PCA are the eignenvectors with the highest eigenvalues. Say that I have 5 predictors for a target variable. I then have 5 pairs of Eigenvectors and Eigenvalues. Say ...
1
vote
1answer
77 views

A proof of within-cluster sum of squares?

Anyone can provide a proof of the following equation as in @cardinal 's answer? $x_i$ and $x_j$ are vectors from the same clusters。 $\sum_{i,j} ||x_i - x_j||^2 = \sum_{i \neq j} ||(x_i - \bar{x}) - (...
0
votes
1answer
199 views

Relation between t-value and correlation coefficient r

Page 6 of this article shows that following holds for an "independent samples t-test". I was wondering how the equation to the Right of equal sign would change if we consider a "paired samples t-...