Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

203 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
0
votes
0answers
47 views

Multi-class LDA (pairwise classification)

From this post: $ w=S_{W}^{-1}(μ1−μ2), $ is used to estimate $w_{0}=\frac{1}{2}(μ_{1}−μ_{2})^{T}S_{W}^{-1}(μ_{1}−μ_{2})−log(\frac{P1}{P2}),$ However, this is for a situation where there are only 2 ...
0
votes
0answers
29 views

Why PCA(Principal component analysis) can reduce the linear relationship between variables

Assuming we have centralized data,the covariance matrix of the sample is X'X. This is Because: $$ Cov(X)=\frac{1}{n-1} \begin{pmatrix} X_1'X_1 &...&X_1'X_n \\ ...&...&...\\ X_n'X_1&...
0
votes
0answers
24 views

How to find a non-linear manifold for an implicit linear function in the neighborhood of a seed point?

I am trying to understand functional analysis as an infinite-dimensional extension of linear-analysis. In this process, I came across the above query and willing to get a solution and validation of ...
0
votes
1answer
20 views

Linear Regression Coefficients

In simple linear regression, we have that given some $(n \times 1)$ matrix of response observations $y$ and a $(n \times p)$ matrix of observations $x$, the least-squares solution for $\beta$ is $$\...
0
votes
0answers
61 views

Laplace transform of sum of $N$ IID random variables where $N$ is itself random

Let $\{Y_i\}$ be a sequence of IID random variables so that $Y_i \sim Exp(\lambda)$ or equivalently $Y_i \sim CPH_1(1, -\lambda)$ (continuous phase-type distribution). Let $N$ be a discrete random ...
0
votes
0answers
31 views

What are some beginner's references on algebraically structured models, and what's their connection with group actions and Fourier transform?

Since I didn't get an answer, I asked it on mathoverflow. I'm looking at a short-term position, whose project is on estimation in algebraically structured models. It also mentions in the required ...
0
votes
0answers
58 views

Why does the smallest eigenvalue of the covariance matrix of a GMM equal the common variance?

I've been reading a paper "Introduction to Tensor Decompositions and Their Applications in Machine Learning". In it, the author describes an algorithm for estimating the means of the ...
0
votes
0answers
16 views

Show that ABA=0 if and only if B is of the form C-PCP for some symmetric matrix C

If B is a symmetric matrix such that ABA is well-defined, then show that ABA=0 if and only if B is of the form C-PACPA for some symmetric matrix C. PA here is the projection matrix of A.
0
votes
0answers
86 views

Probabilities concerning n-spheres

Let $S_{n-1}(R)=\{ {\bf x}\in {\mathbb{R}}^n : ||{\bf x}||^2=R^2\}$ be the sphere in ${\mathbb{R}}^n$ with radious $R>0.$ Let the projection map $\tau_m({\bf x})=(x_1,...,x_m)$ with $m\leq n.$ Now, ...
0
votes
0answers
16 views

Visualizing data using vectors

Say there are 10 houses and we have three pieces of information for each of them, area, nbedrooms, price I can view this as 10 different vectors in space where there are 3 axes. Basically 10 arrows ...
0
votes
0answers
44 views

Determine if orthogonal projection and full rank

Let $X=(X_1,X_2)$ be a $n \times p$ matrix of rank $p$, where $X_1$ is $n\times p_1$ and $X_2$ is $n\times p_2$. Let $P_1$ be orthogonal projection onto $C(X_1)$ and $P_2$ be orthogonal projection ...
0
votes
0answers
34 views

Using matrices to find estimates in GLMs

How do we use matrices to find estimates in Linear Models? A 4 × 400 relay race is run as follows. There are four runners, each of whom runs 400 meters. The first runner carries a baton, which she ...
0
votes
1answer
27 views

how to normalize data such that an estimated OLS regression vector has pre-specified length (= L_2 norm)

I have the following data: $n$ observations on $d$ variables $X$ and one outcome variable $Y$; i.e. $X$ is a $n \times d$ matrix and $Y$ an $n \times 1$ vector. I consider the following Ordinary Least ...
0
votes
0answers
30 views

How vector projection works behind SVD?

I was reading a blog on mathematical intuition behind SVD. Here, author pointed out three information we get after vector decomposition. The directions of projection — the unit vectors (v₁ and v₂) ...
0
votes
0answers
24 views

Condition numbers, invertibility and multicollinearity

The following is an excerpt from Greene's Regression Analysis (Seventh Edition): a) What does it mean to be "difficult" to invert a matrix accurately? Shouldn't all matrixes be either ...
0
votes
0answers
77 views

Good concise, big picture, linear algebra book?

I have looked at this answer and am not satisfied with the results. Reference book for linear algebra applied to statistics? I have briefly looked at two of the books suggested by the answer, the one ...
0
votes
0answers
16 views

Algebra for Intervention Effect in Interrupted Time Series (with delayed effect)

I have run an interrupted time series analysis using GLS regression model in R. My data consists of 48 observations [time 1:48], with the intervention implemented at time 20, but it's effect not ...
0
votes
0answers
81 views

PCA dimension reduction on correlation matrix for invertability

I have a non-singular (correlation) matrix $C$ of dimension $N{\times}N$, this is a modified version of another correlation matrix, and therefore I don't think I am able to apply any calculations on ...
0
votes
0answers
23 views

Published source for D-dimensional behaviour of Dot-Product

I am currently studying the behaviour of the dot product between two random vectors in $R^d$. Specifically I wanted to start with the case of uniform random vectors on $\mathcal{S}^{d-1}$. I found ...
0
votes
0answers
29 views

Must an interpolator exist with linearly independent basis functions?

I have data $(x_1,y_1), \dots, (x_n, y_n) \in \mathbb R^2$ and the $x_i$ are distinct and increasing. I want to interpolate the $y_i$ with a function $$ f(x) = \sum_{i=1}^Na_i h_i(x) $$ where the $...
0
votes
0answers
33 views

Regularizing the difference in the norms of two independent weight matrices in a neural network

Say, there are two neural network layers with weights $W_1$ and $W_2$. These two layers are part of a larger network but their inputs are completely independent of each other and their outputs could ...
0
votes
1answer
46 views

What can you say about spread of data by looking at singular values and clusters?

I have dataset 250X5 and its singular values are [200 50 25.2 2.3 0.35]. Singular values are directly related to variance. Can you say something about the clustering of data and how much is the spread ...
0
votes
0answers
37 views

Matrix formula for the correlation matrix, page 72 from Mathematical Tools for Applied Multivariate Analysis

Book information The book is titled: Mathematical Tools for Applied Multivariate Analysis By Paul E. Green and J. Douglas Carroll The print that I have is from 1976 The ISBN number is ...
0
votes
0answers
41 views

Principal Component Analysis for a Vector with Mean 0

If we are deriving the first principal component for a vector $X$ with mean 0 and covariance matrix $\Sigma$, can we find it in any other form than $Y^{T}X$ where $Y^{T}$ is the eigenvector ...
0
votes
0answers
98 views

What is the first order derivative of linear regression's cost function using matrix calculus?

For linear regression's cost function J(b), where X is a n*m matrix, b is a m*1 vector and y is n*1 vector: First order derivative with respect to vector b (coefficients) is shown to be Using the ...
0
votes
0answers
47 views

Variance of Least Squares Estimate

Universally the literature seems to make a jump in the proof of variance of the least squares estimator and I'm hoping you can fill in the gaps for me. In matrix form, the least squares estimate is: ...
0
votes
0answers
17 views

covariance in 2 dimensional field

I want to know is the covariance equation which is shown below is how valid when we are in 2-dimensional fields (not 2-dimensional data)? $$Cov(x,y)=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-y)}{N-1}$$ ...
0
votes
1answer
27 views

Why do we get a new co-ordinate system when we dot product the transpose of eigen vectors with the transpose of a matrix

I am working on implementing PCA on the MNIST dataset and have calculated the eigen vector and eig Values from the co-variance matrix. Now I want to have a new co-ordinate system represented by PC1 ...
0
votes
0answers
37 views

Two layer network working in PyTorch, but cannot reproduce using pure linear algebra

Rewording the original question to comply with the CV topics. I am trying to build a simple 2 layer network, and was going through the backprop mathematics of it, when I got stuck. I am defining my ...
0
votes
0answers
32 views

eigen value decomposition of co-variance a series generated by factor model

Let's assume $N\times T$ series $Y_t$ is generated by the following equation. $$ Y_t = \begin{bmatrix}A_x & A_m\end{bmatrix}\begin{bmatrix}x_t \\ m_t \end{bmatrix}$$ Where $A_x$ and $A_m$ are $N\...
0
votes
0answers
80 views

Alternating Least Squares for Baseline Predictor

I am trying to figure out how ALS works when minimizing the following formula: $\\ \\$ $\text{min}_{\lbrace b_u,b_i \rbrace} \sum_{(u,i)\in \mathcal{K}} (r_{ui} - \bar{r} - b_u - b_i )^2 + \lambda_{...
0
votes
0answers
42 views

what is a vector perpendicular to a plane of vectors

I have 3 or 4 vectors connected that forms a plane. How can I find the vector that is perpendicular to this plane? it can be a unit vector as long as it preserves this direction. each vector is on ...
0
votes
1answer
57 views

Want to make sense of array dimensions in logistic regression algorithms

I am trying to implement a simple logistic regression algorithm from scratch in python (for learning purposes). Every article I've seen online so far presents the following expression for $z$ (...
0
votes
0answers
67 views

Write Lack-of-fit Sum of Squares in Quadratic Form

Let \begin{equation} SSLF = \sum_{i=1}^{m}n_{i}(\bar{y_{i}} - \hat{y_{i}})^{2} \end{equation} then \begin{equation} \sum_{i=1}^{m}n_{i}(\bar{y_{i}} - \hat{y_{i}})^{2} = n(\bar{\overrightarrow{y}} -...
0
votes
0answers
42 views

Cosine similarity matrix of linearly transformed inputs

Given a matrix $\mathbf{C}$ which contains pairwise cosine similarities between rows of a matrix $\mathbf{A}$, linearly transformed by matrix $\mathbf{U}$: $$ \mathbf{C} = K(\mathbf{UA}, \mathbf{UA}) $...
0
votes
0answers
41 views

Additional Property of Singular Value Decomposition

I am new to SVD so forgive me if the question is trivial. Following is my question. If I have two sets of linear equations, Y1 = T1.X Y2 = T2.X where T1 and T2 are mxn rectangular matrices. Now let'...
0
votes
0answers
139 views

Idea behind change of basis and how it relates to projecting your points onto principal components

I would like to clarify if my understanding is correct. In the traditional X-Y coordinate system, our choice of basis vectors are $\vec{i} = (1, 0)$ and $\vec{j} = (0, 1)$ and when you I have a point $...
0
votes
1answer
62 views

Concept of square in multivariate statistics

This might be more of a linear algebra question, but here we go. I have always been confused about how the concept of squares in $\mathbb{R}^1$ sometimes corresponds to a matrix product $A^{T}A$ and ...
0
votes
1answer
65 views

The miracle of the Lanczos/conjugate gradient algorithm

Lanczos/Arnoldi/Rietz/CG-like algorithm share the same core strategy... In each, a little miracle appears, most of the Gram-Schmidt inner products are zeroes ! In others words, new direction need only ...
0
votes
0answers
192 views

Does Column ordering matter in QR decomposition?

I am trying to understand if the ordering of columns matters in QR decompsoition. In general it seems that column ordering won't matter. I guess for SVD or any matrix factorization the way columns ...
0
votes
0answers
181 views

Normal Equation - Whats the problem with having more than one solution?

I know that if the X matrix when using the normal equation is dependent then the transpose(X)*X will be non invertible and as a result there will be an infinite number of solutions to the problem. I ...
0
votes
0answers
176 views

Linear regression with feature representation confusion - relationship of design matrix column space to the feature space?

I am trying to visualise the geometry of linear regression with feature representation. I have a regression problem with $n$ data pairs $\mathcal{D}:=\{(\mathbf{x},y)_{i}\}_{i=1}^{n}$, independent ...
0
votes
0answers
50 views

Solving weights in least squares to reduce distance to another vector

I'm using a procedure of WLS to find some values in a reduced space that actually generate a vector that is close to my target vector. That is, I have WLS: $\beta=(X^T W X)^{-1} X^T W Y$, and I am ...
0
votes
0answers
10 views

Relationships between sub-samples with limited information

I'm doing an analysis where I've been able to assemble the following information about a population, its overall conversion rate (so, the percent of the members of that population who go on to make a ...
0
votes
0answers
1k views

Confused about finding expectation of vector multiplied by matrix

I am little confused regarding finding expectation of vector multiplied matrix. X is vector having mean $m_x \in \mathbb{R}^{N\times 1}$ and variance vector $s_x \in \mathbb{R}^{N\times N}$. $\phi$ is ...
0
votes
0answers
38 views

Is $Φ^T$ a linear operator which transforms simultaneous equations such that we obtain LMS solution?

If we have $\vec e(w) = \vec t-\Phi \vec w$ and we wish to minimize $\vec e(w)^T \vec e(w) $ w.r.t $\vec w$, This is same as wanting the best fit solution to the the set of $n$ simultaneous ...
0
votes
0answers
388 views

Find equation of line that separates two Classes in SVM

Hence, $w_1=w_2 = -.5$ and $b=7/2$. I'm confused as to how we know that $w_1=w_2$ from $x_1+x_2=7$?
0
votes
1answer
160 views

how to modify covariance matrix of PC scores after rotation of axes

I have performed principal component analysis on a set of observations, retained four principal components and estimated covariance matrix of their scores. Then I have rotated the axes so that the ...
0
votes
0answers
37 views

Reconstruct a vector with a known vector and residual

I observe $\vec y$ and know $\vec x$. I assume that $\vec y$ mostly consists of $\vec x$, with some added residual $\vec r$. This gives me the problem $\vec y = a\vec x + \vec r$, where $a \in [0, 1]$...
0
votes
0answers
52 views

Solving a difficult equation for a variable?

I'm trying to obtain the maximum likelihood estimate of the parameters for a model I'm building. I have constants $\sigma$, $\mu$, and $q_0$; a boolean matrix $\alpha$; and vectors $A, \beta, r, d,$ ...