Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
16 views

How to calculate ${w}^\top \mathrm{\Sigma} w$? [closed]

I have: A vector $w_{2\times1} = [[w_1], [w_2]]$ A matrix $ \mathrm{R_{2\times21}} = \left[ {\begin{array}{ccccc} [\mathrm{R_1}], [\mathrm{R_2}] \end{array} } \right] $ where $\mathrm{R_1}$ and ...
0 votes
0 answers
16 views

Does invariance of PCA under orthogonal transformation hold for data that is not centered?

I read the proof in the top answer to this question, but that page assumes that $\overline{A} = 0$. If the data instead has some nonzero mean $\mu$, I'm not sure if the same logic applies: ...
1 vote
0 answers
27 views

I am not able to understand how did the elementwise multiplication came into the picture of backpropagation in neural networks

I have understood the backpropagation algorithm along with the chain rule well enough that I can derive it on my own, but I don't understand where the elementwise multiplication came from and how does ...
0 votes
1 answer
7 views

Transformation to linearize dataset [duplicate]

How can I transform the following dataset to a more linear representation? R code: ...
  • 1,122
1 vote
0 answers
20 views

Is it possible to apply the kernel trick to a "mahalnobis distance learner" such as GLS?

1.https://arxiv.org/pdf/0804.1441.pdf 2.https://www.sciencedirect.com/science/article/abs/pii/S0925231210001165 These papers describe kernelizing a mahalanobis distance learner. I am interested in ...
0 votes
1 answer
10 views

Sum of multiple covariance matrices looks like identity matrix

Suppose $X$ and $Y$ are two $a \times b$ matrices, randomly sampled from the same normal distribution. I found an interesting phenomenon: If we sum $X X^T$ multiple times, each time $X$ is randomly ...
's user avatar
4 votes
0 answers
40 views

The probability limit of the inverse of an infinite-dimensional matrix

I am considering a question regarding the calculation of the probability limit for a high-dimensional inverse matrix. Specifically, suppose that $A_n, B_n \in \mathbb{R}^{N_n \times N_n}$ where $N_n \...
  • 93
0 votes
0 answers
27 views

IV estimation for when instrument is independent of the outcome

So I came across this post and have some use for it in my own work. Recap of Original Question: In the canonical IV setting, suppose we have a data-generating model $Z\rightarrow X\rightarrow Y$ (...
0 votes
0 answers
11 views

Closure operation of sets is a functor [migrated]

Can be the closure operation $X\to cl(X)$ defined a functor ? For this I assumed that if $F$: $C$$\to$ $C$ be a functor (where $C$ is category of sets), if $A$ and $B$ are two objects of the category $...
2 votes
1 answer
30 views

What is the difference between these two observational update models in an information filter?

The Information Filter is defined as the mathematical inverse of the Kalman filter. As defined in this Wikipedia article, the observation update of the Information Matrix is defined as $$y_{k|k} = y_{...
1 vote
1 answer
55 views

Variance calculation in matrix notation for $var(z-Ax)$

I noted from a post here that $$var(z - \mathbf{A}x)=var(z)+var(\mathbf{A}x)-\mathbf{A}cov(z, -x)-cov(z,-x)\mathbf{A}^T (Eq. 1)$$ (I dropped the conditional part in the original formula from the post ...
  • 197
1 vote
0 answers
36 views

Why does PCA maximize variance between the standard deviations?

Consider an $n \times n$ covariance matrix $\Sigma$ (so semi positive-definite, symmetric and realvalued). We can find the $n$ principle components by $n$ times finding the direction of maximum ...
1 vote
0 answers
13 views

Given a real matrix $M$, find the closest cross-correlation matrix

Say we have an arbitrary real matrix $M$. I am wondering if we can find a cross-correlation matrix that has the least square distance to $M$. In other words, $$\min_{A,B} \|M-A^TB \|^2_F $$ s.t. $$...
  • 271
1 vote
1 answer
21 views

Derivative of Trace of matrix product

I am trying to compute the gradient with respect to a vector $\mathrm x \in \mathbb{R}^d$ of a complicated expression involving the trace of matrix product. The expression is the following: $$ F(\...
0 votes
2 answers
24 views

Derivative of quadratic form of vector-valued function

This seems like a trivial question but I am currently stuck and cannot see what I am doing wrong. So let us consider a function $f(x) : \mathbb{R}^d \rightarrow \mathbb{R}^d$. I want to compute the ...
3 votes
0 answers
44 views

Relation between generalization bounds of Kernel Ridge Regression and largest eigenvalue of the kernel Gram matrix

Consider a positive-definite, symmetric function $k(x_1, x_2)$ which is used, given the dataset $\{(x_i, y_i)\}_{i=1}^m$, to construct the Gram matrix $K = [k(x_i, x_j)]_{i,j \in 1, ..., m}$. What is ...
  • 31
0 votes
0 answers
14 views

How one actually explores data with linear algebra when matrices are just transformations?

I am a beginner in ML and I don't quite understand how we can use linear algebra to extract information from our data matrix. For me matrices only have meaning as a transformation between vector ...
  • 61
3 votes
3 answers
130 views

Hows does coefficient $b_1$ change when estimating $b_1 x_1+b_2 x_2+b_3 x_3$ instead of $b_1 x_1+b_2 x_2$

My question is related to [1], [2] and [3]. Assume we estimate a multiple regression, $$ y = a + b_1x_1 + b_2x_2 + u $$ and are mainly interested in the value of $\hat{b}_1$ (lets denote this specific ...
0 votes
0 answers
15 views

How to get Fisher information matrix from Likelihood?

Since det $R(k) = (1 + \sum_i S/N_i) det N(k)$, only the ex- ponential part of the density function will depend on the delays. Let the signal delay vector D be defined as The likelihood function for $...
  • 51
1 vote
0 answers
21 views

Overfitting with Non Negative Least Squares

I'm trying to reconstruct a function, $A(x)$ from the results of some detectors. Essentially, I have a set of $n$ points which are $ V_{i} = \int_{-\infty}^{\infty} A(x) e^{-(x - v_{i})^{2}} dx $ ...
0 votes
0 answers
16 views

Calculating decision boundary for support Vector machine by hand for a 3 dim dataset

I have previously tried to calculate the decision boundary for a 2d dataset, because through visual inspection the support vectors in the 2d graphs are pretty obvious. I have the following dataset, ...
1 vote
1 answer
25 views

What is the term for summing all of the elements of a vector to produce a scalar?

Is there a specific term for turning a vector into a scalar by summing all of the elements of the vector? I am trying to describe a part of a model that requires this.
0 votes
0 answers
31 views

Numerically solving a sparse matrix equation

I want to find the $X$ that solves the matrix equation $$ AX = B $$ with $A$ and $B$ known - $
A$ and $X$ are rectangular, $A$ is $n \times m $ and $X$ is an $m \times n$, with $m > n$. (...
0 votes
0 answers
39 views

why estimator of covariance matrix will be very bad if samples smaller than random variables

I do not have strong math background but I am currently working on a project that requires me to use a covariance matrix. and it is my first time touching on this topic, I am reading a note, which ...
  • 1
19 votes
2 answers
4k views

Why is XOR not linearly separable?

Let the function $XOR:\{0,1\} \times \{0,1\} \to \{0,1\}$ be the function defined by $$\begin{align} XOR(0,0) &= 0, \\[6pt] XOR(0,1) &= 1, \\[6pt] XOR(1,0) &= 1, \\[6pt] XOR(1,1) &= 0. ...
  • 293
0 votes
0 answers
20 views

Under what condition the left stochastic matrix P(X|Y) is an invertible matrix?

Say we have two random variables $X$, $Y$. They are discrete variables (or discretization of continuous variables), both with $k$ categories. Define the left stochastic matrix as $P(X|Y)_{ij}:=p(x_i|...
4 votes
3 answers
129 views

Different OLS regression procedures that lead to the same coefficients

I've rewritten this question, because my phrasing and notation was confusing. We're assuming OLS regression throughout this post. If we have the data $\mathbf{y} \in \mathbb{R}^N$, $\mathbf{X}\in \...
1 vote
0 answers
15 views

What matrix implements forward differences in regression

I have an equation Yit = XitB The question asks to consider a transformation of data y and X whereby forward differences are calculated so Yit becomes Yit - 1/(T-t)Σ s=1 yi(t+s) so basically it's self ...
1 vote
0 answers
36 views

Generate a random covariance matrix with specified eigenspectra and diagonal elements and first off-diagonal?

I want to generate a random covariance matrix ($c \in \mathcal{R}^{n \times n} $) whose eigenspectra, i.e., $n$ eigenvalues $e_0 \in \mathcal{R}^{n\times 1}$ and diagonal elements $c_{ii} \,\, i=1 \,\,...
1 vote
1 answer
28 views

Why do we use sample mean in the total sum of squares (SST) calculation?

I am trying to understand why the sample mean is used in the Total Sum of Squares or Total Deviation calculation. In other words, what is so important or significant about the sample mean? A lot of ...
  • 13
0 votes
0 answers
12 views

How to have a constant error[b-a] for a range of values [1d array] in decending order that are not linear but are semi-linear?

I am trying to generate n numbers in decending order (for the purpose of decreasing order of weights) such that their sum should be 100 and more importantly the difference between anytwo numbers ...
  • 1
1 vote
0 answers
65 views

Optimal truncation in SVD

I am working with SVD on a matrix $$Y_{m,n} = T_{m,m} \Sigma D^T_{n,n} $$ where $T$ and $D$ describe the row and the column entities of Y, respectively. The truncated SVD takes the first $r$ ...
0 votes
0 answers
25 views

What is the meaning of the inner product between two regression variables?

I have been analyzing the effect of design matrix columns on the contour line of the least squares regression. These contours obviously are ellipses when only two columns $\phi_1$ and $\phi_2$ are ...
  • 11
0 votes
0 answers
40 views

Maximizing a unique trace quadratic form

I am dealing with an unsupervised problem where I have ended up with the following maximization problem: $\max_{C\in \mathbb{R}^{p\times n}}\sum_{i=0}^{m} tr(CA^ixx^\top A^{i^\top}C^\top) \\ \mathrm{...
1 vote
1 answer
71 views

Matrix dimensions in Linear Algebra vs Time series Analysis

I am confused or may misunderstand the dimensions of a Matrix when I was reading about time series analysis. From what I understand in linear Algebra, if we have a Matrix $A \in \mathbf{R}^{m*n}$, ...
  • 13
4 votes
1 answer
58 views

Re-using computations in several least squares problems

I have $K$ least squares problems of the form $Y_k = X_k\beta_k$ for $k = 1, \dots, n$. If the matrix $X_k$ is the same for each index $k$, we can rewrite the problem as $Y_k = X \beta_k$. How can I ...
's user avatar
0 votes
1 answer
28 views

How to understand the relations in matrix multiplications in deep learning?

In the scaled dot product attention they multiply a "softmaxed" matrix (which has shape (sequence_length, sequence_length) I think?) to the V matrix as shown What does the second purple ...
1 vote
0 answers
40 views

Recover linear transformation of Covariance Matrix

Suppose that I can observe $\Sigma_1$ (r x r), and $\Sigma_2$ (r x r)= $A \Sigma_1 A'$, for some arbitrary (r x r) rotation matrix $A$. Is it possible to recover $A$ in any meaningful way?
  • 11
1 vote
0 answers
10 views

duel regression with orthogonal constraints on coefficients [closed]

I'm trying to solve a problem where I need to find two deming style regression models onto two different data sets of equal dimension who's coefficients(A,B) satisfy the following criteria with ...
  • 11
1 vote
0 answers
64 views

Positive semi definite matrix with negative eigenvalues?

From what I know, for any square real matrix A, a matrix generated with the following should be a positive semidefinite (PSD) matrix: ...
0 votes
0 answers
75 views

How to measure whether a set of vectors are more similar to each other than a set of randomly selected vectors would be?

I have ~15 000 word embedding vectors of length 256, that can be categorized into several groups (sizes 2 to 1500) via heuristic considerations. My aim is to measure whether the embedding vectors ...
0 votes
1 answer
14 views

Product of elements of a vector which has very large values and very low values ordered in decreasing order (MATLAB)

I have to compute the product of the elements in a vector V. The elements of my vector are in decreasing order and go from very large numbers (eg 5e^5) to very small numbers (e.g 1.8e^-8). I am using ...
  • 137
0 votes
0 answers
36 views

Dot product and linear combination of Basis

I am currently going through the SVD intuition provided here. In the section "From intuition to definition", It says that, First, note that any vector $\textbf{x}$ can be described using ...
0 votes
0 answers
39 views

how do I eliminate redundant columns in matrix of nominal data?

I have data like this: c1 c2 c3 c4 c5 r1 a a b c d r2 NA a b c a r3 NA b b b c r4 a b c b c Here, each row is a data ...
  • 101
0 votes
0 answers
65 views

Statistical significance and angular comparison (similarity test) between two vectors, MCMC and resampling approaches

I want to compare statistically if two vectors of PC1 loadings differ in terms of their direction which can be measured by the angle between them (e.g. this CV thread). I would like if someone can ...
  • 377
0 votes
0 answers
62 views

what is the difference between factor analysis and SVD? factanal() vs svd()

I am doing factor analysis. Some sources tell me that I should use factanal() to do (exploratory) factor analysis; my goal is to find common sources of latent ...
  • 93
0 votes
0 answers
29 views

Closed form solution for a linear problem

Consider the following problem: $W = \text{argmin} \, || SWE^T - P ||_2$ , where $S \in R^{m\times k}, W \in R^{k\times l}, E\in R^{n\times l}, P\in R^{m\times n}$. Matrices S, E, P are known and I'm ...
5 votes
2 answers
594 views

Introduction to Statistical Learning Eq. 4.32

Can someone please explain how the third line becomes the fourth line?
  • 93
0 votes
0 answers
27 views

how to find the best linear unbiased estimator for scalar product [duplicate]

if we have known the E[Y]=τ, How could we prove that the best linear unbiased estimator of the scalar x.τ is x.Y
  • 1
3 votes
1 answer
66 views

How to express the sample variance as a homogeneous quadratic form when the population variance is known?

Given $\widetilde{S}^{2}=\frac{\sum_{i=1}^{n}{(X_{i}-\mu)^2}}{n}$ where $X=(X_{1},X_{2},...,X_{n})^T$. I was trying to find the $n\times n$ matrix $A$ for this quadratic form; that is, $$\widetilde{S}^...
  • 31

1
2 3 4 5
13