Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Calculate the inverse of the sum of an identity and a Kronecker product

Let $Q=\alpha\mathbb{I}+(S\otimes S)^T(S\otimes S)=\alpha\mathbb{I}+S^TS\otimes S^TS$, where $\mathbb{I}$ is $n^2\times n^2$ matrix, $S$ is an $m\times n$ binary matrix and $\otimes$ is a Kronecker ...
0
votes
0answers
13 views

prove for Eckart-young in Frobenius norm

On page 74, linear algebra and learning from data. P74 the prove for Eckart-young in the frobenius norm. I couldnot understand why G = 0 in the proof, anybody can help me? Thank you!
0
votes
1answer
12 views

(SVMs) Do the specific higher dimensional mappings of attributes not matter when calculating a kernel?

From what I know, one of the strategies employed by an SVM is to increase dimensionality of your data until they are linearly separable. (I guess there's some mathematical proof that your data will ...
0
votes
0answers
19 views

Assessing the size of a cone by the singular values of $M$

Suppose I work with vectors from a high dimensional space with $100<N<1000$, e.g. word-embeddings. Say I have, already selected $R$ vectors, with $R\simeq10$, which form a matrix $M \in \mathbb{...
0
votes
0answers
32 views

Two layer network working in PyTorch, but cannot reproduce using pure linear algebra

Rewording the original question to comply with the CV topics. I am trying to build a simple 2 layer network, and was going through the backprop mathematics of it, when I got stuck. I am defining my ...
0
votes
0answers
35 views

Why are the eigenvalues of $X'X$ equal to that of $XX'$ when $X$ is a design matrix? [duplicate]

The title says it all. If $X$ is a design matrix (columns containing variables, rows containing observations), I have observed that eigs($X'X$)=eigs($XX'$). I actually found this by accident when I ...
2
votes
0answers
26 views

Geometric interpretation of Cholesky Decomposition

I understand that a square matrix, say $A$, can be thought of as a linear transformation within the same space. I could be as simple as basis change or some other transformation. In this way of ...
0
votes
1answer
18 views

Distance between 2 hyperplanes in SVM formulation

During the SVM formulation, the 2 hyperplanes is given by the equations: wᵀx + b = 1 ---------(1) wᵀx + b = -1 ---------(2) Now, the margin between these 2 hyperplanes is given by: 2/||w|| ...
1
vote
1answer
34 views

Why is the following choice of factor loadings optimal in two-state MLE for factor analysis?

Suppose we have $n$, $p$-dimensional, samples $\overrightarrow{X_i} \sim \mathcal{N}(\mu, \Psi+\mathbf{w^Tw})$. $\Psi$ is a diagonal matrix of specific variances, while $\mathbf{w^Tw}$ composes the ...
0
votes
1answer
41 views

Matrix representation of PCA

In both Wikipedia and this medium post, I see the succinct principle components decomposition of X represented as $$T=XW$$ However, it seems to me that it should be $T = WX$ instead, if according ...
1
vote
1answer
35 views

Determinants of the sum of two matrices and their eigenvalues

I know some basic property of determinant. I read an article and see this formula: \begin{equation} |(\delta-p-1)D+S|=\bigg(\prod_{i=1}^{p}\lambda_{i}(D)\bigg) \bigg(\prod_{i=1}^{p}(\delta-p-1)\...
1
vote
1answer
54 views

How to merge two different sum of squared errors?

Given two datasets $ U \in R$ and $P \in R$. $n_1$ and $n_2$ are respectively the number of points contained in U and P. The sum of squared errors of U and P are as follow: $$ SSE_U=\sum_{i=1}^{n_1}||...
0
votes
0answers
26 views

Why does Alternating Least Squares (ALS) give us good results for missing values?

I was reading about the alternating least squares algorithm and could follow the math but somehow it didn't click for me. We start with random values for $U$ and $V$ and run the algorithm until we ...
0
votes
0answers
25 views

eigen value decomposition of co-variance a series generated by factor model

Let's assume $N\times T$ series $Y_t$ is generated by the following equation. $$ Y_t = \begin{bmatrix}A_x & A_m\end{bmatrix}\begin{bmatrix}x_t \\ m_t \end{bmatrix}$$ Where $A_x$ and $A_m$ are $N\...
2
votes
1answer
29 views

Random matrix theory and research - a lot like doing linear algebra?

I've been searching for a subfield of research to get into and wonder whether random matrix theory could suit me well; it seems like it does, because the stuff I read, and the seminars that I watch ...
3
votes
1answer
102 views

If I have only $(\pmb{X}'\pmb{X})^{-1}$, how can I find $\pmb{X}$ (the design matrix)?

My teacher gave me a problem, but he only give me the $(\pmb{X}'\pmb{X})^{-1}$ matrix. If I have only $(\pmb{X}'\pmb{X})^{-1}$, how can I find $\pmb{X}$ (the design matrix)? I think this is an ...
1
vote
1answer
17 views

A derivation regarding kernel regression for the support vector machine

THis is from the Elements of Statistical Learning book page 437 in the section of support vector machine. Can anyone give me some hint for the missing derivation steps for why 12.49 is true (as seen ...
0
votes
1answer
21 views

Expansion of inner product for polynomial kernel for SVMs

On page 424 in "The Elements of Statistical Learning" by Hastie et al (2013) (https://web.stanford.edu/~hastie/Papers/ESLII.pdf), we see the following expansion of a polynomial kernel with degree 2: ...
1
vote
1answer
81 views

Proof $E[Z'TZ]^2=\operatorname{tr}^2(T)+\operatorname{tr}(T^2)$ [duplicate]

How to prove second moment of a quadratic form where $Z$ has normal distribution with mean zero and covariance matrix identical?
1
vote
0answers
22 views

Difference between two dimensions sampled from Dirichlet distribution

Say I'm doing Bayesian inference on a Dirichlet-Multinomial model: $$ x \in [1,2,3]; \\ x \sim Multinomial(p_1, p_2, p_3); \\ p_1, p_2, p_3 \sim Dirichlet(\alpha_1, \alpha_2, \alpha_3); \\ \alpha_n =...
2
votes
0answers
38 views

How to characterize the effect of $(\textrm{Diag}(\Sigma^{-1}))^{-1}$ badly approximating $\textrm{Diag}(\Sigma)$

I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's. If I were to try to approximate ...
0
votes
1answer
27 views

Gradient descent derivation in Eigenspace [duplicate]

I am trying to decode article on https://distill.pub/2017/momentum/ I was able to follow everything until the part with a change of basis x$^k=Q^T(w^k−w^⋆)$ to eigenspace... I conceptually understand ...
0
votes
0answers
34 views

Variables estimation with Cholesky decomposition

I have the covariance matrix between log-returns of n variables. I suppose the distribution of the log-returns is normal for all the variables with average=0 but standard deviation in general $\neq$ 1....
1
vote
0answers
25 views

Understanding PCA from a linear transformation perspective

I've came across several great questions and answers here regarding PCA, but I would like to have a look at it from a linear transformation perspective. Let's say I have a (demeaned) data matrix $X$ ...
0
votes
0answers
25 views

The least squares estimator for beta in matrix notation in connection to normal equations and orthogonal projection in linear algebra [duplicate]

Somebody posted this link here about DERIVATION OF THE LEAST SQUARES ESTIMATOR FOR BETA IN MATRIX NOTATION-(https://economictheoryblog.com/2015/02/19/ols_estimator/). My question is 'How does THE ...
0
votes
0answers
21 views

Is the prediction with and without mean normalisation different in Collaborative Filtering?

In case of Collaborative Filtering: Given an output matrix I wish to learn parameters $\Theta$ (Parameter Vector) and X (Feature Vector). Now if I mean normalise the output matrix the values of $\...
2
votes
1answer
47 views

Question about the gradient of weight normalization

In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$ \mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v $$ Then they ...
0
votes
0answers
42 views

Alternating Least Squares for Baseline Predictor

I am trying to figure out how ALS works when minimizing the following formula: $\\ \\$ $\text{min}_{\lbrace b_u,b_i \rbrace} \sum_{(u,i)\in \mathcal{K}} (r_{ui} - \bar{r} - b_u - b_i )^2 + \lambda_{...
2
votes
1answer
33 views

Struggling on second to last part deriving linear discriminant function

From this post here I am struggling with the matrix multiplication to get from: $\log \pi _{k} - \frac{1}{2}(x-\mu _k)^T{\sum }^{-1}(x-\mu _k)$ to $\log \pi _{k} - \frac{1}{2}[x^{T}{\sum }^{-1}x +\...
0
votes
0answers
35 views

PCA whitened data

Let $X$ be a feature matrix of size $m\times d$. I understand that the standard PCA whitening process follows three steps. (Centerization) ${X} \to \hat{X}:=({X} - \mu)$, where $\mu$ is a matrix of ...
4
votes
1answer
190 views

Chebyshev inequality in terms of RMS

I'm self studying the book Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares In page 48, the author write: "It says,for example, that no more than 1/25 = 4% of the entries ...
0
votes
0answers
27 views

what is a vector perpendicular to a plane of vectors

I have 3 or 4 vectors connected that forms a plane. How can I find the vector that is perpendicular to this plane? it can be a unit vector as long as it preserves this direction. each vector is on ...
0
votes
0answers
8 views

Find the fitting lines in a data with multiple general distribution axis

I have the following data which the breaking point can be varying. How I can find 2 fitting lines? Obviously I can't use minimum square error or RANSAC on raw data, because the work on one major axis.
3
votes
0answers
24 views

Are there extant deep learning analogs to random coefficient (aka mixed) models?

Random coef models, applied to longitudinal data, capture response heterogeneity by cross-sectional unit. I've got a longitudinal prediction problem, in which I know that some "features" (or ...
0
votes
1answer
67 views

how this transform 2d to 3d is work in mathmatically?

this is the 2d->3d projection for svm. they used the kernel trick to change the dimension of the vector for easier classification. I want to understand the detail math behind this projectin which ...
1
vote
2answers
229 views

Proving efficiency of OLS over GLS

I'm trying to prove the efficiency of OLS over GLS when the covariance matrix of the error $\varepsilon$ is mistakenly assumed to be $\sigma^2\Sigma$ instead of $\sigma^2 I$. After deriving the ...
0
votes
0answers
9 views

SEM resources from a linear model prospective

I'm starting a new project at work that requires theory and application of structural equation models, but my background is quite low in this area. I have a very good background in regression, linear ...
0
votes
1answer
45 views

Matrix Orthogonal to Vector: why take transpose?

In econometrics, we often have n observations (in a column vector $y$) which we want to explain with k$<$n regressors (the observations are in an nxk matrix $X$). In this case we use least squares ...
2
votes
0answers
113 views

Does anyone know the rank of the Netflix Prize dataset?

I'm looking into the Netflix Prize at the moment. We model the dataset as an $n \times m$ matrix, where $n$ is the number of users and $m$ is the number of movies. Does anyone know the rank of the ...
1
vote
0answers
90 views

Distribution of dot products of two random independent unit vectors in $D$ dimensions

Duplicate of the stats stack exchange question here; however, I need some help with some of the steps in the accepted answer. A uniform distribution on the unit sphere $\mathbb{S}^{D-1}$ is ...
0
votes
2answers
41 views

Is there a mistake in the expression of this variance?

I'm busy reading through an econometrics textbook (page 147), and I don't understand the step $$\mathrm {Var}\left(n^{\frac 12}\left(\hat\beta - \beta\right)\right) = \boldsymbol{A^{-1}}\sigma^2\...
0
votes
1answer
40 views

Want to make sense of array dimensions in logistic regression algorithms

I am trying to implement a simple logistic regression algorithm from scratch in python (for learning purposes). Every article I've seen online so far presents the following expression for $z$ (...
4
votes
2answers
743 views

Inverse of the covariance matrix of a multivariate normal distribution

Is the covariance matrix of a multivariate normal distribution always invertible?
4
votes
1answer
100 views

Incremental solution for matrix inverse using Shermann-Morrison in $O(n^2)$

I have been reading a presentation on Value Function Approximation by David Silver (Introduction to Reinforcement Learning Course). On page 43 he finds a solution for linear least squares for an ...
0
votes
1answer
45 views

What is matrix norm in (Collins, 2001)

I am studying "A generalization of PCA to the exponential family" (Collins et al., 2001) and I don't understand some notations. What is the meaning of the matrix squared norm on page 6 ? Is it a ...
0
votes
0answers
52 views

Write Lack-of-fit Sum of Squares in Quadratic Form

Let \begin{equation} SSLF = \sum_{i=1}^{m}n_{i}(\bar{y_{i}} - \hat{y_{i}})^{2} \end{equation} then \begin{equation} \sum_{i=1}^{m}n_{i}(\bar{y_{i}} - \hat{y_{i}})^{2} = n(\bar{\overrightarrow{y}} -...
3
votes
1answer
375 views

Mean and Variance of SSE

First, let \begin{equation} SSE = \overrightarrow{y}'(I - H) \overrightarrow{y} \end{equation} where \begin{equation} \overrightarrow{y} \sim MN(\textbf{X} \overrightarrow{\beta}, \sigma^{2}I) \\ H ...
0
votes
0answers
16 views

Is sketching a method for dimensionality reduction and its relation to random projection

I want to know if sketching can be categorized as a method of dimensionality reduction and more specifically feature extraction. Also, i want to understand if its related to random projection.
0
votes
0answers
24 views

Cosine similarity matrix of linearly transformed inputs

Given a matrix $\mathbf{C}$ which contains pairwise cosine similarities between rows of a matrix $\mathbf{A}$, linearly transformed by matrix $\mathbf{U}$: $$ \mathbf{C} = K(\mathbf{UA}, \mathbf{UA}) $...
2
votes
1answer
92 views

Reference point in projection axis of SVD (singular value decomposition)

I am watching a YouTube video on SVD, and attempting to recreate some of its examples to better understand the internal machinery of the algorithm. In one of the slides, the instructor mentions that ...