Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
0
votes
1answer
17 views

Solving system of linear equations for the parameters generated from linear regression

I have divided my whole dataset into four subsets and did fit a linear regression model to each subset. I got three different parameters and a constant for each model. I would like to use the ...
3
votes
1answer
95 views

If I have only $(\pmb{X}'\pmb{X})^{-1}$, how can I find $\pmb{X}$ (the design matrix)?

My teacher gave me a problem, but he only give me the $(\pmb{X}'\pmb{X})^{-1}$ matrix. If I have only $(\pmb{X}'\pmb{X})^{-1}$, how can I find $\pmb{X}$ (the design matrix)? I think this is an ...
1
vote
1answer
74 views

Proof $E[Z'TZ]^2=\operatorname{tr}^2(T)+\operatorname{tr}(T^2)$ [duplicate]

How to prove second moment of a quadratic form where $Z$ has normal distribution with mean zero and covariance matrix identical?
1
vote
0answers
19 views

Difference between two dimensions sampled from Dirichlet distribution

Say I'm doing Bayesian inference on a Dirichlet-Multinomial model: $$ x \in [1,2,3]; \\ x \sim Multinomial(p_1, p_2, p_3); \\ p_1, p_2, p_3 \sim Dirichlet(\alpha_1, \alpha_2, \alpha_3); \\ \alpha_n =...
2
votes
0answers
35 views

How to characterize the effect of $(\textrm{Diag}(\Sigma^{-1}))^{-1}$ badly approximating $\textrm{Diag}(\Sigma)$

I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's. If I were to try to approximate ...
0
votes
1answer
21 views

Gradient descent derivation in Eigenspace [duplicate]

I am trying to decode article on https://distill.pub/2017/momentum/ I was able to follow everything until the part with a change of basis x$^k=Q^T(w^k−w^⋆)$ to eigenspace... I conceptually understand ...
0
votes
0answers
28 views

Variables estimation with Cholesky decomposition

I have the covariance matrix between log-returns of n variables. I suppose the distribution of the log-returns is normal for all the variables with average=0 but standard deviation in general $\neq$ 1....
1
vote
0answers
23 views

Understanding PCA from a linear transformation perspective

I've came across several great questions and answers here regarding PCA, but I would like to have a look at it from a linear transformation perspective. Let's say I have a (demeaned) data matrix $X$ ...
0
votes
0answers
22 views

The least squares estimator for beta in matrix notation in connection to normal equations and orthogonal projection in linear algebra [duplicate]

Somebody posted this link here about DERIVATION OF THE LEAST SQUARES ESTIMATOR FOR BETA IN MATRIX NOTATION-(https://economictheoryblog.com/2015/02/19/ols_estimator/). My question is 'How does THE ...
0
votes
0answers
19 views

Is the prediction with and without mean normalisation different in Collaborative Filtering?

In case of Collaborative Filtering: Given an output matrix I wish to learn parameters $\Theta$ (Parameter Vector) and X (Feature Vector). Now if I mean normalise the output matrix the values of $\...
2
votes
1answer
32 views

Question about the gradient of weight normalization

In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$ \mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v $$ Then they ...
0
votes
0answers
39 views

Alternating Least Squares for Baseline Predictor

I am trying to figure out how ALS works when minimizing the following formula: $\\ \\$ $\text{min}_{\lbrace b_u,b_i \rbrace} \sum_{(u,i)\in \mathcal{K}} (r_{ui} - \bar{r} - b_u - b_i )^2 + \lambda_{...
2
votes
1answer
32 views

Struggling on second to last part deriving linear discriminant function

From this post here I am struggling with the matrix multiplication to get from: $\log \pi _{k} - \frac{1}{2}(x-\mu _k)^T{\sum }^{-1}(x-\mu _k)$ to $\log \pi _{k} - \frac{1}{2}[x^{T}{\sum }^{-1}x +\...
0
votes
0answers
29 views

PCA whitened data

Let $X$ be a feature matrix of size $m\times d$. I understand that the standard PCA whitening process follows three steps. (Centerization) ${X} \to \hat{X}:=({X} - \mu)$, where $\mu$ is a matrix of ...
3
votes
1answer
157 views

Chebyshev inequality in terms of RMS

I'm self studying the book Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares In page 48, the author write: "It says,for example, that no more than 1/25 = 4% of the entries ...
0
votes
0answers
26 views

what is a vector perpendicular to a plane of vectors

I have 3 or 4 vectors connected that forms a plane. How can I find the vector that is perpendicular to this plane? it can be a unit vector as long as it preserves this direction. each vector is on ...
0
votes
0answers
7 views

Find the fitting lines in a data with multiple general distribution axis

I have the following data which the breaking point can be varying. How I can find 2 fitting lines? Obviously I can't use minimum square error or RANSAC on raw data, because the work on one major axis.
3
votes
0answers
24 views

Are there extant deep learning analogs to random coefficient (aka mixed) models?

Random coef models, applied to longitudinal data, capture response heterogeneity by cross-sectional unit. I've got a longitudinal prediction problem, in which I know that some "features" (or ...
0
votes
1answer
40 views

how this transform 2d to 3d is work in mathmatically?

this is the 2d->3d projection for svm. they used the kernel trick to change the dimension of the vector for easier classification. I want to understand the detail math behind this projectin which ...
1
vote
2answers
170 views

Proving efficiency of OLS over GLS

I'm trying to prove the efficiency of OLS over GLS when the covariance matrix of the error $\varepsilon$ is mistakenly assumed to be $\sigma^2\Sigma$ instead of $\sigma^2 I$. After deriving the ...
0
votes
0answers
9 views

SEM resources from a linear model prospective

I'm starting a new project at work that requires theory and application of structural equation models, but my background is quite low in this area. I have a very good background in regression, linear ...
0
votes
1answer
30 views

Matrix Orthogonal to Vector: why take transpose?

In econometrics, we often have n observations (in a column vector $y$) which we want to explain with k$<$n regressors (the observations are in an nxk matrix $X$). In this case we use least squares ...
2
votes
0answers
107 views

Does anyone know the rank of the Netflix Prize dataset?

I'm looking into the Netflix Prize at the moment. We model the dataset as an $n \times m$ matrix, where $n$ is the number of users and $m$ is the number of movies. Does anyone know the rank of the ...
1
vote
0answers
74 views

Distribution of dot products of two random independent unit vectors in $D$ dimensions

Duplicate of the stats stack exchange question here; however, I need some help with some of the steps in the accepted answer. A uniform distribution on the unit sphere $\mathbb{S}^{D-1}$ is ...
0
votes
2answers
40 views

Is there a mistake in the expression of this variance?

I'm busy reading through an econometrics textbook (page 147), and I don't understand the step $$\mathrm {Var}\left(n^{\frac 12}\left(\hat\beta - \beta\right)\right) = \boldsymbol{A^{-1}}\sigma^2\...
0
votes
1answer
38 views

Want to make sense of array dimensions in logistic regression algorithms

I am trying to implement a simple logistic regression algorithm from scratch in python (for learning purposes). Every article I've seen online so far presents the following expression for $z$ (...
4
votes
2answers
640 views

Inverse of the covariance matrix of a multivariate normal distribution

Is the covariance matrix of a multivariate normal distribution always invertible?
4
votes
1answer
71 views

Incremental solution for matrix inverse using Shermann-Morrison in $O(n^2)$

I have been reading a presentation on Value Function Approximation by David Silver (Introduction to Reinforcement Learning Course). On page 43 he finds a solution for linear least squares for an ...
0
votes
1answer
45 views

What is matrix norm in (Collins, 2001)

I am studying "A generalization of PCA to the exponential family" (Collins et al., 2001) and I don't understand some notations. What is the meaning of the matrix squared norm on page 6 ? Is it a ...
0
votes
0answers
45 views

Write Lack-of-fit Sum of Squares in Quadratic Form

Let \begin{equation} SSLF = \sum_{i=1}^{m}n_{i}(\bar{y_{i}} - \hat{y_{i}})^{2} \end{equation} then \begin{equation} \sum_{i=1}^{m}n_{i}(\bar{y_{i}} - \hat{y_{i}})^{2} = n(\bar{\overrightarrow{y}} -...
3
votes
1answer
274 views

Mean and Variance of SSE

First, let \begin{equation} SSE = \overrightarrow{y}'(I - H) \overrightarrow{y} \end{equation} where \begin{equation} \overrightarrow{y} \sim MN(\textbf{X} \overrightarrow{\beta}, \sigma^{2}I) \\ H ...
0
votes
0answers
10 views

Is sketching a method for dimensionality reduction and its relation to random projection

I want to know if sketching can be categorized as a method of dimensionality reduction and more specifically feature extraction. Also, i want to understand if its related to random projection.
0
votes
0answers
21 views

Cosine similarity matrix of linearly transformed inputs

Given a matrix $\mathbf{C}$ which contains pairwise cosine similarities between rows of a matrix $\mathbf{A}$, linearly transformed by matrix $\mathbf{U}$: $$ \mathbf{C} = K(\mathbf{UA}, \mathbf{UA}) $...
2
votes
1answer
65 views

Reference point in projection axis of SVD (singular value decomposition)

I am watching a YouTube video on SVD, and attempting to recreate some of its examples to better understand the internal machinery of the algorithm. In one of the slides, the instructor mentions that ...
1
vote
0answers
43 views

Higher moments of linear regression residuals?

I previously asked this on Math StackExchange, with no success, but this post will add to that with some simulations. Background In the following linear regression with i.i.d $\epsilon_i$ $(i = 1, \...
0
votes
0answers
28 views

Solving for variable in a multivariate adaptive regression spline model with logistic link function

I am working on solving for a variable within a MARS model where the probability is set at .5 and so far I have the following equation: log(0.5/0.5) = -1.31 + -2.81(var1 - -0.81) + 4.47(.04 – var2) + ...
4
votes
2answers
74 views

Steps of Matrix Multiplication

It may seem kind of silly, but can anyone please show me the intermediate steps implied by the second equality in this derivation? $$e^\prime e = \left(y - Xb\right)^\prime\left(y - Xb\right) = y^\...
3
votes
1answer
103 views

The form of the Log-Likelihood Function in Mixed Linear Models

Let us assume the following mixed effects model: $y = X\beta+Zu+e$ where $y$ is a vector of n observable random variables, $\beta$ is a vector of $p$ fixed effects, $X$ and $Z$ are known matrices, ...
1
vote
1answer
28 views

Given $m$ $n$-dimensional vectors, how to create a vector perpendicular to all of them?

Given $m$ vectors, $x_1$, $x_2$, ... $x_m$ with all $x_i \,\, \epsilon \,\, \mathcal{R}^n$, $i=1,2... m$ and $m < n$. How to sample a vector $x_{m+1}$ perpendicular to all the vectors $x_1$, $...
0
votes
0answers
30 views

Additional Property of Singular Value Decomposition

I am new to SVD so forgive me if the question is trivial. Following is my question. If I have two sets of linear equations, Y1 = T1.X Y2 = T2.X where T1 and T2 are mxn rectangular matrices. Now let'...
3
votes
2answers
123 views

What is the Joint Density Function of a Three-Level Mixed-Effects Model?

This is a follow-up question to a question I posted earlier. Obviously, maximum-likelihood estimation of mixed-effects models requires the joint density function. Let us assume a two-level mixed ...
0
votes
0answers
39 views

Idea behind change of basis and how it relates to projecting your points onto principal components

I would like to clarify if my understanding is correct. In the traditional X-Y coordinate system, our choice of basis vectors are $\vec{i} = (1, 0)$ and $\vec{j} = (0, 1)$ and when you I have a point $...
1
vote
1answer
57 views

Is this dataset lineary seperable? How can I find it out using (linear) algebra?

I have this dataset: I want to know if it is linearly separable (fully separable). I want to use this rule, but I'm not sure if it's correct: Make $X'$ - matrix with d+1 column of all 1's. Then ...
7
votes
1answer
310 views

Mixed Models: How to derive Henderson's mixed-model equations?

In the context of best linear unbiased predictors (BLUP), Henderson specified the mixed-model equations (see Henderson (1950): Estimation of Genetic Parameters. Annals of Mathematical Statistics, 21, ...
1
vote
1answer
105 views

Problem with Softmax decision boundary

While reading this paper: sphere face on page 2, it explains that original softmax boundary is given by: $$(W_1 −W_2)x+b_1 −b_2 = 0$$ While trying to obtain the boundary on a toy generated 2D dataset ...
7
votes
2answers
756 views

What is the problem with $p > n$?

I know that this is the solving system of linear equation problem. But my question is why it is a problem the number of observation is lower than the number of predictors how can that thing happen? ...
1
vote
0answers
55 views

Multilevel Modeling: Minimization Problem when A = B + TCT'

I currently study multilevel model using Leeuw & Meijer (2008) Handbook of multilevel analysis. On page 65, they state the following theorem: If $A = B + TCT'$ with $A$ and $B$ positive definite, ...
-1
votes
1answer
77 views

How to cluster a (directional) dissimilarity matrix with both positive and negative values?

I may be thinking of this incorrectly but what would be the best way to cluster a dissimilarity measure that has direction? For example, if someone had condition A ...
0
votes
0answers
42 views

Is there a problem with the “low-rank matrix approximation”?

I know that "rank" is the number or independent rows in the matrix and I know that the resulting matrix of the "low-rank matrix approximation" algorithm has lower rank than the original matrix. 1- But ...
-1
votes
1answer
40 views

PCA with zero and high correlation in data

How would the eigen values look like when we apply PCA to a dataset with zero correlation between variables and when there is very high correlation between variables