All Questions
Tagged with dot-product or linear-algebra
694 questions
1
vote
2
answers
63
views
Covariance matrix in terms of $X^TX$
If I have a matrix $X\in \mathbb{R}^{n\times p}$, then I can write the covariance as
$$\text{Cov}(X) = \mathbb{E}[(X-\mu_X)(X-\mu_X)^T]$$
Now, assuming the data is centered, this becomes $\text{Cov}(X)...
- 190
1
vote
1
answer
42
views
IV Rank/Relevance Condition Linear Algebra Intuition
Consider the following econometric model (IV) : $Y_1 = X'\beta + e$, where $Y_1 \in \mathbb{R}$ is some outcome variable of interest, and we have a set of regressors $X = \begin{bmatrix} Z_1 \\ Y_2 \...
- 101
1
vote
0
answers
34
views
Least Absolute Deviations – Geometric Intuition
I've recently been exposed to the geometric intuition regarding Least Squares (OLS) regression:
The vector of the outcome variable $Y$, is not not in the linear span of $X_1, X_2, ..., X_{p-1}$:
The ...
- 31
1
vote
1
answer
40
views
If $n\operatorname{var}( \sum_{ij}M_{ij}v_{i}v_{j}) = (\sum_{i}v_{i}^{2})^{2} - \sum_{i}v_{i}^{4}$ for any $v_i$, what can we say about $M_{ij}$?
Let $M_{ij}$ be a real random matrix, constrained to be symmetric $M_{ij}=M_{ji}$, and with zero diagonal, $M_{ii}=0$.
Suppose we know that, for any real vector $v_i$, the following holds:
$$\...
- 4,552
4
votes
2
answers
109
views
Avoiding tensors when differentiating with respect to weight matrices in backpropagation
Consider a neural network consisting of only a single affine transformation with no non-linearity. Use the following notation:
$\textbf{Inputs}: x \in \mathbb{R}^n$
$\textbf{Weights}: W \in \mathbb{R}...
- 181
4
votes
1
answer
127
views
Closed form solution for bayesian linear regression with 2 responses?
I am thinking about first principles from the point of view of a frequentist moving from regression with 1 response to regression with 2 responses. Reflecting on that I am trying to figure out how to ...
- 1,662
3
votes
0
answers
70
views
How to show this property of the Covariance matrix? [closed]
Hi all, may I ask how the pink highlighted equation is derived? By expanding the matrix, how does the remaining three terms result in n(avg X)(avg X)T? Apologies for the rather basic question, but am ...
- 39
1
vote
0
answers
14
views
How to implement and notate the replication/transformation of a 2D matrix to a 3D tensor and the summation/transformation of a 3D tensor to 2D matrix?
Background:
I have a model with a dimension $T$ representing $time$, a dimension $N$ representing $technologies$ and a dimension $P$ representing $prices$. During calculations in this model, I would ...
0
votes
0
answers
21
views
Correct algebraic notation for contrasts from statistical model
In a pre-post study, we can use a linear mixed model to estimate the treatment effect as the coefficient of the time x treatment interaction, see here (section 19.3):
https://www.middleprofessor.com/...
- 923
1
vote
0
answers
39
views
PDF and moments for a linear transformation of multivariate lognormal
I have a multivariate lognormal variable $Y$ of dimension $d$. $Y=e^X$ where $X\sim N(\mu,\Sigma)$.
The PDF for the lognormal distribution in terms of $\mu$ and $\Sigma$ is:
$$
p(\vec{y}|\mu,\Sigma) = ...
- 171
13
votes
6
answers
2k
views
Expected value of a matrix = matrix of expected value?
I'm wondering about the following statement : the expectation of a matrix equals to the matrix of expectations.
For instance, let $A$ be a matrix of 4 random variables $W,X,Y,Z$, i.e., $A = \begin{...
- 133
1
vote
1
answer
40
views
Derive the posterior multivariate normal distribution
I have a question when I was reading the book Latent Variable Models and Factor Analysis: A Unified Approach by Bartholomew, Knott and Moustaki. Here it is:
Suppose that $\mathbf{x}=(x_1, x_2, ..., ...
- 19
0
votes
1
answer
26
views
Sequential sum of squares with svd
I am studying some methods to determine the coefficients of a linear regression and I am wondering how to find the sequential sum of squares, or the second column of the ANOVA table which shows ...
- 281
0
votes
0
answers
14
views
Uniqueness of idiosyncratic error in factor model
I have been learning about the uniqueness problem of idiosyncratic error $\sum$, and found two relevant references:
Anderson and Rubin, 1956:
Bekker and ten Berge, 1997:
The result from Bekker and ...
1
vote
0
answers
14
views
in matlab, is a matrix of condition number of 1e20 definitely more ill-conditioned than a matrix of condition number of 1e19？
I am working with some ill-conditioned matrices, trying to find the relationship between the matrix's ill-conditioning and the results. However, I have noticed that the condition numbers of these ...
- 11
9
votes
2
answers
376
views
Sums of exponentials joint probability
If we have that: $\tau_i \overset{\text{independent}}{\sim}
\exp(\lambda_i)$, for $i=1,2,3,...,n$, where $\lambda_i\neq \lambda_j, \forall i\neq j$ then I would like to find a general form for the ...
- 930
0
votes
1
answer
46
views
Pls Help! AR(1) Covariance derivation query [closed]
Can someone pls explain to me why they can do this when deriving the autocovariance of AR(1)
Why can they just add a $\phi$ in front of μ ? Shouldn't it be: $\phi y_{t-1} + \epsilon_t - \mu$
Much ...
- 3
1
vote
0
answers
43
views
how to approximate the eigendecomposition of a correlation matrix when the data have been standardized?
Context
I am working to develop a penalized regression framework that will scale up to analyzing high dimensional data with a certain correlation structure. Let $X$ represent an $n \times p$ matrix of ...
4
votes
1
answer
58
views
Why do OLS libraries fit models using the MP Pseudoinverse of the design matrix?
For the linear model $y = X\beta$ for design matrix $X$, it's well known that the optimal solution is $\hat{\beta} = (X'X)^{-1}X'y$.
Some statistical libraries (such as Python's statsmodels) estimate ...
- 612
0
votes
0
answers
28
views
Constrained Cholesky Decomposition
Suppose that I have an $(n\times 1)$ vector of random variables, $\varepsilon$. However, I know that $k$ linear combinations of $\varepsilon$ are 0. Specifically, I know that for a $(k\times n)$ ...
- 1
5
votes
1
answer
187
views
Rasmussen Equation 5.9
Can any one add the steps showing how Rasmussen (Gaussian Processes for Machine Learning, the MIT Press, 2006) got from line 1 to line 2 of equation 5.9. (pg 114)? It is calculating the gradient of ...
- 331
10
votes
2
answers
308
views
Does the conditional expectation operator have an interpretable decomposition like the projection matrix does in linear algebra?
I'm trying to draw a parallel between the concept of projections in a finite linear space to an infinite linear space.
Here is the set-up, first in the finite dimensional case, and then second in the ...
- 111
3
votes
1
answer
98
views
The Math Behind the Conditional Probability of a Probabilistic PCA
I am trying to understand how to calculate the conditional distribution of probabilistic principal component analysis. This is explained in the book "Pattern Recognition and Machine Learning"...
- 75
1
vote
0
answers
16
views
Positive distance weighting
I have an overdetermined linear system of equations that's solved with least squares. I'd like to weight the equations to penalize a bunch of inputs clumped up together.
Ideally if two (or more) ...
- 11
1
vote
0
answers
53
views
What exactely is "the part of the interaction orthogonal to factors $A$ and $B$" in a two-way ANOVA?
Consider a two-way ANOVA with factors $A$ and $B$ and the interaction $A\times B$.
The author of this answer answer https://stats.stackexchange.com/a/608301/359647 (@svendvn) explains that the Type ...
- 312
0
votes
0
answers
19
views
Error term in SGD with momentum
I am reading the article "How Momentum really works" (https://distill.pub/2017/momentum/), and i am confused in one point:
I am trying to derive the convergence rate for momentum from the ...
- 121
0
votes
0
answers
10
views
Decomposing model volatility with respect to factor contributions
Consider a linear model
$\textbf{y} = \textbf{x}\pmb{\beta} + \pmb{\varepsilon}$
with $\textbf{y}$ a $T \times 1$ vector of random variables, $\pmb{\beta}$ a $K \times 1$ vector and $\textbf{x}$ a $T \...
- 190
6
votes
1
answer
236
views
Why does this matrix form of weighted least squares not match sklearn's weight?
I coded up the answer to this question and it turned out not to match:
https://math.stackexchange.com/questions/1021812/matrix-form-for-weighted-least-squares
The solutions are close, and I'm ...
- 165
0
votes
2
answers
86
views
simple ANN as a set of linear transformations
We cannot classify the points of the XOR problem with a single perceptron in the hidden layer. However, we can achieve this by using two perceptrons in the hidden layer and one for the output layer, ...
- 1
1
vote
1
answer
26
views
Finding a design matrix
I am trying to understand how a design matrix was obtained in this problem below.
Consider the one sample problem:
$Y_i \sim N(\mu, \sigma^2), 1 \le i \le n$. with the $Y_i's$ i.i.d. The MLE is:
$\hat\...
- 345
3
votes
1
answer
98
views
Geometric understanding of linear regression
I am reading up on linear regression from mit 16.850
Here is how the lecture goes:
Given: $Y_{n,1}$ (targets), $X_{n, p}$ (data), $t_{p, 1}$ (the parameters I'm optimizing over), True model: $Y = \...
- 2,693
0
votes
1
answer
35
views
Principal Component Analysis and Relation to the SVD of a matrix [duplicate]
We are learning about Principal Component analysis in our class, and I having trouble understanding how to compute the principal component given a matrix. For example, here is the matrix we were given....
- 345
4
votes
2
answers
153
views
Linear algebra properties of a confusion matrix (eigenvalues, eigenvectors, and determinants)
This answer to a question on Math Stack Exchange got me thinking about a confusion matrix as more than just a rectangular array of numbers. We don’t talk about a confusion matrix as a linear ...
- 67k
0
votes
0
answers
25
views
Proving how scaling of predictor variable in linear regression, affects the fitted coefficient [duplicate]
In linear regression the OLS solution is given by:
$$
\hat{\beta} = (X^TX)^{-1}X^TY
$$
I want to show that if you scale the $i$th predictor variable by a constant, then the corresponding $i$th ...
- 379
3
votes
1
answer
42
views
Converting Adjusted R²
I just examined the $R^2_\text{adj}$ Formula on Wikipedia and found two ways to calculate the adjusted $R^2$.
Firstly as
$$R^2_\text{adj}=1-\frac{\frac{SS_\text{res}}{(n-p-1)}}{\frac{SS_\text{tot}}{(n-...
- 53
0
votes
2
answers
80
views
The Impact of Vector Magnitudes in Recommendation Systems Matrix Factorization Models
I'm currently exploring latent factor models in recommendation systems, specifically focusing on the interaction between vector magnitudes and the angles between these vectors. While it's clear that ...
- 77
3
votes
0
answers
50
views
How much is the data energy loss in PCA?
Recently in a slide in about PCA (Principal Component Analysis) I saw a question: "How much is the data energy loss in PCA?&...
- 131
1
vote
0
answers
38
views
Product of Two t-distribution Formulas
Does the product of two t-distribution formulas with same degrees of freedom simplify?
$T_v(x; \mu_1, \Sigma_1)T_v(x; \mu_2, \Sigma_2) =\ ?...$
In the normal case it simplifies to:
$\mathcal{N}(x; \...
- 331
3
votes
1
answer
147
views
Dual form of the least square solution (ridge rigression)
I was reading this introductory material and on the 5th page, it describes the dual form of the least-square solution (with ridge regression) as $$A(aI + A^\top A)^{-1} = (aI + AA^\top)^{-1}A$$ for a $...
- 125
0
votes
0
answers
27
views
Calculating the Orthogonal Distance to Kernel PCA subspace (with a new data)
I am studying Kernel PCA methods and now I'm trying to calculate orthogonal distances (OD) on the feature space. What I've found is, you can calculate ODs with a kernel trick if you are interested in ...
1
vote
0
answers
21
views
How to compare different clusters of different size, rotation, scale and translation?
Assume that you have a matrix $X$ that contains the data inside the left image. The data inside $X$ is not classified. The matrix $X$ also contains outliers/noise.
On the right, we can se the template ...
- 425
0
votes
1
answer
52
views
What does $(x_i - \xi_k)_+$ mean in this regression spline formula? [duplicate]
I have seen regression models with a continuous predictor fitted as a spline written like this:
What is the meaning of the little "addition symbol" subscript that I have circled in red? Is ...
- 923
5
votes
2
answers
255
views
Covariance matrix square root
Consider a random variable $r_t$ which represents the return of an asset at time t. In the univariate case, we just consider $r_t$ to be the return of a single security at time t. Generally, we assume ...
- 101
6
votes
2
answers
882
views
Is the expectation of a random vector multiplied by its transpose equal to the product of the expectation of the vector and that of the transpose
I'm taking a course in advance statistics and we have to prove whether the following expression is true: $E[zz^T]=E[z]E[z^T]$. I am assuming it is not, since the formula of the covariante matrix is $...
- 91
0
votes
0
answers
26
views
Covariance matrix for data
Assume $n*p$ data matrix $X$, where n is the number of observations and p is the number of features. We are interested in the covariance among features.
I have seen notations where covariance matrix ...
- 61
1
vote
0
answers
23
views
Heteroscedastic Asymptotic Variance Simple Transformation
Let's denote the asymptotic variance under heteroscedasticity as:
$$\hat{\text{Avar}}(\hat{\beta}) = 1/N * \left(\frac{1}{N} \sum_{i}{x_i x_i'}\right)^{-1} \left(\frac{1}{N} \sum_{i} \hat{u}^2_i x_i ...
- 677
3
votes
1
answer
97
views
Spiked tensor decomposition vs canonical polyadic decomposition
What are the similarities and differences between Spiked tensor decomposition and canonical polyadic (CP) decomposition?
My understanding is that CP decomposition aims to find a low-rank approximation ...
- 95
0
votes
0
answers
25
views
How to adjust similarity scores by removing the influence of a common vector?
I have a similarity score function, $s(x,y)$. I know that I have two items that I'm trying to compare the similarity of, but both are based on the same template. How would I remove the template from ...
- 101
2
votes
1
answer
438
views
How to sample efficiently from an inverse Wishart distribution?
I am trying to understand the code from pybasicbayes, which defines the following function to sample from an inverse Wishart:
...
0
votes
0
answers
18
views
Estimate null hypothesis for correlation of linear combinations of variables?
Setting up the problem
Suppose I have a variable $x$ of length $n$ and I have another $p$ variables $y_1, y_2, \dots, y_p$, where $y_i$ is also of length $n$.
Based on the y's, I can make a linear ...
- 43