# Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

519 questions
Filter by
Sorted by
Tagged with
82 views

23 views

### how to normalize data such that an estimated OLS regression vector has pre-specified length (= L_2 norm)

I have the following data: $n$ observations on $d$ variables $X$ and one outcome variable $Y$; i.e. $X$ is a $n \times d$ matrix and $Y$ an $n \times 1$ vector. I consider the following Ordinary Least ...
44 views

### Why can't same-size layers of a neural network be combined or compressed into a single layer?

Probably a dumb question, obviously not seen in practice, but after reviewing linear algebra, I can't pinpoint the misunderstanding in this logic: We can represent input data to the network as a ...
117 views

### Adjoint relationship in Neural ODEs

The Chen et. al paper Neural ODE (https://arxiv.org/pdf/1806.07366.pdf) uses the adjoint method to take derivatives of solutions generated by an ODE solver with respect to neural network parameters θ. ...
158 views

### Human intuition behind SVD in case of recommendation system

This does not answer my question. I struggled very hard to understand the SVD from a linear-algebra point of view. But in some cases I failed to connect the dots. So, I started to see all the ...
30 views

### How vector projection works behind SVD?

I was reading a blog on mathematical intuition behind SVD. Here, author pointed out three information we get after vector decomposition. The directions of projection — the unit vectors (v₁ and v₂) ...
44 views

### Relationship between the SVD and correlation matrices

I'm reading Data Driven Science and Engineering by Kutz and Brunton to understand more about the SVD. Consider $X = U\Sigma V$, $XX^*$, and $X^*X$ where $X \in \mathbb{R}^{m\times n}$ In particular, ...
146 views

### Does training loss go to zero in kernel regression?

Edit Have left the original post in tact, scroll to bottom for updated thinking High Level Problem Statement While studying kernel regression, after playing around with some linear algebra, I appear ...
203 views

86 views

### What is the difference between np.linalg.norm(x-y,axis=1) and np.linalg.norm(x-y)?

I'm creating a K-Medoids algorithm from scratch in Python using numpy, and I'm in the process of using a distance function to determine the cluster center. I want the center to be the point in the ...
28 views

### Regularizing the difference in the norms of two independent weight matrices in a neural network

Say, there are two neural network layers with weights $W_1$ and $W_2$. These two layers are part of a larger network but their inputs are completely independent of each other and their outputs could ...
198 views

### Angle between PCA vector spaces?

I have two datasets of the same shape, one for condition A, the other for condition B. I would like to test if the major axes of variance of condition A are different than those of B. Here is my idea. ...
29 views

I´m new to Multivariable Statistics and just started to learn about Factor Analysis. I do understand that for p=k (there are as many factors as dimensions in the data) L (loadings) is unique. But ...
63 views

### What is this projection matrix doing?

Let’s say we have a $m\times d$ zero mean multivariate Gaussian matrix $X$. Its covariance matrix is $X^{T}X$. Let $V$ be the $d\times d$ matrix of eigenvectors of $X^{T}X$, with the columns sorted in ...
55 views

### Is this statement about linear-activation neural network in the PRML in error?

I found the following statement in "Pattern Recognition and Machine Learning" (C. M. Bishop, 2016) p.229. If the activation functions of all the hidden units in a network are taken to be linear, ...
46 views

### What can you say about spread of data by looking at singular values and clusters?

I have dataset 250X5 and its singular values are [200 50 25.2 2.3 0.35]. Singular values are directly related to variance. Can you say something about the clustering of data and how much is the spread ...
49 views

### Can anyone show how the concept of Identifiability is geometrically/intuitively presented?

The motivation for this question comes from the following: When I was studying statistics for the first time long ago, no one presented the mathematical concepts behind linear regression, like the one ...
101 views

### Prove variance of locally weighted regression increases with degree

I am interested in proving the following fact for locally weighted polynomial regression from The Elements of Statistical Learning by Hastie et. al. It can be shown that $||l(x_0)||$ increases with ...
163 views

### How to do multiple linear regressions with overlapping predictors efficiently

I'm using covariance matrices to do a linear regression. I have the same number of predictors and dependent variables (3000 of each). For each $i$, $X_{i}$ and $Y_{i}$ have a kind of symmetry. I ...
26 views

### Finding $G$ in a solved least squares $d = Gm$?

This may be odd, but say someone solved $d = Gm$ using the least squares method and calculated the parameters vector $m$. I have the vector of data $d$, how could I solve for $G$ to see the matrix ...
28 views

### Why is it necessary to “ignore” a level when applying sum contrasts?

I am confused about how sum contrasts are set up. As I understand, if I have some $K$-leveled factor, I can use sum contrasts to compare each level to the grand mean ($M_G$), effectively testing ...
36 views

### Matrix formula for the correlation matrix, page 72 from Mathematical Tools for Applied Multivariate Analysis

Book information The book is titled: Mathematical Tools for Applied Multivariate Analysis By Paul E. Green and J. Douglas Carroll The print that I have is from 1976 The ISBN number is ...
37 views

### Matrix Formulation of Equations

I am trying to learn statistical learning and machine learning independently. One of the main challenges I am facing is that many resources/books use matrix formulation when expressing equations. I am ...
112 views

### Why is my A.T*A (A transpose A) matrix singular?

I'm running into an wall on my intuition when using least squares. I'm trying to simulate some data, for fun, and I'm getting a result that says my (A.T * A) matrix is singular. In order to condense ...
38 views

### Principal Component Analysis for a Vector with Mean 0

If we are deriving the first principal component for a vector $X$ with mean 0 and covariance matrix $\Sigma$, can we find it in any other form than $Y^{T}X$ where $Y^{T}$ is the eigenvector ...
842 views

### Why does logistic regression's likelihood function have no closed form?

The derivative of Log likelihood function of logistic regression with respect to theta is Why can't we equate it to zero and solve for theta so that we can obtain a 'closed form solution' for theta? ...
94 views

### What is the first order derivative of linear regression's cost function using matrix calculus?

For linear regression's cost function J(b), where X is a n*m matrix, b is a m*1 vector and y is n*1 vector: First order derivative with respect to vector b (coefficients) is shown to be Using the ...
51 views

### In most machine learning books, how are matrices of data organized?

I'm trying understand linear algebra but am still a bit confused. I think I am missing some of the conventions. Let's say there is a supervised learning problem. We have 100 observations, 7 ...
158 views

### In a linear regression, trying to show $R^2 = r_{xy}^2$ using projections / geometric intuition

In a linear regression $$Y = X\beta + \varepsilon,$$ I define two (standard) projection matrices. The projection matrix into subspace spanned by columns of the design matrix $X$:  H := X(X^\top ...
61 views

### Interpretation of $\mathbf{w^{\top}Cw}$ [closed]

I'm reading a piece on portfolio optimization where it is stated that $\mathbf{w^{\top}Cw}$ is the variance of the expected return, where $\mathbf{C}$ is a covariance matrix and $\mathbf{w}$ is a ...
16 views

### How to approximate a Hermitian matrix with a transposed cross product of a single matrix?

I have a complex square matrix, and wish to learn latent factors (equally weighted latent factors, so not SVD) from this matrix. Given a Hermitian matrix A of ...