Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [matrix-calculus]

Matrix calculus deals with the problems of differentiating (possibly matrix-valued) functions of matrices

2
votes
1answer
69 views

Integrating out parameter with improper prior

I got this problem while I was reading the book "Machine Learning: A Probabilistic Perspective" by Kevin Murphy. It is in section 7.6.1 of the book. Assume the likelihood is given by $$ \begin{...
1
vote
0answers
113 views

How to understand Jacobian Matrix from the geometric perspective?

I found a good lecture about Jacobian Matrix which was part of a statistics course. However, it was published 20 years ago and lack of explanation. As a beginner of statistics, I'm not able to find ...
2
votes
1answer
44 views

Minimizing a least square [duplicate]

I'm a bit confused with matrix calculus. Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$ I don't see how this works. My plan was to ...
0
votes
0answers
52 views

Minimum variance linear combination of estimate and score, matrix derivative optimization

I want to get a combined estimate from two estimating equations, $S_1 = \sum_{i} S_{1,i}(\beta_1)$, and $S_2 = \sum_{i} S_{2,i}(\beta_2)$, where $S_1$ and $S_2$ are vectors of dimension $p,q$ ...
1
vote
3answers
97 views

Hessian of Log of Matrix-t distribution

I am trying to calculate the hessian of the log of the matrix-t distribution. I know that the log of the matrix-t distribution can be written: $$\log T_{N\times P}(X| \nu, M, \Sigma, \Omega) \propto -\...
1
vote
2answers
219 views

Backpropagation gradients don't match approximated gradients

I am in the process of implementing back propagation into my image classification neural net. I am using this cost function with a sigmoid output layer and ReLU hidden layers. The neural net has 3 ...
1
vote
1answer
211 views

Derivative of covariance w.r.t. inverse covariance when elements are function of a vector

I have this equation: $$\nabla f^T x+ \nabla f^T \Sigma^{-1} (\Sigma \circ Q)x = -\frac{1}{2}\nabla f^T \Sigma^{-1} \nabla \tag{1}f$$ where $\nabla f,x$ are vectors, and $$\nabla f_i =a_i - E[\...
1
vote
1answer
41 views

What is the solution of B for equation derivative of B^TΛΒ wrt B = 0?

So we are at a state where $\partial B^TΛΒ / \partial B = 0$ Trying to solve it using formula (53) of Matrix Cookbook. We derive that B = 0 Is this correct ?? ...
1
vote
0answers
56 views

Wrong vector calculus in lecture note 5 of cs224n, Stanford

I am studying NLP via cs224n from Stanford. I am reading this lecture note now. When you refer to the 5th page, they want to derive the gradient with respect to W for RNN, to show the mathematical ...
0
votes
1answer
114 views

Question regarding backpropagation on a minibatch

Below is a simple model with 2 layers and no nonlinearity: $X$ is a minibatch of vector inputs, $\hat{y}$ is a vector of scalar outputs, $y$ is a vector of scalar responses, $l$ is a scalar loss, and ...
1
vote
3answers
303 views

Unsolvable Integral?

Is the following integral solvable? $$P(X) = \int^{\infty}_{-\infty} \int^{\infty}_{-\infty} P(X|\mu,K)P(\mu|K)P(K) d\mu dK$$ with $$P(K) = \frac{|K| ^{(v-d-1)/2}}{2^{vd/2}|V|^{v/2}\Gamma_d|\frac{...
3
votes
0answers
109 views

Dimensions in single layer NN gradient

Given a neural network with one hidden sigmoid layer and softmax output layer, I want to derive the gradient of the cross entropy loss with respect to the first weight matrix. This is equivalent to ...
5
votes
2answers
251 views

Derivative of a quadratic form wrt a parameter in the matrix

I want to compute the derivative of: $\frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}$, (Note that C is a covariance matrix that depends on a set of parameters $\theta$) for which I used ...
5
votes
1answer
1k views

What if do not use any activation function in the neural network? [duplicate]

or, for example, is it good to use activation function only for a last layer? As I know, if there are no activation functions in neural network, feedforward will be like simple matrix multiplication, ...
3
votes
1answer
84 views

What is the derivative of $\|X^T-S^TAX^T\|_F^2$ w.r.t $A$?

What is the derivative of $F = \|X^T-S^TAX^T\|_F^2$ w.r.t $A$, where $X \in\mathbb R^{d \times N}$, $S \in\mathbb R^{k \times N}$, and $A \in \mathbb{R}^{k \times N}$? I have tried, and it is as ...
0
votes
0answers
38 views

Derivative of matrix w.r.t vector [duplicate]

I'm quite out of my element trying to do some matrix calculus. I would like to know what the derivative of $z^{T}y$ w.r.t $z$ is, where z, y are n length vectors. Can anyone suggest good resources ...
1
vote
1answer
73 views

Difference between minima of L1-regularized quadratics

How can I find $$F(A,b,x,c)=\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta+x^\top\theta+c||\theta||_1)-\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta-x^\top\theta+c||\...
2
votes
2answers
460 views

Derivative of $(y-XB)' h(y-XB)$ with respect to $B$

Let $X$ be a $n\times p$ matrix, $y_{n\times 1}$ a vector and $B_{p\times 1}$ coefficients so that $y=XB$. Then what is the derivative of $$ (y-XB)' h(y-XB) $$ with respect to B, where $h(.)$ is an $...
2
votes
0answers
279 views

Vectorized gradients for neural networks: matrix multiplication

In Stanford's 231N course, they discuss computing a vectorized form of the gradient of the loss function w.r.t. the matrix $W$ as the derivative of the matrix multiplication $WX$ w.r.t. $W$ multiplied ...
2
votes
1answer
4k views

How do you derive the gradient for weighted least squares?

So in my previous "adventures in statsland" episode, I believe I was able to convert the weighted sum of squares cost function into matrix form (Formula $\ref{cost}$). $$ J(w) = (Xw - y)^T U(Xw-y) \...
3
votes
2answers
96 views

Handling linear algebraic differentiation in OLS parameter estimation

Could someone please explain why: \begin{equation} \frac{\partial (Y-\beta^T X)^T (Y-\beta^T X)}{\partial \beta}=2X^T(Y-\beta^T X) \end{equation} and why: \begin{equation} \frac{\partial \lambda \...
3
votes
1answer
274 views

Change of Variable technique for two variables?

If, $\theta_1 = \ln \frac p{1-p}$ $\theta_2 = \ln \frac q{1-q}$ $\theta_2|\theta_1 \sim N(\theta_1, \sigma^2)$ which means $f(\theta_1,\theta_2) \propto e^{\frac{-(\theta_1-\theta_2)^2}{2\...
3
votes
1answer
58 views

What reparametrization of vector parameters makes the Jeffreys prior correspond to the uniform prior?

What reparametrization of vector of parameters $\theta$ makes the Jeffreys prior $$\sqrt{\det I(\theta)}$$ correspond to the uniform prior? A change of parametrization from $\theta$ to $\eta$ changes ...
1
vote
0answers
20 views

What are the steps to solve this problem on multi function optimization

Problem: I have 3 rooms with limited food consumption and limited oxygen. Only a certain amount of people with certain food consumption and breathing can be allowed in a room. Room 1 has space for 100 ...
4
votes
1answer
452 views

Gaussian process regression - Matérn kernel gradient issue

I'm trying to use a Matérn 5/2 kernel for GP regression, so my kernel function is $ K(x,x')\triangleq\theta_0(1+\sqrt{5r(x,x')}+5/3r)\exp(-\sqrt{5r}), $ where $r(x,x')\triangleq\sum_{d=1}^D (x_d-x'_d)^...
3
votes
0answers
82 views

Back-prop question: can this gradient be decomposed?

So, I was going over the lectures for the Oxford 2015 deep learning course, and in the lectures, they introduce back-propagation as a recursive procedure which involves two key formulas: The ...
20
votes
4answers
3k views

Textbooks on Matrix Calculus?

See this question on Math SE. Short story: I read The Elements of Statistical Learning and got frustrated when I was trying to verify some of the results, e.g., given $$\text{RSS}(\beta) = \left(\...
4
votes
1answer
483 views

Derivative of $x^T A^Ty$ with respect to $\Sigma$ where $A$ is (an upper triangle matrix and ) Cholesky decomposition of $\Sigma$

I would like to evaluate: $$ \frac{ \partial x^T A^Ty}{\partial \Sigma} $$ where $A$ is a Cholesky decomposition of $\Sigma$ and an upper triangle matrix such that $\Sigma = A^T A$, $x$ and $y$ are a ...
3
votes
1answer
566 views

How to differentiate with respect to a matrix?

How can I differentiate the following by $\mathbf{W}$ ? \begin{equation} \mathbf{Y} = (\mathbf{W}^T\mathbf{x} + b)^2 \end{equation} Where $\mathbf{W} \in \mathcal{R}^{d\times D}$ and $\mathbf(...
2
votes
2answers
190 views

Neural Network General Learning Dynamics of Gradient Descent

This might be simple to you but can someone tell me step by step how is matrix form of updating rule of $W^{32}$ and $W^{21}$ derived in this case? Consider linear three layer neural network model ...
5
votes
1answer
623 views

What are the 2nd derivatives of the log multivariate normal density?

I develop open-source statistical software (http://openmx.psyc.virginia.edu/), but matrix calculus is not my strong point. I need the 1st and 2nd derivatives of the log multivariate normal density. I ...
5
votes
2answers
685 views

Resources for matrix calculus for optimization

I'm a grad student trying to absorb from the book Pattern Recognition and Machine Learning. However, I found that I really need a good grasp of matrix calculus before I can deduct the formulas myself (...
3
votes
1answer
2k views

Proof of normal equation in regression using tensor notation

I'm struggling with a proof of the normal equation, so I posted a question which hopefully will get resolved soon. However, I mentioned there that I'm uncomfortable with the proofs dealing with matrix ...