Questions tagged [matrix-calculus]

Matrix calculus deals with the problems of differentiating (possibly matrix-valued) functions of matrices

Filter by
Sorted by
Tagged with
1
vote
0answers
20 views

Matrix form of elementwise derivations

The elementwise derivations w.r.t e of $$ J = \frac{1}{2}[\Sigma_{r,s=1}^{R}a_{rt}a_{st}k(e_r,e_s) - 2\Sigma_{r=1}^{R}a_{rt}k(e_r, x_t)]$$ can be given by: $$ \frac{\partial J}{\partial e_r} = \Sigma_{...
0
votes
0answers
54 views

What is the first order derivative of linear regression's cost function using matrix calculus?

For linear regression's cost function J(b), where X is a n*m matrix, b is a m*1 vector and y is n*1 vector: First order derivative with respect to vector b (coefficients) is shown to be Using the ...
0
votes
0answers
28 views

Increased computation time for training and prediction with reduced feature space?

I implemented a PCA algorithm to reduce the input feature space of my neural network from 230 to 110 features. My naive expectation was that if I train a neural network using the same hyper ...
0
votes
0answers
73 views

Finding the gradient $\nabla$ of the logistic regression cost function

I want to use vector calculus to derive the gradient $\nabla_wJ(w)$ of the logistic regression cost function $J(w) = -\textbf{y}\cdot ln\textbf{ s} - (\mathbf{1} - \textbf{y}) \cdot ln( \mathbf{1} - \...
0
votes
0answers
18 views

How to derive mathematically that derivative of |Ax-y|^2 with respect to A is 2|Ax-y| x^T [duplicate]

How to get transpose part when derive mathematically $$ \frac {\partial|Ax-y|^2}{\partial A} = 2|Ax-y|x^T $$
1
vote
0answers
24 views

Optimize Log Likelihood Model based on Gaussian Process involving Matrix Calculus

Given \begin{equation} \text{temperature(t, y)} = a_0 + a_1t + X(t) \end{equation} where temperature(t, year) is the dataset temperature at day $t$ in year $y$. $a_0, a_1 \in \mathbb{R}$, and $X(t)$ ...
3
votes
1answer
95 views

Minimize SSE function

Consider a data set in which each target $t_n$ is associated with a weighting factor $r_n > 0$, so that the sum-of-squares error funtion becomes $$SE(w)= \frac{1}{2} \sum_{n=1}^N r_n \left(\...
0
votes
1answer
57 views

Taking derivative for RNN back propogation

I am trying to understand the derivation of backpropagation for recurrent neural networks (RNNs) from this source: https://github.com/go2carter/nn-learn/blob/master/grad-deriv-tex/rnn-grad-deriv.pdf ...
1
vote
0answers
99 views

Computing gradient of KL-divergence

I'm trying to compute the gradient of the following quantity with respect to $\mathbf{w} \in \mathbb{R}^N$ based on the KL divergence between two normal distribution $\mathrm{KL}(N(\boldsymbol{\mu}_w,\...
1
vote
1answer
29 views

Granger's representation theorem: Johansen's version

In his book 'Likelihood based inference in cointegrated Var', in order to get the expression for the Granger's representation theorem,, Johansen claims that: (1) $$\beta \bot(\alpha' \bot \beta \bot ...
3
votes
1answer
509 views

Integrate out (covariance) matrix in Normal-Wishart distribution

In Gelman's Bayesian Data Analysis Chapter 3.6, he introduces the multivariate normal with unknown mean and variance, with the priors $\Sigma\sim \text{Inv-Wishart}_{\nu_0}(\Lambda_0^{-1})$ $\mu\...
2
votes
0answers
33 views

Deriving Bayesian Predictive Distribution?

Given $ p(\mathbf{w} | \mathbf{t}, \alpha, \beta) = \mathcal{N}\left(\mathbf{w} | \mathbf{m}_{N}, \mathbf{S}_{N}\right)$ and $p(t | \mathbf{w}, \beta) = \mathcal{N}\left(t | \mathbf{w}^{\mathrm{T}} \...
2
votes
1answer
89 views

Integrating out parameter with improper prior

I got this problem while I was reading the book "Machine Learning: A Probabilistic Perspective" by Kevin Murphy. It is in section 7.6.1 of the book. Assume the likelihood is given by $$ \begin{...
1
vote
0answers
258 views

How to understand Jacobian Matrix from the geometric perspective?

I found a good lecture about Jacobian Matrix which was part of a statistics course. However, it was published 20 years ago and lack of explanation. As a beginner of statistics, I'm not able to find ...
2
votes
1answer
58 views

Minimizing a least square [duplicate]

I'm a bit confused with matrix calculus. Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$ I don't see how this works. My plan was to ...
2
votes
3answers
142 views

Hessian of Log of Matrix-t distribution

I am trying to calculate the hessian of the log of the matrix-t distribution. I know that the log of the matrix-t distribution can be written: $$\log T_{N\times P}(X| \nu, M, \Sigma, \Omega) \propto -\...
1
vote
2answers
333 views

Backpropagation gradients don't match approximated gradients

I am in the process of implementing back propagation into my image classification neural net. I am using this cost function with a sigmoid output layer and ReLU hidden layers. The neural net has 3 ...
2
votes
1answer
770 views

Derivative of covariance w.r.t. inverse covariance when elements are function of a vector

I have this equation: $$\nabla f^T x+ \nabla f^T \Sigma^{-1} (\Sigma \circ Q)x = -\frac{1}{2}\nabla f^T \Sigma^{-1} \nabla \tag{1}f$$ where $\nabla f,x$ are vectors, and $$\nabla f_i =a_i - E[\...
1
vote
1answer
45 views

What is the solution of B for equation derivative of B^TΛΒ wrt B = 0?

So we are at a state where $\partial B^TΛΒ / \partial B = 0$ Trying to solve it using formula (53) of Matrix Cookbook. We derive that B = 0 Is this correct ?? ...
1
vote
0answers
65 views

Wrong vector calculus in lecture note 5 of cs224n, Stanford

I am studying NLP via cs224n from Stanford. I am reading this lecture note now. When you refer to the 5th page, they want to derive the gradient with respect to W for RNN, to show the mathematical ...
1
vote
1answer
194 views

Question regarding backpropagation on a minibatch

Below is a simple model with 2 layers and no nonlinearity: $X$ is a minibatch of vector inputs, $\hat{y}$ is a vector of scalar outputs, $y$ is a vector of scalar responses, $l$ is a scalar loss, and ...
1
vote
3answers
317 views

Unsolvable Integral?

Is the following integral solvable? $$P(X) = \int^{\infty}_{-\infty} \int^{\infty}_{-\infty} P(X|\mu,K)P(\mu|K)P(K) d\mu dK$$ with $$P(K) = \frac{|K| ^{(v-d-1)/2}}{2^{vd/2}|V|^{v/2}\Gamma_d|\frac{...
3
votes
0answers
179 views

Dimensions in single layer NN gradient

Given a neural network with one hidden sigmoid layer and softmax output layer, I want to derive the gradient of the cross entropy loss with respect to the first weight matrix. This is equivalent to ...
5
votes
2answers
575 views

Derivative of a quadratic form wrt a parameter in the matrix

I want to compute the derivative of: $\frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}$, (Note that C is a covariance matrix that depends on a set of parameters $\theta$) for which I used ...
5
votes
1answer
4k views

What if do not use any activation function in the neural network? [duplicate]

or, for example, is it good to use activation function only for a last layer? As I know, if there are no activation functions in neural network, feedforward will be like simple matrix multiplication, ...
3
votes
2answers
138 views

What is the derivative of $\|X^T-S^TAX^T\|_F^2$ w.r.t $A$?

What is the derivative of $F = \|X^T-S^TAX^T\|_F^2$ w.r.t $A$, where $X \in\mathbb R^{d \times N}$, $S \in\mathbb R^{k \times N}$, and $A \in \mathbb{R}^{k \times N}$? I have tried, and it is as ...
0
votes
0answers
39 views

Derivative of matrix w.r.t vector [duplicate]

I'm quite out of my element trying to do some matrix calculus. I would like to know what the derivative of $z^{T}y$ w.r.t $z$ is, where z, y are n length vectors. Can anyone suggest good resources ...
1
vote
1answer
78 views

Difference between minima of L1-regularized quadratics

How can I find $$F(A,b,x,c)=\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta+x^\top\theta+c||\theta||_1)-\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta-x^\top\theta+c||\...
2
votes
2answers
985 views

Derivative of $(y-XB)' h(y-XB)$ with respect to $B$

Let $X$ be a $n\times p$ matrix, $y_{n\times 1}$ a vector and $B_{p\times 1}$ coefficients so that $y=XB$. Then what is the derivative of $$ (y-XB)' h(y-XB) $$ with respect to B, where $h(.)$ is an $...
2
votes
0answers
353 views

Vectorized gradients for neural networks: matrix multiplication

In Stanford's 231N course, they discuss computing a vectorized form of the gradient of the loss function w.r.t. the matrix $W$ as the derivative of the matrix multiplication $WX$ w.r.t. $W$ multiplied ...
2
votes
1answer
6k views

How do you derive the gradient for weighted least squares?

So in my previous "adventures in statsland" episode, I believe I was able to convert the weighted sum of squares cost function into matrix form (Formula $\ref{cost}$). $$ J(w) = (Xw - y)^T U(Xw-y) \...
3
votes
2answers
102 views

Handling linear algebraic differentiation in OLS parameter estimation

Could someone please explain why: \begin{equation} \frac{\partial (Y-\beta^T X)^T (Y-\beta^T X)}{\partial \beta}=2X^T(Y-\beta^T X) \end{equation} and why: \begin{equation} \frac{\partial \lambda \...
3
votes
1answer
392 views

Change of Variable technique for two variables?

If, $\theta_1 = \ln \frac p{1-p}$ $\theta_2 = \ln \frac q{1-q}$ $\theta_2|\theta_1 \sim N(\theta_1, \sigma^2)$ which means $f(\theta_1,\theta_2) \propto e^{\frac{-(\theta_1-\theta_2)^2}{2\...
3
votes
1answer
90 views

What reparametrization of vector parameters makes the Jeffreys prior correspond to the uniform prior?

What reparametrization of vector of parameters $\theta$ makes the Jeffreys prior $$\sqrt{\det I(\theta)}$$ correspond to the uniform prior? A change of parametrization from $\theta$ to $\eta$ changes ...
1
vote
0answers
21 views

What are the steps to solve this problem on multi function optimization

Problem: I have 3 rooms with limited food consumption and limited oxygen. Only a certain amount of people with certain food consumption and breathing can be allowed in a room. Room 1 has space for 100 ...
4
votes
1answer
533 views

Gaussian process regression - Matérn kernel gradient issue

I'm trying to use a Matérn 5/2 kernel for GP regression, so my kernel function is $ K(x,x')\triangleq\theta_0(1+\sqrt{5r(x,x')}+5/3r)\exp(-\sqrt{5r}), $ where $r(x,x')\triangleq\sum_{d=1}^D (x_d-x'_d)^...
4
votes
0answers
87 views

Back-prop question: can this gradient be decomposed?

So, I was going over the lectures for the Oxford 2015 deep learning course, and in the lectures, they introduce back-propagation as a recursive procedure which involves two key formulas: The ...
22
votes
4answers
6k views

Textbooks on Matrix Calculus?

See this question on Math SE. Short story: I read The Elements of Statistical Learning and got frustrated when I was trying to verify some of the results, e.g., given $$\text{RSS}(\beta) = \left(\...
4
votes
1answer
593 views

Derivative of $x^T A^Ty$ with respect to $\Sigma$ where $A$ is (an upper triangle matrix and ) Cholesky decomposition of $\Sigma$

I would like to evaluate: $$ \frac{ \partial x^T A^Ty}{\partial \Sigma} $$ where $A$ is a Cholesky decomposition of $\Sigma$ and an upper triangle matrix such that $\Sigma = A^T A$, $x$ and $y$ are a ...
6
votes
1answer
2k views

How to differentiate with respect to a matrix?

How can I differentiate the following by $\mathbf{W}$ ? \begin{equation} \mathbf{Y} = (\mathbf{W}^T\mathbf{x} + b)^2 \end{equation} Where $\mathbf{W} \in \mathcal{R}^{d\times D}$ and $\mathbf(...
2
votes
2answers
214 views

Neural Network General Learning Dynamics of Gradient Descent

This might be simple to you but can someone tell me step by step how is matrix form of updating rule of $W^{32}$ and $W^{21}$ derived in this case? Consider linear three layer neural network model ...
5
votes
1answer
864 views

What are the 2nd derivatives of the log multivariate normal density?

I develop open-source statistical software (http://openmx.psyc.virginia.edu/), but matrix calculus is not my strong point. I need the 1st and 2nd derivatives of the log multivariate normal density. I ...
6
votes
2answers
1k views

Resources for matrix calculus for optimization

I'm a grad student trying to absorb from the book Pattern Recognition and Machine Learning. However, I found that I really need a good grasp of matrix calculus before I can deduct the formulas myself (...
5
votes
2answers
2k views

Proof of normal equation in regression using tensor notation

I'm struggling with a proof of the normal equation, so I posted a question which hopefully will get resolved soon. However, I mentioned there that I'm uncomfortable with the proofs dealing with matrix ...