Questions tagged [matrix-calculus]
Matrix calculus deals with the problems of differentiating (possibly matrix-valued) functions of matrices
58
questions
4
votes
1
answer
414
views
Derivative of vector Y=XW with respect to matrix W
In the first derivation of dL/dW, I use the rule for the derivative of a constant with respect to a matrix and then apply the chain rule.
\begin{gather*}
Y\ =\ XW\ +\ B\\
X=\begin{bmatrix}
x_{0} & ...
- 141
0
votes
0
answers
36
views
Is it possible to find an explicit equation for this maximum likelihood for a particular variable of mean function
I have a log-likelihood equation that involves multivariate normal. Let's say, $le = \sum_{i=1}^n logf(y_i)$ and
$f(y)=(2\pi)^{-\dfrac{n}{2}}|\sigma^2I_n|^{-\dfrac{1}{2}}exp[-\dfrac{1}{2}(y-x(t))^T|\...
1
vote
0
answers
29
views
Matrix completion with proximal gradient method
I am trying to solve the matrix completion problem with proximal gradient method: $$\min_{||X||_* \leq \theta} \frac{1}{2}\sum_{(i, j) \in \Omega} (X_{ij} - M_{ij})^2$$ or in terms of the projection ...
- 11
0
votes
0
answers
72
views
Finding the optimal dual variables in svm by hand
I need to find the optimal dual variables $\lambda$ for SVM using the quadratic kernel:
\begin{align}\max_{\overset{\rightarrow}{\lambda}}&\sum_{i=1}^N\lambda_i-\frac12\sum_{i=1}^N\sum_{j=1}^N\...
- 125
2
votes
1
answer
130
views
Differentiation of multiple correlation coefficient
Hi is there anyone that can help me with this calculation?
So, I don’t know the exact way to differentiate the fraction of vector function. I tried to use the formal way but the components of the ...
- 33
0
votes
1
answer
156
views
Back Propagation Derivation - where am I going wrong
This is a rather long question. Sorry for that. The main thing is that I tried to derive out backpropagation via chain rule etc. I am aware that the index notation changes to transpose the matrix ...
- 221
0
votes
0
answers
78
views
Maximizing a unique trace quadratic form
I am dealing with an unsupervised problem where I have ended up with the following maximization problem:
$\max_{C\in \mathbb{R}^{p\times n}}\sum_{i=0}^{m} tr(CA^ixx^\top A^{i^\top}C^\top) \\ \mathrm{...
- 43
1
vote
0
answers
183
views
Matrix Derivation for Neural Network Formula
I am learning some insights of Neural network but I have some problem with the derivation of matrix for backpropagation. On an assumption that the formula for calculating for one node in a neural ...
- 111
8
votes
2
answers
1k
views
matrix-calculus - Understanding numerator/denominator layouts
Also see this question for more external references!
Consider the following machine-learning model:
Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of ...
- 240
0
votes
1
answer
45
views
Confusion surrounding differentiation of parameter vector ridge regression
Let $\mathbf{\beta}$ be the parameter vector of a ridge regression.
Now we can say that:
\begin{equation}
\frac{\partial \lambda \beta^T \beta}{\partial \beta}=2\lambda\beta.
\end{equation}
Why is ...
- 107
2
votes
2
answers
82
views
Computing $\frac{\partial}{\partial W} || t_n -W^T \phi (x_n) ||^2$
In Ch 3.1.5 of Pattern Recognition and Machine Learning how do we take the derivative wrt $W$ of 3.33:
$$ln(p(T|X,W,\beta))=\frac{NK}{2}ln(\frac{\beta}{2\pi}) - \frac{\beta}{2}\sum_{n=1}^N || t_n -W^T ...
- 259
1
vote
0
answers
62
views
Fisher Information Matrix of Matrix Variate F Distribution
Let $\mathbf{X}$ follow a matrix variate F distribution with pdf
$$
\begin{align}
f\left(\mathbf{X} | \mathbf{\Sigma}, n, \nu\right) = \frac{\Gamma_p(\frac{n + \nu }{2})}{\Gamma_p(\frac{n}{2})\...
- 832
3
votes
1
answer
857
views
How do I calculate the optimal hyperplane of SVMs by hand?
I am trying to get a better understanding of SVMs and their optimization process. I understand that we have a constrained optimization problem that we have to solve with Lagrange multipliers.
The ...
- 343
0
votes
0
answers
25
views
Computing the Jacobian $J_F$ with $F = h \circ f$
Let
$$
f: \mathbb{R}^l \rightarrow{} \mathbb{R}^m\\[.7ex]
h: \mathbb{R}^m \rightarrow{} \mathbb{R}^o$$
and let $$F = h \circ f \quad (F : \mathbb{R}^l \rightarrow{} \mathbb{R}^o)$$
I want to compute ...
- 227
4
votes
1
answer
173
views
Differentiation step in OLS
In deriving the parameter estimate in OLS, we differentiate the following (in matrix form)
$$y^T y - 2\beta^T X^T y + \beta^T X^T X \beta$$
The part of the differentiation I don't understand is why $$\...
- 566
1
vote
0
answers
31
views
Matrix form of elementwise derivations
The elementwise derivations w.r.t e of $$ J = \frac{1}{2}[\Sigma_{r,s=1}^{R}a_{rt}a_{st}k(e_r,e_s) - 2\Sigma_{r=1}^{R}a_{rt}k(e_r, x_t)]$$
can be given by:
$$ \frac{\partial J}{\partial e_r} = \Sigma_{...
- 216
0
votes
0
answers
261
views
What is the first order derivative of linear regression's cost function using matrix calculus?
For linear regression's cost function J(b), where X is a n*m matrix, b is a m*1 vector and y is n*1 vector:
First order derivative with respect to vector b (coefficients) is shown to be
Using the ...
- 313
0
votes
0
answers
388
views
Finding the gradient $\nabla$ of the logistic regression cost function
I want to use vector calculus to derive the gradient $\nabla_wJ(w)$ of the logistic regression cost function $J(w) = -\textbf{y}\cdot ln\textbf{ s} - (\mathbf{1} - \textbf{y}) \cdot ln( \mathbf{1} - \...
- 199
0
votes
0
answers
23
views
How to derive mathematically that derivative of |Ax-y|^2 with respect to A is 2|Ax-y| x^T [duplicate]
How to get transpose part when derive mathematically $$ \frac {\partial|Ax-y|^2}{\partial A} = 2|Ax-y|x^T $$
1
vote
0
answers
25
views
Optimize Log Likelihood Model based on Gaussian Process involving Matrix Calculus
Given
\begin{equation}
\text{temperature(t, y)} = a_0 + a_1t + X(t)
\end{equation}
where temperature(t, year) is the dataset temperature at day $t$ in year $y$. $a_0, a_1 \in \mathbb{R}$, and $X(t)$ ...
- 13
3
votes
1
answer
977
views
Minimize SSE function
Consider a data set in which each target $t_n$ is associated with a weighting factor $r_n > 0$, so that the sum-of-squares error funtion becomes
$$SE(w)= \frac{1}{2} \sum_{n=1}^N r_n \left(\...
- 235
0
votes
1
answer
449
views
Taking derivative for RNN back propogation
I am trying to understand the derivation of backpropagation for recurrent neural networks (RNNs) from this source:
https://github.com/go2carter/nn-learn/blob/master/grad-deriv-tex/rnn-grad-deriv.pdf
...
- 147
2
votes
0
answers
213
views
Computing gradient of KL-divergence
Consider a normal distribution $\mathcal N(\boldsymbol{\mu}(w), \boldsymbol{\Sigma}(w))$, with mean $\boldsymbol{\mu}(w)$ and covariance $\boldsymbol{\Sigma}(w)$ that are parameterized by a vector of ...
- 21
1
vote
1
answer
101
views
Granger's representation theorem: Johansen's version
In his book 'Likelihood based inference in cointegrated Var', in order to get the expression for the Granger's representation theorem,, Johansen claims that:
(1)
$$\beta \bot(\alpha' \bot \beta \bot ...
- 131
4
votes
1
answer
1k
views
Integrate out (covariance) matrix in Normal-Wishart distribution
In Gelman's Bayesian Data Analysis Chapter 3.6, he introduces the multivariate normal with unknown mean and variance, with the priors
$\Sigma\sim \text{Inv-Wishart}_{\nu_0}(\Lambda_0^{-1})$
$\mu\...
- 145
2
votes
0
answers
49
views
Deriving Bayesian Predictive Distribution?
Given $ p(\mathbf{w} | \mathbf{t}, \alpha, \beta) = \mathcal{N}\left(\mathbf{w} | \mathbf{m}_{N}, \mathbf{S}_{N}\right)$ and $p(t | \mathbf{w}, \beta) = \mathcal{N}\left(t | \mathbf{w}^{\mathrm{T}} \...
- 21
3
votes
1
answer
181
views
Integrating out parameter with improper prior
I got this problem while I was reading the book "Machine Learning: A Probabilistic Perspective" by Kevin Murphy. It is in section 7.6.1 of the book.
Assume the likelihood is given by
$$
\begin{...
- 33
1
vote
0
answers
489
views
How to understand Jacobian Matrix from the geometric perspective?
I found a good lecture about Jacobian Matrix which was part of a statistics course. However, it was published 20 years ago and lack of explanation. As a beginner of statistics, I'm not able to find ...
- 111
2
votes
1
answer
78
views
Minimizing a least square [duplicate]
I'm a bit confused with matrix calculus.
Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$
and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$
I don't see how this works. My plan was to ...
- 651
2
votes
3
answers
235
views
Hessian of Log of Matrix-t distribution
I am trying to calculate the hessian of the log of the matrix-t distribution. I know that the log of the matrix-t distribution can be written:
$$\log T_{N\times P}(X| \nu, M, \Sigma, \Omega) \propto -\...
- 978
2
votes
2
answers
498
views
Backpropagation gradients don't match approximated gradients
I am in the process of implementing back propagation into my image classification neural net.
I am using this cost function with a sigmoid output layer and ReLU hidden layers.
The neural net has 3 ...
- 216
2
votes
1
answer
1k
views
Derivative of covariance w.r.t. inverse covariance when elements are function of a vector
I have this equation:
$$\nabla f^T x+ \nabla f^T \Sigma^{-1} (\Sigma \circ Q)x = -\frac{1}{2}\nabla f^T \Sigma^{-1} \nabla \tag{1}f$$
where $\nabla f,x$ are vectors, and
$$\nabla f_i =a_i - E[\...
- 505
2
votes
1
answer
55
views
What is the solution of B for equation derivative of B^TΛΒ wrt B = 0?
So we are at a state where $\partial B^TΛΒ / \partial B = 0$
Trying to solve it using formula (53) of Matrix Cookbook.
We derive that B = 0
Is this correct ??
...
1
vote
0
answers
78
views
Wrong vector calculus in lecture note 5 of cs224n, Stanford
I am studying NLP via cs224n from Stanford.
I am reading this lecture note now.
When you refer to the 5th page, they want to derive the gradient with respect to W for RNN, to show the mathematical ...
- 623
3
votes
1
answer
320
views
Question regarding backpropagation on a minibatch
Below is a simple model with 2 layers and no nonlinearity:
$X$ is a minibatch of vector inputs, $\hat{y}$ is a vector of scalar outputs, $y$ is a vector of scalar responses, $l$ is a scalar loss, and ...
- 921
1
vote
3
answers
377
views
Unsolvable Integral?
Is the following integral solvable?
$$P(X) = \int^{\infty}_{-\infty} \int^{\infty}_{-\infty} P(X|\mu,K)P(\mu|K)P(K) d\mu dK$$
with
$$P(K) = \frac{|K| ^{(v-d-1)/2}}{2^{vd/2}|V|^{v/2}\Gamma_d|\frac{...
- 191
3
votes
0
answers
298
views
Dimensions in single layer NN gradient
Given a neural network with one hidden sigmoid layer and softmax output layer, I want to derive the gradient of the cross entropy loss with respect to the first weight matrix. This is equivalent to ...
- 211
6
votes
2
answers
1k
views
Derivative of a quadratic form wrt a parameter in the matrix
I want to compute the derivative of:
$\frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}$,
(Note that C is a covariance matrix that depends on a set of parameters $\theta$)
for which I used ...
- 61
6
votes
1
answer
10k
views
What if do not use any activation function in the neural network? [duplicate]
or, for example, is it good to use activation function only for a last layer?
As I know, if there are no activation functions in neural network, feedforward will be like simple matrix multiplication, ...
- 617
4
votes
2
answers
229
views
What is the derivative of $\|X^T-S^TAX^T\|_F^2$ w.r.t $A$?
What is the derivative of $F = \|X^T-S^TAX^T\|_F^2$ w.r.t $A$,
where $X \in\mathbb R^{d \times N}$, $S \in\mathbb R^{k \times N}$, and $A \in \mathbb{R}^{k \times N}$?
I have tried, and it is as ...
- 63
0
votes
0
answers
40
views
Derivative of matrix w.r.t vector [duplicate]
I'm quite out of my element trying to do some matrix calculus.
I would like to know what the derivative of $z^{T}y$ w.r.t $z$ is, where z, y are n length vectors.
Can anyone suggest good resources ...
- 11
1
vote
1
answer
89
views
Difference between minima of L1-regularized quadratics
How can I find
$$F(A,b,x,c)=\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta+x^\top\theta+c||\theta||_1)-\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta-x^\top\theta+c||\...
- 416
2
votes
2
answers
2k
views
Derivative of $(y-XB)' h(y-XB)$ with respect to $B$
Let $X$ be a $n\times p$ matrix, $y_{n\times 1}$ a vector and $B_{p\times 1}$ coefficients so that $y=XB$. Then what is the derivative of
$$
(y-XB)' h(y-XB)
$$
with respect to B, where $h(.)$ is an $...
- 29
2
votes
0
answers
467
views
Vectorized gradients for neural networks: matrix multiplication
In Stanford's 231N course, they discuss computing a vectorized form of the gradient of the loss function w.r.t. the matrix $W$ as the derivative of the matrix multiplication $WX$ w.r.t. $W$ multiplied ...
- 8,170
2
votes
1
answer
11k
views
How do you derive the gradient for weighted least squares?
So in my previous "adventures in statsland" episode, I believe I was able to convert the weighted sum of squares cost function into matrix form (Formula $\ref{cost}$).
$$
J(w) = (Xw - y)^T U(Xw-y) \...
- 347
3
votes
2
answers
154
views
Handling linear algebraic differentiation in OLS parameter estimation
Could someone please explain why:
\begin{equation}
\frac{\partial (Y-\beta^T X)^T (Y-\beta^T X)}{\partial \beta}=2X^T(Y-\beta^T X)
\end{equation}
and why:
\begin{equation}
\frac{\partial \lambda \...
4
votes
1
answer
535
views
Change of Variable technique for two variables?
If,
$\theta_1 = \ln \frac p{1-p}$
$\theta_2 = \ln \frac q{1-q}$
$\theta_2|\theta_1 \sim N(\theta_1, \sigma^2)$ which means
$f(\theta_1,\theta_2) \propto e^{\frac{-(\theta_1-\theta_2)^2}{2\...
- 899
4
votes
1
answer
188
views
What reparametrization of vector parameters makes the Jeffreys prior correspond to the uniform prior?
What reparametrization of vector of parameters $\theta$ makes the Jeffreys prior
$$\sqrt{\det I(\theta)}$$
correspond to the uniform prior?
A change of parametrization from $\theta$ to $\eta$ changes ...
- 14.8k
1
vote
0
answers
24
views
What are the steps to solve this problem on multi function optimization
Problem:
I have 3 rooms with limited food consumption and limited oxygen. Only a certain amount of people with certain food consumption and breathing can be allowed in a room. Room 1 has space for 100 ...
- 11
5
votes
1
answer
678
views
Gaussian process regression - Matérn kernel gradient issue
I'm trying to use a Matérn 5/2 kernel for GP regression, so my kernel function is
$ K(x,x')\triangleq\theta_0(1+\sqrt{5r(x,x')}+5/3r)\exp(-\sqrt{5r}), $
where $r(x,x')\triangleq\sum_{d=1}^D (x_d-x'_d)^...
- 191