Skip to main content

Questions tagged [matrix-calculus]

Matrix calculus deals with the problems of differentiating (possibly matrix-valued) functions of matrices

Filter by
Sorted by
Tagged with
3 votes
1 answer
103 views

Kalman Filter to minimize weighted errors on the states: what's wrong with my derivation

I am thinking about how to implement a "weighted Kalman Filter". Note that the weights here are on the states. Basically the classical KF minimizes $\sum (x_i - \hat{x_i} )^2$ but I want to ...
Taylor Fang's user avatar
0 votes
0 answers
28 views

Simple matrix calculus but I am struggling to understand [duplicate]

Here is my problem: We have $\mathbf{D} \in \Re^{m n}$, $\mathbf{W} \in \Re^{m q}$, and $\mathbf{X} \in \Re^{q n}$. Furthermore, $\mathbf{D} = \mathbf{W}\mathbf{X}$. (NOT an element wise ...
wrek's user avatar
  • 195
4 votes
2 answers
774 views

Derivative of vector Y=XW with respect to matrix W

In the first derivation of dL/dW, I use the rule for the derivative of a constant with respect to a matrix and then apply the chain rule. \begin{gather*} Y\ =\ XW\ +\ B\\ X=\begin{bmatrix} x_{0} & ...
BPDev's user avatar
  • 141
0 votes
0 answers
36 views

Is it possible to find an explicit equation for this maximum likelihood for a particular variable of mean function

I have a log-likelihood equation that involves multivariate normal. Let's say, $le = \sum_{i=1}^n logf(y_i)$ and $f(y)=(2\pi)^{-\dfrac{n}{2}}|\sigma^2I_n|^{-\dfrac{1}{2}}exp[-\dfrac{1}{2}(y-x(t))^T|\...
user378967's user avatar
1 vote
0 answers
32 views

Matrix completion with proximal gradient method

I am trying to solve the matrix completion problem with proximal gradient method: $$\min_{||X||_* \leq \theta} \frac{1}{2}\sum_{(i, j) \in \Omega} (X_{ij} - M_{ij})^2$$ or in terms of the projection ...
np-hard's user avatar
  • 11
2 votes
1 answer
150 views

Differentiation of multiple correlation coefficient

Hi is there anyone that can help me with this calculation? So, I don’t know the exact way to differentiate the fraction of vector function. I tried to use the formal way but the components of the ...
Jay's user avatar
  • 33
0 votes
1 answer
194 views

Back Propagation Derivation - where am I going wrong

This is a rather long question. Sorry for that. The main thing is that I tried to derive out backpropagation via chain rule etc. I am aware that the index notation changes to transpose the matrix ...
Alex Punnen's user avatar
0 votes
0 answers
98 views

Maximizing a unique trace quadratic form

I am dealing with an unsupervised problem where I have ended up with the following maximization problem: $\max_{C\in \mathbb{R}^{p\times n}}\sum_{i=0}^{m} tr(CA^ixx^\top A^{i^\top}C^\top) \\ \mathrm{...
ARandomName's user avatar
4 votes
1 answer
279 views

Matrix Derivation for Neural Network Formula

I am learning some insights of Neural network but I have some problem with the derivation of matrix for backpropagation. On an assumption that the formula for calculating for one node in a neural ...
Hoang Nam's user avatar
  • 141
8 votes
2 answers
2k views

matrix-calculus - Understanding numerator/denominator layouts

Also see this question for more external references! Consider the following machine-learning model: Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of ...
x.projekt's user avatar
  • 240
0 votes
1 answer
49 views

Confusion surrounding differentiation of parameter vector ridge regression

Let $\mathbf{\beta}$ be the parameter vector of a ridge regression. Now we can say that: \begin{equation} \frac{\partial \lambda \beta^T \beta}{\partial \beta}=2\lambda\beta. \end{equation} Why is ...
Daniel De Wet's user avatar
2 votes
2 answers
83 views

Computing $\frac{\partial}{\partial W} || t_n -W^T \phi (x_n) ||^2$

In Ch 3.1.5 of Pattern Recognition and Machine Learning how do we take the derivative wrt $W$ of 3.33: $$ln(p(T|X,W,\beta))=\frac{NK}{2}ln(\frac{\beta}{2\pi}) - \frac{\beta}{2}\sum_{n=1}^N || t_n -W^T ...
tail_recursion's user avatar
1 vote
0 answers
66 views

Fisher Information Matrix of Matrix Variate F Distribution

Let $\mathbf{X}$ follow a matrix variate F distribution with pdf $$ \begin{align} f\left(\mathbf{X} | \mathbf{\Sigma}, n, \nu\right) = \frac{\Gamma_p(\frac{n + \nu }{2})}{\Gamma_p(\frac{n}{2})\...
stollenm's user avatar
  • 842
3 votes
1 answer
986 views

How do I calculate the optimal hyperplane of SVMs by hand?

I am trying to get a better understanding of SVMs and their optimization process. I understand that we have a constrained optimization problem that we have to solve with Lagrange multipliers. The ...
DocDriven's user avatar
  • 343
0 votes
0 answers
27 views

Computing the Jacobian $J_F$ with $F = h \circ f$

Let $$ f: \mathbb{R}^l \rightarrow{} \mathbb{R}^m\\[.7ex] h: \mathbb{R}^m \rightarrow{} \mathbb{R}^o$$ and let $$F = h \circ f \quad (F : \mathbb{R}^l \rightarrow{} \mathbb{R}^o)$$ I want to compute ...
lalaland's user avatar
  • 247
4 votes
1 answer
190 views

Differentiation step in OLS

In deriving the parameter estimate in OLS, we differentiate the following (in matrix form) $$y^T y - 2\beta^T X^T y + \beta^T X^T X \beta$$ The part of the differentiation I don't understand is why $$\...
Sparsity's user avatar
  • 584
1 vote
0 answers
31 views

Matrix form of elementwise derivations

The elementwise derivations w.r.t e of $$ J = \frac{1}{2}[\Sigma_{r,s=1}^{R}a_{rt}a_{st}k(e_r,e_s) - 2\Sigma_{r=1}^{R}a_{rt}k(e_r, x_t)]$$ can be given by: $$ \frac{\partial J}{\partial e_r} = \Sigma_{...
MathLearner's user avatar
0 votes
0 answers
280 views

What is the first order derivative of linear regression's cost function using matrix calculus?

For linear regression's cost function J(b), where X is a n*m matrix, b is a m*1 vector and y is n*1 vector: First order derivative with respect to vector b (coefficients) is shown to be Using the ...
Preetham_tsp's user avatar
0 votes
0 answers
406 views

Finding the gradient $\nabla$ of the logistic regression cost function

I want to use vector calculus to derive the gradient $\nabla_wJ(w)$ of the logistic regression cost function $J(w) = -\textbf{y}\cdot ln\textbf{ s} - (\mathbf{1} - \textbf{y}) \cdot ln( \mathbf{1} - \...
Jake's user avatar
  • 199
0 votes
0 answers
23 views

How to derive mathematically that derivative of |Ax-y|^2 with respect to A is 2|Ax-y| x^T [duplicate]

How to get transpose part when derive mathematically $$ \frac {\partial|Ax-y|^2}{\partial A} = 2|Ax-y|x^T $$
Rishi Jaiswal's user avatar
1 vote
0 answers
25 views

Optimize Log Likelihood Model based on Gaussian Process involving Matrix Calculus

Given \begin{equation} \text{temperature(t, y)} = a_0 + a_1t + X(t) \end{equation} where temperature(t, year) is the dataset temperature at day $t$ in year $y$. $a_0, a_1 \in \mathbb{R}$, and $X(t)$ ...
Yoona's user avatar
  • 13
3 votes
1 answer
1k views

Minimize SSE function

Consider a data set in which each target $t_n$ is associated with a weighting factor $r_n > 0$, so that the sum-of-squares error funtion becomes $$SE(w)= \frac{1}{2} \sum_{n=1}^N r_n \left(\...
Marcel Braasch's user avatar
0 votes
1 answer
484 views

Taking derivative for RNN back propogation

I am trying to understand the derivation of backpropagation for recurrent neural networks (RNNs) from this source: https://github.com/go2carter/nn-learn/blob/master/grad-deriv-tex/rnn-grad-deriv.pdf ...
prony's user avatar
  • 147
2 votes
0 answers
228 views

Computing gradient of KL-divergence

Consider a normal distribution $\mathcal N(\boldsymbol{\mu}(w), \boldsymbol{\Sigma}(w))$, with mean $\boldsymbol{\mu}(w)$ and covariance $\boldsymbol{\Sigma}(w)$ that are parameterized by a vector of ...
evo99's user avatar
  • 21
1 vote
1 answer
106 views

Granger's representation theorem: Johansen's version

In his book 'Likelihood based inference in cointegrated Var', in order to get the expression for the Granger's representation theorem,, Johansen claims that: (1) $$\beta \bot(\alpha' \bot \beta \bot ...
Alchemy's user avatar
  • 131
5 votes
1 answer
2k views

Integrate out (covariance) matrix in Normal-Wishart distribution

In Gelman's Bayesian Data Analysis Chapter 3.6, he introduces the multivariate normal with unknown mean and variance, with the priors $\Sigma\sim \text{Inv-Wishart}_{\nu_0}(\Lambda_0^{-1})$ $\mu\...
bayes's user avatar
  • 165
2 votes
0 answers
51 views

Deriving Bayesian Predictive Distribution?

Given $ p(\mathbf{w} | \mathbf{t}, \alpha, \beta) = \mathcal{N}\left(\mathbf{w} | \mathbf{m}_{N}, \mathbf{S}_{N}\right)$ and $p(t | \mathbf{w}, \beta) = \mathcal{N}\left(t | \mathbf{w}^{\mathrm{T}} \...
nickT's user avatar
  • 21
3 votes
1 answer
185 views

Integrating out parameter with improper prior

I got this problem while I was reading the book "Machine Learning: A Probabilistic Perspective" by Kevin Murphy. It is in section 7.6.1 of the book. Assume the likelihood is given by $$ \begin{...
zwcikyf's user avatar
  • 33
1 vote
0 answers
565 views

How to understand Jacobian Matrix from the geometric perspective?

I found a good lecture about Jacobian Matrix which was part of a statistics course. However, it was published 20 years ago and lack of explanation. As a beginner of statistics, I'm not able to find ...
Tab's user avatar
  • 111
2 votes
1 answer
81 views

Minimizing a least square [duplicate]

I'm a bit confused with matrix calculus. Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$ I don't see how this works. My plan was to ...
Mr.Sh4nnon's user avatar
2 votes
3 answers
248 views

Hessian of Log of Matrix-t distribution

I am trying to calculate the hessian of the log of the matrix-t distribution. I know that the log of the matrix-t distribution can be written: $$\log T_{N\times P}(X| \nu, M, \Sigma, \Omega) \propto -\...
jds's user avatar
  • 978
2 votes
2 answers
506 views

Backpropagation gradients don't match approximated gradients

I am in the process of implementing back propagation into my image classification neural net. I am using this cost function with a sigmoid output layer and ReLU hidden layers. The neural net has 3 ...
random0620's user avatar
2 votes
1 answer
2k views

Derivative of covariance w.r.t. inverse covariance when elements are function of a vector

I have this equation: $$\nabla f^T x+ \nabla f^T \Sigma^{-1} (\Sigma \circ Q)x = -\frac{1}{2}\nabla f^T \Sigma^{-1} \nabla \tag{1}f$$ where $\nabla f,x$ are vectors, and $$\nabla f_i =a_i - E[\...
ben's user avatar
  • 515
2 votes
1 answer
61 views

What is the solution of B for equation derivative of B^TΛΒ wrt B = 0?

So we are at a state where $\partial B^TΛΒ / \partial B = 0$ Trying to solve it using formula (53) of Matrix Cookbook. We derive that B = 0 Is this correct ?? ...
George Pligoropoulos's user avatar
1 vote
0 answers
81 views

Wrong vector calculus in lecture note 5 of cs224n, Stanford

I am studying NLP via cs224n from Stanford. I am reading this lecture note now. When you refer to the 5th page, they want to derive the gradient with respect to W for RNN, to show the mathematical ...
DongukJu's user avatar
  • 663
3 votes
1 answer
338 views

Question regarding backpropagation on a minibatch

Below is a simple model with 2 layers and no nonlinearity: $X$ is a minibatch of vector inputs, $\hat{y}$ is a vector of scalar outputs, $y$ is a vector of scalar responses, $l$ is a scalar loss, and ...
tmakino's user avatar
  • 941
1 vote
3 answers
384 views

Unsolvable Integral?

Is the following integral solvable? $$P(X) = \int^{\infty}_{-\infty} \int^{\infty}_{-\infty} P(X|\mu,K)P(\mu|K)P(K) d\mu dK$$ with $$P(K) = \frac{|K| ^{(v-d-1)/2}}{2^{vd/2}|V|^{v/2}\Gamma_d|\frac{...
ejlouw's user avatar
  • 191
3 votes
0 answers
306 views

Dimensions in single layer NN gradient

Given a neural network with one hidden sigmoid layer and softmax output layer, I want to derive the gradient of the cross entropy loss with respect to the first weight matrix. This is equivalent to ...
PKuhn's user avatar
  • 211
6 votes
2 answers
2k views

Derivative of a quadratic form wrt a parameter in the matrix

I want to compute the derivative of: $\frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}$, (Note that C is a covariance matrix that depends on a set of parameters $\theta$) for which I used ...
Emily W's user avatar
  • 61
6 votes
1 answer
10k views

What if do not use any activation function in the neural network? [duplicate]

or, for example, is it good to use activation function only for a last layer? As I know, if there are no activation functions in neural network, feedforward will be like simple matrix multiplication, ...
Dmytro Nalyvaiko's user avatar
4 votes
2 answers
268 views

What is the derivative of $\|X^T-S^TAX^T\|_F^2$ w.r.t $A$?

What is the derivative of $F = \|X^T-S^TAX^T\|_F^2$ w.r.t $A$, where $X \in\mathbb R^{d \times N}$, $S \in\mathbb R^{k \times N}$, and $A \in \mathbb{R}^{k \times N}$? I have tried, and it is as ...
Elyor's user avatar
  • 63
0 votes
0 answers
40 views

Derivative of matrix w.r.t vector [duplicate]

I'm quite out of my element trying to do some matrix calculus. I would like to know what the derivative of $z^{T}y$ w.r.t $z$ is, where z, y are n length vectors. Can anyone suggest good resources ...
Daniel Cole's user avatar
1 vote
1 answer
91 views

Difference between minima of L1-regularized quadratics

How can I find $$F(A,b,x,c)=\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta+x^\top\theta+c||\theta||_1)-\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta-x^\top\theta+c||\...
Waqas's user avatar
  • 426
2 votes
2 answers
2k views

Derivative of $(y-XB)' h(y-XB)$ with respect to $B$

Let $X$ be a $n\times p$ matrix, $y_{n\times 1}$ a vector and $B_{p\times 1}$ coefficients so that $y=XB$. Then what is the derivative of $$ (y-XB)' h(y-XB) $$ with respect to B, where $h(.)$ is an $...
Rose's user avatar
  • 29
2 votes
0 answers
475 views

Vectorized gradients for neural networks: matrix multiplication

In Stanford's 231N course, they discuss computing a vectorized form of the gradient of the loss function w.r.t. the matrix $W$ as the derivative of the matrix multiplication $WX$ w.r.t. $W$ multiplied ...
B_Miner's user avatar
  • 8,810
2 votes
1 answer
11k views

How do you derive the gradient for weighted least squares?

So in my previous "adventures in statsland" episode, I believe I was able to convert the weighted sum of squares cost function into matrix form (Formula $\ref{cost}$). $$ J(w) = (Xw - y)^T U(Xw-y) \...
vega's user avatar
  • 347
3 votes
2 answers
172 views

Handling linear algebraic differentiation in OLS parameter estimation

Could someone please explain why: \begin{equation} \frac{\partial (Y-\beta^T X)^T (Y-\beta^T X)}{\partial \beta}=2X^T(Y-\beta^T X) \end{equation} and why: \begin{equation} \frac{\partial \lambda \...
Ceyer Wakilpoor's user avatar
4 votes
1 answer
556 views

Change of Variable technique for two variables?

If, $\theta_1 = \ln \frac p{1-p}$ $\theta_2 = \ln \frac q{1-q}$ $\theta_2|\theta_1 \sim N(\theta_1, \sigma^2)$ which means $f(\theta_1,\theta_2) \propto e^{\frac{-(\theta_1-\theta_2)^2}{2\...
Jenna Maiz's user avatar
4 votes
1 answer
192 views

What reparametrization of vector parameters makes the Jeffreys prior correspond to the uniform prior?

What reparametrization of vector of parameters $\theta$ makes the Jeffreys prior $$\sqrt{\det I(\theta)}$$ correspond to the uniform prior? A change of parametrization from $\theta$ to $\eta$ changes ...
Neil G's user avatar
  • 15.4k
1 vote
0 answers
24 views

What are the steps to solve this problem on multi function optimization

Problem: I have 3 rooms with limited food consumption and limited oxygen. Only a certain amount of people with certain food consumption and breathing can be allowed in a room. Room 1 has space for 100 ...
mac's user avatar
  • 11