Questions tagged [matrix-calculus]

Matrix calculus deals with the problems of differentiating (possibly matrix-valued) functions of matrices

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Deriving Optimizer of Quadratic Loss for Classification

I'm currently considering a binary classification problem where we have data points $X_1,\dots,X_n\in\mathbb{R}^d$ and labels $y_i=\pm1$. I'm using a simple linear model to model $y_i$, and it has the ...
user avatar
  • 157
0 votes
0 answers
24 views

Maximizing a unique trace quadratic form

I am dealing with an unsupervised problem where I have ended up with the following maximization problem: $\max_{C\in \mathbb{R}^{p\times n}}\sum_{i=0}^{m} tr(CA^ixx^\top A^{i^\top}C^\top) \\ \mathrm{...
user avatar
1 vote
0 answers
39 views

Matrix Derivation for Neural Network Formula

I am learning some insights of Neural network but I have some problem with the derivation of matrix for backpropagation. On an assumption that the formula for calculating for one node in a neural ...
user avatar
  • 111
6 votes
2 answers
505 views

matrix-calculus - Understanding numerator/denominator layouts

Consider the following machine-learning model: Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of training-examples. While performing reverse-mode ...
user avatar
  • 206
0 votes
1 answer
33 views

Confusion surrounding differentiation of parameter vector ridge regression

Let $\mathbf{\beta}$ be the parameter vector of a ridge regression. Now we can say that: \begin{equation} \frac{\partial \lambda \beta^T \beta}{\partial \beta}=2\lambda\beta. \end{equation} Why is ...
user avatar
0 votes
0 answers
20 views

Multiple Linear Regression Expressed with a Vector Function (Not Traditional Matrix Notation)

I wanted to deepen my understanding of linear regression models and stumbled upon an introduction that stirred up some questions. The standard multiple linear regression model is introduced as a first-...
user avatar
0 votes
0 answers
10 views

Differentiating a function with respect to a matrix [duplicate]

I'm new to matrix calculus and I'm trying to find the formulas for matrix differentiation. e.g. $\frac{\partial f}{\partial z}$ = zzx where z is a KxK matrix, and x is a vector in K I found a few ...
user avatar
0 votes
0 answers
19 views

batch renormalization questions

I was going in details through paper about batch renormalization (arxiv link). I don't quite understand two things there. Maybe there is anyone who faced similar issues / knows the answer and could ...
user avatar
2 votes
2 answers
74 views

Computing $\frac{\partial}{\partial W} || t_n -W^T \phi (x_n) ||^2$

In Ch 3.1.5 of Pattern Recognition and Machine Learning how do we take the derivative wrt $W$ of 3.33: $$ln(p(T|X,W,\beta))=\frac{NK}{2}ln(\frac{\beta}{2\pi}) - \frac{\beta}{2}\sum_{n=1}^N || t_n -W^T ...
user avatar
0 votes
0 answers
11 views

Intergrating product of multivariate normal and univariate normal to find marginal density [duplicate]

Suppose, $y_i|u_i\sim MN(X_i(t_i), \sigma_e^2I_{m_i})$ and $U\sim MN(0,I_p)$. Now how to find the marginal distribution of $f(y_i)=\int f(y_i|u_i)f(u_i)du_i$? Since one is multivariate normal and ...
user avatar
  • 1
1 vote
0 answers
31 views

Fisher Information Matrix of Matrix Variate F Distribution

Let $\mathbf{X}$ follow a matrix variate F distribution with pdf $$ \begin{align} f\left(\mathbf{X} | \mathbf{\Sigma}, n, \nu\right) = \frac{\Gamma_p(\frac{n + \nu }{2})}{\Gamma_p(\frac{n}{2})\...
user avatar
  • 762
2 votes
1 answer
316 views

How do I calculate the optimal hyperplane of SVMs by hand?

I am trying to get a better understanding of SVMs and their optimization process. I understand that we have a constrained optimization problem that we have to solve with Lagrange multipliers. The ...
user avatar
  • 283
0 votes
0 answers
24 views

Computing the Jacobian $J_F$ with $F = h \circ f$

Let $$ f: \mathbb{R}^l \rightarrow{} \mathbb{R}^m\\[.7ex] h: \mathbb{R}^m \rightarrow{} \mathbb{R}^o$$ and let $$F = h \circ f \quad (F : \mathbb{R}^l \rightarrow{} \mathbb{R}^o)$$ I want to compute ...
user avatar
  • 143
4 votes
1 answer
141 views

Differentiation step in OLS

In deriving the parameter estimate in OLS, we differentiate the following (in matrix form) $$y^T y - 2\beta^T X^T y + \beta^T X^T X \beta$$ The part of the differentiation I don't understand is why $$\...
user avatar
  • 556
1 vote
0 answers
26 views

Matrix form of elementwise derivations

The elementwise derivations w.r.t e of $$ J = \frac{1}{2}[\Sigma_{r,s=1}^{R}a_{rt}a_{st}k(e_r,e_s) - 2\Sigma_{r=1}^{R}a_{rt}k(e_r, x_t)]$$ can be given by: $$ \frac{\partial J}{\partial e_r} = \Sigma_{...
user avatar
0 votes
0 answers
173 views

What is the first order derivative of linear regression's cost function using matrix calculus?

For linear regression's cost function J(b), where X is a n*m matrix, b is a m*1 vector and y is n*1 vector: First order derivative with respect to vector b (coefficients) is shown to be Using the ...
user avatar
0 votes
0 answers
248 views

Finding the gradient $\nabla$ of the logistic regression cost function

I want to use vector calculus to derive the gradient $\nabla_wJ(w)$ of the logistic regression cost function $J(w) = -\textbf{y}\cdot ln\textbf{ s} - (\mathbf{1} - \textbf{y}) \cdot ln( \mathbf{1} - \...
user avatar
  • 157
0 votes
0 answers
22 views

How to derive mathematically that derivative of |Ax-y|^2 with respect to A is 2|Ax-y| x^T [duplicate]

How to get transpose part when derive mathematically $$ \frac {\partial|Ax-y|^2}{\partial A} = 2|Ax-y|x^T $$
user avatar
1 vote
0 answers
25 views

Optimize Log Likelihood Model based on Gaussian Process involving Matrix Calculus

Given \begin{equation} \text{temperature(t, y)} = a_0 + a_1t + X(t) \end{equation} where temperature(t, year) is the dataset temperature at day $t$ in year $y$. $a_0, a_1 \in \mathbb{R}$, and $X(t)$ ...
user avatar
  • 13
3 votes
1 answer
521 views

Minimize SSE function

Consider a data set in which each target $t_n$ is associated with a weighting factor $r_n > 0$, so that the sum-of-squares error funtion becomes $$SE(w)= \frac{1}{2} \sum_{n=1}^N r_n \left(\...
user avatar
0 votes
1 answer
170 views

Taking derivative for RNN back propogation

I am trying to understand the derivation of backpropagation for recurrent neural networks (RNNs) from this source: https://github.com/go2carter/nn-learn/blob/master/grad-deriv-tex/rnn-grad-deriv.pdf ...
user avatar
  • 147
2 votes
0 answers
139 views

Computing gradient of KL-divergence

Consider a normal distribution $\mathcal N(\boldsymbol{\mu}(w), \boldsymbol{\Sigma}(w))$, with mean $\boldsymbol{\mu}(w)$ and covariance $\boldsymbol{\Sigma}(w)$ that are parameterized by a vector of ...
user avatar
  • 21
1 vote
1 answer
53 views

Granger's representation theorem: Johansen's version

In his book 'Likelihood based inference in cointegrated Var', in order to get the expression for the Granger's representation theorem,, Johansen claims that: (1) $$\beta \bot(\alpha' \bot \beta \bot ...
user avatar
  • 131
3 votes
1 answer
999 views

Integrate out (covariance) matrix in Normal-Wishart distribution

In Gelman's Bayesian Data Analysis Chapter 3.6, he introduces the multivariate normal with unknown mean and variance, with the priors $\Sigma\sim \text{Inv-Wishart}_{\nu_0}(\Lambda_0^{-1})$ $\mu\...
user avatar
  • 135
2 votes
0 answers
41 views

Deriving Bayesian Predictive Distribution?

Given $ p(\mathbf{w} | \mathbf{t}, \alpha, \beta) = \mathcal{N}\left(\mathbf{w} | \mathbf{m}_{N}, \mathbf{S}_{N}\right)$ and $p(t | \mathbf{w}, \beta) = \mathcal{N}\left(t | \mathbf{w}^{\mathrm{T}} \...
user avatar
  • 21
3 votes
1 answer
145 views

Integrating out parameter with improper prior

I got this problem while I was reading the book "Machine Learning: A Probabilistic Perspective" by Kevin Murphy. It is in section 7.6.1 of the book. Assume the likelihood is given by $$ \begin{...
user avatar
  • 33
1 vote
0 answers
329 views

How to understand Jacobian Matrix from the geometric perspective?

I found a good lecture about Jacobian Matrix which was part of a statistics course. However, it was published 20 years ago and lack of explanation. As a beginner of statistics, I'm not able to find ...
user avatar
  • 111
2 votes
1 answer
68 views

Minimizing a least square [duplicate]

I'm a bit confused with matrix calculus. Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$ I don't see how this works. My plan was to ...
user avatar
2 votes
3 answers
182 views

Hessian of Log of Matrix-t distribution

I am trying to calculate the hessian of the log of the matrix-t distribution. I know that the log of the matrix-t distribution can be written: $$\log T_{N\times P}(X| \nu, M, \Sigma, \Omega) \propto -\...
user avatar
  • 878
1 vote
2 answers
425 views

Backpropagation gradients don't match approximated gradients

I am in the process of implementing back propagation into my image classification neural net. I am using this cost function with a sigmoid output layer and ReLU hidden layers. The neural net has 3 ...
user avatar
2 votes
1 answer
1k views

Derivative of covariance w.r.t. inverse covariance when elements are function of a vector

I have this equation: $$\nabla f^T x+ \nabla f^T \Sigma^{-1} (\Sigma \circ Q)x = -\frac{1}{2}\nabla f^T \Sigma^{-1} \nabla \tag{1}f$$ where $\nabla f,x$ are vectors, and $$\nabla f_i =a_i - E[\...
user avatar
  • 505
2 votes
1 answer
54 views

What is the solution of B for equation derivative of B^TΛΒ wrt B = 0?

So we are at a state where $\partial B^TΛΒ / \partial B = 0$ Trying to solve it using formula (53) of Matrix Cookbook. We derive that B = 0 Is this correct ?? ...
user avatar
1 vote
0 answers
76 views

Wrong vector calculus in lecture note 5 of cs224n, Stanford

I am studying NLP via cs224n from Stanford. I am reading this lecture note now. When you refer to the 5th page, they want to derive the gradient with respect to W for RNN, to show the mathematical ...
user avatar
  • 603
3 votes
1 answer
268 views

Question regarding backpropagation on a minibatch

Below is a simple model with 2 layers and no nonlinearity: $X$ is a minibatch of vector inputs, $\hat{y}$ is a vector of scalar outputs, $y$ is a vector of scalar responses, $l$ is a scalar loss, and ...
user avatar
  • 869
1 vote
3 answers
336 views

Unsolvable Integral?

Is the following integral solvable? $$P(X) = \int^{\infty}_{-\infty} \int^{\infty}_{-\infty} P(X|\mu,K)P(\mu|K)P(K) d\mu dK$$ with $$P(K) = \frac{|K| ^{(v-d-1)/2}}{2^{vd/2}|V|^{v/2}\Gamma_d|\frac{...
user avatar
  • 91
3 votes
0 answers
235 views

Dimensions in single layer NN gradient

Given a neural network with one hidden sigmoid layer and softmax output layer, I want to derive the gradient of the cross entropy loss with respect to the first weight matrix. This is equivalent to ...
user avatar
  • 201
6 votes
2 answers
954 views

Derivative of a quadratic form wrt a parameter in the matrix

I want to compute the derivative of: $\frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}$, (Note that C is a covariance matrix that depends on a set of parameters $\theta$) for which I used ...
user avatar
  • 61
6 votes
1 answer
8k views

What if do not use any activation function in the neural network? [duplicate]

or, for example, is it good to use activation function only for a last layer? As I know, if there are no activation functions in neural network, feedforward will be like simple matrix multiplication, ...
user avatar
3 votes
2 answers
163 views

What is the derivative of $\|X^T-S^TAX^T\|_F^2$ w.r.t $A$?

What is the derivative of $F = \|X^T-S^TAX^T\|_F^2$ w.r.t $A$, where $X \in\mathbb R^{d \times N}$, $S \in\mathbb R^{k \times N}$, and $A \in \mathbb{R}^{k \times N}$? I have tried, and it is as ...
user avatar
  • 51
0 votes
0 answers
40 views

Derivative of matrix w.r.t vector [duplicate]

I'm quite out of my element trying to do some matrix calculus. I would like to know what the derivative of $z^{T}y$ w.r.t $z$ is, where z, y are n length vectors. Can anyone suggest good resources ...
user avatar
1 vote
1 answer
86 views

Difference between minima of L1-regularized quadratics

How can I find $$F(A,b,x,c)=\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta+x^\top\theta+c||\theta||_1)-\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta-x^\top\theta+c||\...
user avatar
  • 386
2 votes
2 answers
2k views

Derivative of $(y-XB)' h(y-XB)$ with respect to $B$

Let $X$ be a $n\times p$ matrix, $y_{n\times 1}$ a vector and $B_{p\times 1}$ coefficients so that $y=XB$. Then what is the derivative of $$ (y-XB)' h(y-XB) $$ with respect to B, where $h(.)$ is an $...
user avatar
  • 29
2 votes
0 answers
440 views

Vectorized gradients for neural networks: matrix multiplication

In Stanford's 231N course, they discuss computing a vectorized form of the gradient of the loss function w.r.t. the matrix $W$ as the derivative of the matrix multiplication $WX$ w.r.t. $W$ multiplied ...
user avatar
  • 7,650
2 votes
1 answer
9k views

How do you derive the gradient for weighted least squares?

So in my previous "adventures in statsland" episode, I believe I was able to convert the weighted sum of squares cost function into matrix form (Formula $\ref{cost}$). $$ J(w) = (Xw - y)^T U(Xw-y) \...
user avatar
  • 307
3 votes
2 answers
118 views

Handling linear algebraic differentiation in OLS parameter estimation

Could someone please explain why: \begin{equation} \frac{\partial (Y-\beta^T X)^T (Y-\beta^T X)}{\partial \beta}=2X^T(Y-\beta^T X) \end{equation} and why: \begin{equation} \frac{\partial \lambda \...
user avatar
4 votes
1 answer
437 views

Change of Variable technique for two variables?

If, $\theta_1 = \ln \frac p{1-p}$ $\theta_2 = \ln \frac q{1-q}$ $\theta_2|\theta_1 \sim N(\theta_1, \sigma^2)$ which means $f(\theta_1,\theta_2) \propto e^{\frac{-(\theta_1-\theta_2)^2}{2\...
user avatar
4 votes
1 answer
126 views

What reparametrization of vector parameters makes the Jeffreys prior correspond to the uniform prior?

What reparametrization of vector of parameters $\theta$ makes the Jeffreys prior $$\sqrt{\det I(\theta)}$$ correspond to the uniform prior? A change of parametrization from $\theta$ to $\eta$ changes ...
user avatar
  • 13.7k
1 vote
0 answers
21 views

What are the steps to solve this problem on multi function optimization

Problem: I have 3 rooms with limited food consumption and limited oxygen. Only a certain amount of people with certain food consumption and breathing can be allowed in a room. Room 1 has space for 100 ...
user avatar
  • 11
4 votes
1 answer
607 views

Gaussian process regression - Matérn kernel gradient issue

I'm trying to use a Matérn 5/2 kernel for GP regression, so my kernel function is $ K(x,x')\triangleq\theta_0(1+\sqrt{5r(x,x')}+5/3r)\exp(-\sqrt{5r}), $ where $r(x,x')\triangleq\sum_{d=1}^D (x_d-x'_d)^...
user avatar
3 votes
0 answers
89 views

Back-prop question: can this gradient be decomposed?

So, I was going over the lectures for the Oxford 2015 deep learning course, and in the lectures, they introduce back-propagation as a recursive procedure which involves two key formulas: The ...
user avatar
  • 508