Let $X$ be a $n\times p$ matrix, $y_{n\times 1}$ a vector and $B_{p\times 1}$ coefficients so that $y=XB$. Then what is the derivative of $$ (y-XB)' h(y-XB) $$ with respect to B, where $h(.)$ is an $R^n\rightarrow R^n$ differentiable function e.g. ($h(z)=z$) and $(')$ is the transpose of a matrix?

  • $\begingroup$ Use the Chain Rule and the Product Rule. $\endgroup$
    – whuber
    Nov 18 '16 at 16:21
  • $\begingroup$ @whuber I did but the result looks too strange since the dimension is not right for example the first derivative is $p\times 1$ and the second one seems impossible in dimension $\endgroup$
    – Rose
    Nov 18 '16 at 16:29
  • $\begingroup$ Are you sure the derivative with respect to B is defined? Because as you describe your problem, B is a vector of coefficients and X seems to be a variable of the space. So, as I see it, your question is equivalent to: how do I derive 2 * x with respect to 2, which does not make sense to me... Or maybe precise what is B. $\endgroup$
    – Eskapp
    Nov 18 '16 at 16:41
  • $\begingroup$ @Eskapp It is a linear function. Assume h(z)=z then the problem reduces to $(y-XB)'(y-xB)$, the familiar form of the multiple regression. $\endgroup$
    – Rose
    Nov 18 '16 at 16:45
  • $\begingroup$ @whuber please, can you explain to me how to do it? I really need help $\endgroup$
    – Rose
    Nov 18 '16 at 17:00

Apply the Chain Rule. (It's the only rule you need to know.) To do so, you need to break the overall function into the composition of functions whose derivatives you can find. This is typically done by inspecting its formula and unwinding it from the outside in.

Let all vectors be column vectors and identify $\mathbb{R}^n\times \mathbb{R}^n$ with $\mathbb{R}^{2n}$ by stacking the two components, $(\mathbf{x}, \mathbf{y}) = \pmatrix{\mathbf{x}\\\mathbf{y}}.$

The last operation is a function

$$u: \mathbb{R}^{2n} \to \mathbb{R};\quad u(\pmatrix{\mathbf{x}\\\mathbf{y}})=\mathbf{x}^\prime \mathbf{y}.$$

The penultimate operation is

$$v:\mathbb{R}^n \to \mathbb{R}^{2n};\quad v(\mathbf{x})=\pmatrix{\mathbf{x}\\ h(\mathbf{x})}.$$

The first, innermost operation is

$$w:\mathbb{R}^p \to \mathbb{R}^n; \quad w(\mathbf{b}) = y - Xb.$$

Their composition is the function

$$u\circ v \circ w: \mathbb{R}^p {\,\xrightarrow{\ w\ }}\,\mathbb{R}^n\,{\xrightarrow{\ v\ }}\,\mathbb{R}^{2n}\,{\xrightarrow{\ u\ }}\,\mathbb{R}; \quad (u\circ v\circ w)(\mathbf{b}) = u(v(w(\mathbf{b})))=h(\mathbf{b}).$$

The Chain Rule asserts that $h$ is differentiable when each of $u,v,w$ are differentiable and its derivative $Dh:\mathbb{R}^p \to \mathbb{R}$ (which will be written as a $1\times p$ matrix) is the composition of the derivatives (each evaluated at the appropriate values),

$$Dh = Du \circ Dv \circ Dw.$$

You need to find those derivatives. They are

$$(Dw)(\mathbf{b}) = -X,$$

$$(Dv)(\mathbf{x}) = \pmatrix{\mathbb{I}_n \\ (Dh)_\mathbf{x}};\quad \mathbf{x} = w(\mathbf{b});$$

(remember, $Dh$ is an $n\times n$ matrix), and

$$(Du)(\mathbb{x}, \mathbb{y}) = \pmatrix{\mathbf{y},&\mathbf{x}};\quad \mathbf{x}=w(\mathbf{b});\quad \mathbf{y} = h(w(\mathbf{b})).$$

To obtain the answer, do the matrix multiplication and plug in the values $\mathbb{x} = w(\mathbb{b})$ and $\mathbb{y} = h(\mathbb{x}) = h(w(\mathbb{b}))$.


Michael Spivak, Calculus on Manifolds (1965).

  • 1
    $\begingroup$ Your use of the self-study tag indicates your intention is to learn methods and principles ("explain ... how to do it") rather than to get a one-line answer. Moreover, there is little interest on this site (or anywhere else on Stackexchange) in providing one-liners: we want to help future readers who may have similar problems that can be solved the same way. Regardless, this answer is detailed and thorough: all the work has been laid out for you and performed, except for the mechanical procedures of multiplying the matrices and making the substitutions described by $u, v,$ and $w$. $\endgroup$
    – whuber
    Nov 18 '16 at 22:21
  • $\begingroup$ You actually made me more confused, why there are two $h$ in the formula? why Dv comes up with an $I_n$ matrix? $\endgroup$
    – Rose
    Nov 19 '16 at 11:01
  • $\begingroup$ Rose, the main point is that all these functions are linear, which makes finding the derivatives dead simple, because the derivative of a linear function is itself. In particular, since $\mathbf{x}=\mathbb{I}_n\mathbf{x},$ $(D\mathbf{x})=\mathbb{I}_n.$ $\endgroup$
    – whuber
    Oct 4 '19 at 21:13

For clarity, I'll use the convention that uppercase letters represent matrices, lowercase vectors, and Greek letters represent scalars.

First, introduce the vector $$z=y-Xb$$ and note that a function like $h(z)$ only makes sense if applied element-wise. Had the function argument been a square matrix, then there is some ambiguity since one could interpret it as a matrix-wise function. But with rectangular matrices (including vectors) there is no ambiguity.

Using a scalar argument, we get the ordinary derivative $$h=h(\lambda),\,\,\,\,\,g=\frac{dh}{d\lambda},\,\,\,\,\,dh=g\,d\lambda$$ With a vector argument, both $h$ and $g$ must be applied element-wise; we must also use an elementwise/Hadamard product (denoted by $\odot$). $$h=h(z),\,\,\,\,\,g=g(z),\,\,\,\,\,dh=g\odot dz$$ One final bit of notation is the use of a colon for the trace/Frobenius product. $$A:B={\rm Tr}(A^TB)=B:A$$ Putting it all together, write the function of interest in terms of these new variables, then find its differential and gradient. $$\eqalign{ \phi &= z:h \cr d\phi &= h:dz + z:dh \cr &= h:dz + z:(g\odot dz) \cr &= (h + g\odot z):dz \cr &= (h + g\odot z):(-X\,db) \cr &= -X^T(h + g\odot z):db \cr &= -X^T\Big(h + g\odot (y-Xb)\Big):db \cr \frac{\partial\phi}{\partial b} &= X^T\Big(g\odot (Xb-y)-h\Big) \cr }$$ One final trick is to replace the Hadamard product with the matrix product of a diagonal matrix. $$\eqalign{ G &= {\rm Diag}(g) \cr \frac{\partial\phi}{\partial b} &= X^TG(Xb-y)-X^Th \cr }$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.