Questions tagged [derivative]

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

Filter by
Sorted by
Tagged with
49
votes
5answers
97k views

Backpropagation with Softmax / Cross Entropy

I'm trying to understand how backpropagation works for a softmax/cross-entropy output layer. The cross entropy error function is $$E(t,o)=-\sum_j t_j \log o_j$$ with $t$ and $o$ as the target and ...
32
votes
4answers
39k views

How is the cost function from Logistic Regression derivated

I am doing the Machine Learning Stanford course on Coursera. In the chapter on Logistic Regression, the cost function is this: Then, it is derivated here: I tried getting the derivative of the cost ...
31
votes
1answer
12k views

Step-by-step example of reverse-mode automatic differentiation

Not sure if this question belongs here, but it's closely related to gradient methods in optimization, which seems to be on-topic here. Anyway, feel free to migrate if you think some other community ...
14
votes
3answers
428 views

How can I fit a spline to data that contains values and 1st/2nd derivatives?

I have a dataset that contains, let's say, some measurements for position, speed and acceleration. All come from the same "run". I could construct a linear system and fit a polynomial to all of those ...
12
votes
2answers
6k views

Derivative of a Gaussian Process

I believe that the derivative of a Gaussian process (GP) is a another GP, and so I would like to know if there are closed form equations for the prediction equations of the derivative of a GP? In ...
12
votes
1answer
1k views

Interpretation of Radon-Nikodym derivative between probability measures?

I have seen at some points the use of the Radon-Nikodym derivative of one probability measure with respect to another, most notably in the Kullback-Leibler divergence, where it is the derivative of ...
11
votes
1answer
860 views

Second order approximation of the loss function (Deep learning book, 7.33)

In Goodfellow's (2016) book on deep learning, he talked about equivalence of early stopping to L2 regularisation (https://www.deeplearningbook.org/contents/regularization.html page 247). Quadratic ...
11
votes
3answers
2k views

Can a neural network learn a functional, and its functional derivative?

I understand that neural networks (NNs) can be considered universal approximators to both functions and their derivatives, under certain assumptions (on both the network and the function to ...
10
votes
1answer
564 views

What justifies this calculation of the derivative of a matrix function?

In Andrew Ng's machine learning course, he uses this formula: $\nabla_A tr(ABA^TC) = CAB + C^TAB^T$ and he does a quick proof which is shown below: $\nabla_A tr(ABA^TC) \\ = \nabla_A tr(f(A)A^TC) \\...
10
votes
2answers
9k views

Equation of a fitted smooth spline and its analytical derivative [duplicate]

I need to fit a spline function to a data set. I tried with bs, ns and smooth.spline. In my ...
9
votes
1answer
10k views

Differentiation of Cross Entropy

I have been trying to create a program for training Neural Networks on my computer. For the Network in question, I have decided to use the Cross Entropy Error function: $$E = -\sum_jt_j\ln o_j$$ ...
9
votes
4answers
8k views

How to find derivative of softmax function for the purpose of gradient descent?

I'm trying to understand back propagation algorithm for multiclass classification using gradient descent. I'm using https://www.cs.toronto.edu/~graves/phd.pdf . The output layer is a softmax layer, in ...
9
votes
1answer
5k views

Gradient and vector derivatives: row or column vector?

Quite a lot of references (including wikipedia, and http://www.atmos.washington.edu/~dennis/MatrixCalculus.pdf and http://michael.orlitzky.com/articles/the_derivative_of_a_quadratic_form.php) define ...
8
votes
2answers
26k views

Derivative of Softmax with respect to weights

I'm new to deep learning and am attempting to calculate the derivative of the following function with respect to the matrix $\mathbf w$: $$p(a) = \frac{e^{w_a^\top x}}{\Sigma_{d} e^{w_d^\top x}}$$ ...
8
votes
2answers
46k views

How to get the derivative of a normal distribution w.r.t its parameters?

We normally calculate the derivative of normal density w.r.t its parameters, mean and variance. But can we calculate the derivative of normal distribution wrt the parameters(not the variable, I know ...
8
votes
4answers
185 views

Where does the logistic function come from?

I first learned the logistic function in machine learning course, where it is just a function that map a real number to 0 to 1. We can use calculus to get its derivative and use the derivative for ...
8
votes
1answer
7k views

How to compute the gradient and hessian of logarithmic loss? (question is based on a numpy example script from xgboost's github repository)

I would like to understand how the gradient and hessian of the logloss function are computed in an xgboost sample script. I've simplified the function to take numpy arrays, and generated ...
7
votes
3answers
631 views

Derivation of MLE of linear regression: and now? Why is there discrepancy to lm in R?

I want to understand the ML Estimation of the linear model from top to bottom or vice versa ;-). I totally get the part of formulating the LogLikelihood function and how to get the derivatives of beta ...
7
votes
1answer
6k views

Neural network softmax activation

I'm trying to perform backpropagation on a neural network using Softmax activation on the output layer and a cross-entropy cost function. Here are the steps I take: Calculate the error gradient with ...
6
votes
1answer
1k views

How To quickly do derivatives with respect to matrices

Whats a quick way to work the below (and problems similar) out? For me to take this derivative it involves a lot of time and boring calculation, there has to be a better way. This is taken from ...
6
votes
1answer
975 views

Derivation of Group Lasso

I've been reading the book Statistical Learning with Sparsity and I just came across the Group Lasso section. I can follow the maths to the final derivation of the Group Lasso solutions when the ${\...
5
votes
2answers
494 views

Derivative of a quadratic form wrt a parameter in the matrix

I want to compute the derivative of: $\frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}$, (Note that C is a covariance matrix that depends on a set of parameters $\theta$) for which I used ...
5
votes
1answer
238 views

If we have auto differentiate tool, do we also need EM algorithm?

I my opinion, EM algorithm is used to estimate the parameters of some complex log likelihood function. Because sometimes, it's hard to get the derivative, we can use EM algorithm. But if we have some ...
4
votes
2answers
577 views

If $f(x)$ is a unimodal probability density function, how can I show that its mode is at $f'(x)=0$?

If the function $f(x)$ is continuous and a probability density function (PDF), how can I show that its mode is at $f'(x)=0$?
4
votes
1answer
2k views

Computing gradients via Gaussian Process Regression

I have a set of noisy data that I am fitting using Gaussian Process Regression via Python's sklearn package. The posterior mean of the GP is essentially my output with an associated error. Based on ...
4
votes
3answers
1k views

Calculating the expression for the derivative of a Gaussian process

I know based on the answers to this question Derivative of a Gaussian Process that the derivative of a Gaussian process is another Gaussian process, but I was wondering if someone could tell (or show) ...
4
votes
1answer
7k views

Mean Absolute Error (MAE) derivative

$MAE=|y_{pred} - y_{true}|$ $\dfrac{dMAE}{dy_{pred}} = ?$ I'm trying to understand how MAE works as a loss function in neural networks using backpropogation. I know it can be used directly in some ...
4
votes
2answers
127 views

Slope of Curve with Unknown Functional Form

I have a monotonically-increasing curve whose functional form is not known a priori and would like to compute the curve's slope at the rightmost endpoint. Typically, when the functional form is known, ...
4
votes
1answer
732 views

How to obtain the functional derivative in variational inference?

Referring to David Blei's notes on variational inference, I wonder how to get the derivative of $q(z_k)$, the distribution of $z_k$, in eq. 23 from eq. 22. Namely, eq. 22 is $L_k = \int q(z_k) \...
4
votes
1answer
242 views

Second directional derivate and Hessian matrix

I am reading the following from the book Deep Learning, and I have the following questions. I don't quite understand second directional derivatives. The first directional derivative of a function $f:\...
4
votes
1answer
207 views

Question about variational autoencoder gradient

The following is from Section 2.2 of the Auto-Encoding Variational Bayes paper, It says the gradient of the lower bound w.r.t $\phi$ is a bit problematic because the Monte Carlo estimator exhibits ...
4
votes
1answer
619 views

Marginal Effects of Discrete Variables in Quantile Regression

I find myself puzzled by a passage about marginal effects of discrete variables in quantile regression. On p. 217 of Cameron and Trivedi's MUS book, the authors write: For the $j$th (continuous) ...
4
votes
1answer
40 views

Normal distribution “stable” under derivative?

Suppose that $\theta(t)\sim\mathcal N(\mu(t),\Sigma(t))$ where $t$ is some parameter. Then it holds that $$\theta(t) = \mu(t) + \Sigma(t)^{0.5}\xi$$ for $\xi\sim\mathcal N(0, I)$. I am interested in ...
4
votes
1answer
2k views

How to differentiate with respect to a matrix?

How can I differentiate the following by $\mathbf{W}$ ? \begin{equation} \mathbf{Y} = (\mathbf{W}^T\mathbf{x} + b)^2 \end{equation} Where $\mathbf{W} \in \mathcal{R}^{d\times D}$ and $\mathbf(...
4
votes
0answers
86 views

Incorrect computation in Knight and Fu (2000)?

I'm currently reading Knight and Fu's 2000 paper on the asymptotics of "Bridge" estimators with a particular focus on LASSO as a special case. In the proof of theorem 2, they make the claim that under ...
4
votes
0answers
452 views

How to calculate uncertainty in bacterial growth rates (or in the slope of any local regression)?

I'm using a plate reader to measure optical density of different bacterial strains so I can compare their responses (growth rates and changes in them over time) to stress conditions. The growth curves ...
3
votes
2answers
8k views

Derivative of softmax and squared error

I'm trying to understand the derivatives w.r.t. the softmax arguments when used in conjunction with a squared loss (for example as the last layer of a neural network). I am using the following ...
3
votes
2answers
471 views

A question in directional derivatives of a quantile regression object function

The question comes from the paper ``Regression Quantiles'' by Roger Koenker and Gilbert Bassett(Econometrica, 1978). $0< \theta <1$. Define $\psi(b;\theta,y,X)=\sum^{T}_{t=1}[\theta-1/2+1/2 \; ...
3
votes
2answers
497 views

Derivative of bivariate normal CDF with common mean parameters

I am trying to calculate a derivative of the form $\frac{d}{dz}\Phi_2(\mu_1(z),\mu_2(z),\rho)$, where $\Phi_2$ is the standard bivariate normal CDF. I am thinking it might be an application of a ...
3
votes
1answer
436 views

What is the second derivative of a B-spline?

A B-spline of degree $j$ is defined at knots $\vec k$ by the Cox-de Boor recursion formula \begin{align} B_{i,1}(x) &= \left\{ \begin{matrix} 1 & \mathrm{if} \quad ...
3
votes
2answers
76 views

When using OLS on $\ln(y) = \beta_1 \ln(x) + \epsilon$, is $\beta_1$ the elasticity of $E[y\vert x]$, or the $y$ in the data (or both)?

Specifically, suppose we are estimating $$ \ln(y)=\beta_1\ln(x) + \epsilon $$ I understand that $\beta_1 = \frac{\partial \ln(y)}{\partial \ln(x)}$ which is the elasticity of $y$ with respect to $x$ ...
3
votes
1answer
51 views

Correlation between the j-feature and partial residual

I am reading a book for the lasso-regression. However, I think that this is a more general question regarding the derivative of the OLS-term: Can someone explain me why this $c_j$ term is ...
3
votes
1answer
59 views

Derivation of score vector

Can anyone explain the process of this derivation, step by step? This derivation is from Joint Models for Longitudinal and Time-to Event Data by Dimitris Rizopoulos. \begin{equation} \begin{aligned} ...
3
votes
1answer
61 views

understanding derivatives of a regression spline

I am trying to understand why regression splines are continuous at their knots Suppose I am fitting a regression spline $$ E[Y|X] = \alpha + \beta_1 x + \beta_2 (x - t)^+ $$ where $(x - t)^+ = \...
3
votes
1answer
97 views

What is the derivative of $\|X^T-S^TAX^T\|_F^2$ w.r.t $A$?

What is the derivative of $F = \|X^T-S^TAX^T\|_F^2$ w.r.t $A$, where $X \in\mathbb R^{d \times N}$, $S \in\mathbb R^{k \times N}$, and $A \in \mathbb{R}^{k \times N}$? I have tried, and it is as ...
3
votes
1answer
64 views

PDF of mixture of random variables that are not necessarily independent

I am trying to derive the expression for the PDF of a weighted mixture of n random variables. Let us taken $n=3$ and define $$X = \alpha_1 S_1 + \alpha_2 S_2 + \alpha_3 S_3$$ $$E[X^2] = 1$$ $s_1$, $...
3
votes
1answer
225 views

Derivative of Gaussian Process (continued)

This is to extend the discussion of the derivative of the GP. The formulation provided in the previous post describes the gradient of GP as derivative of kernel function as follows with respect to $(x^...
3
votes
2answers
58 views

When are the linear regression parameters of Y and X the same as the parameters of Y' and X'?

I am working on some simple linear modeling of a physical system and assumed that taking the derivative of an equation $$Y = \beta_1 + \beta_2 X + \varepsilon$$ would give me $$\frac{dY}{dt} = \...
3
votes
1answer
235 views

Using derivative information to improve Gaussian Process regression

I do not exactly know whether the question fits SE stats criteria, if it doesn't let me know. I am looking for a approach which can be used to improve a GP regression esimate if the derivatives are ...
3
votes
0answers
13 views

Meaning of a notation regarding mean square derivative

I'm reading a paper (On Differentiable Functionals, Van der Vaart, 1991, Annals of Statistics), and I've got a question regarding a notation in the following part: My Question: Does $dP^{1/2}$ mean $...