Questions tagged [derivative]

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

Filter by
Sorted by
Tagged with
1 vote
1 answer
19 views

Comparative statics for conditional expectations

Let $f\left(x,y\right)\in\left[0,\frac{1}{2}\right)$ a function such that $\frac{\partial f}{\partial x}>0$, $\frac{\partial f}{\partial y}<0$, and $\frac{\partial^2 f}{\partial x \partial y}<...
user avatar
1 vote
1 answer
35 views

XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: \begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation} \begin{equation}...
user avatar
1 vote
0 answers
21 views

Help for understanding Leibniz Integral Rule [migrated]

Why is it that according to Leibniz Integral Rule, we cannot take derivative inside integration when both derivative and integration are over the same variable? For example: Let's say we have $f(x,t)$....
user avatar
  • 61
1 vote
1 answer
52 views

How to write the derivative of the inverse gamma function?

I have recently been writing an R program on the inverse of the gamma function and the derivative of the inverse function. Now there is some confusion I would like to ask for advice. I have written ...
user avatar
  • 13
2 votes
1 answer
24 views

Derivatives of output w.r.t input on a neural network trained with standardized data

I'm using a neural network to model an unknown function for which I would also like to know the derivatives. The nn has four inputs and four outputs, and the training data is preprocessed using scikit-...
user avatar
  • 25
0 votes
0 answers
16 views

Difference between analytic and numeric gradient in Matlab

I am using matlab to solve a optimization problem. When I check the anlaytic and numeric gradient reported by matlab, they are quite different. So I want to ask if there is a mistake in the analytic ...
user avatar
  • 105
0 votes
0 answers
37 views

Difference between forward-mode and reverse-mode automatic differentiation?

I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ...
user avatar
  • 2,031
0 votes
0 answers
20 views

Fisher Information of Weight in Mixture distribution

Let's assume $x$ follows a mixture of two arbitrary continuous probability distributions with probability density functions $p_1(x)$ and $p_2(x)$, respectively. The probability density function of $x$ ...
user avatar
  • 762
0 votes
0 answers
12 views

How is derivative calculated for Grad-CAM if the final output is multidimensional?

For Grad-CAM, the derivative of the final output is found with respect to the elements of the channel considered Selvaraju et al. 2019. But if the output is a multidimensional matrix how is the ...
user avatar
14 votes
3 answers
370 views

Proper regression for determining correlations between derivatives of functions

Say we have a permanent-magnet DC motor that roughly obeys the system equation $\ddot{x}(t) = \alpha \dot{x}(t) + \beta u(t) + \gamma$, where $x(t)$ is the displacement of the rotor, and $u(t)$ the ...
user avatar
2 votes
1 answer
262 views

Why do the derivatives of a function lead towards the extremum of the function?

Is there some theorem in mathematics that formalizes the idea that "for some function, at a given point, moving in the negative direction of the gradient leads you to some (local) extremum point&...
user avatar
  • 5,750
1 vote
0 answers
68 views

Derivative of a probability

If $Y$ is a discrete random variable, and I define $F(x)=P(Y \leq x),$ where $x \in \mathbb{R},$ can I differentiate $F(x)$?
user avatar
  • 21
0 votes
0 answers
30 views

Deriving initial weights for IRLS in DESeq2's GLM model

DeSEQ2 is a frequently-used R package for researchers studying differential gene expression via changes in molecular markers such as Poly(A) RNA. Understanding how ...
user avatar
  • 3,403
1 vote
0 answers
13 views

What does the SVD of the numerical derivative matrix of paired instances tell us about our data? [closed]

Let's say we have real-valued random variables $X$ and $Y$, and that under simple random sampling we obtain paired values $\{(x_1,y_1), \cdots, (x_i,y_i), \cdots, (x_m,y_m) \}$. From this sample we ...
user avatar
  • 3,403
6 votes
2 answers
507 views

matrix-calculus - Understanding numerator/denominator layouts

Consider the following machine-learning model: Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of training-examples. While performing reverse-mode ...
user avatar
  • 206
1 vote
1 answer
91 views

reverse sigmoid and its derivative

I wonder, if someone could please check/help me with this simple code: ...
user avatar
  • 1,306
1 vote
0 answers
43 views

Minimizing Expectation [closed]

I am not entirely sure how the derivative follows from the preceding line in this example. $f(x)$ is a PDF. You are supposed to set the derivative to 0 as the expectation needs to be minimised. ...
user avatar
1 vote
1 answer
52 views

derivative of the error w.r.t parameters

let's assume my function is as simple as $y = xW + b$ We define an error function as $E = {\frac{(t - y)}{2}}^2$ I wonder if you can help me to write the derivative of the error w.r.t parameters (W ...
user avatar
  • 1,068
1 vote
1 answer
33 views

Inverse of a noisy derivative

I have a series of samples (x(t), y(t)), where both are noisy and with (assumed) iid errors (sx(t), sy(t)). I need to measure a ...
user avatar
1 vote
1 answer
460 views

How to calculate derivative of cross entropy loss function?

I have a cross entropy loss function. $$ L = -{1 \over N} \sum_i {y_i \cdot \log {1 \over {1+e^{-\vec x \cdot \vec w}}} + (1-y_i) \cdot \log (1-{1 \over {1+e^{-\vec x \cdot \vec w}}})} $$ I want to ...
user avatar
  • 113
0 votes
0 answers
20 views

Are numerical solutions appropriate for inference (eg, estimating variance for confidence intervals)?

For a nicely differentiable objective function, we traditionally always derive the gradients to use for e.g. estimating the variance. (1) Is it common nowadays to use numerical rather than analytical ...
user avatar
  • 5,176
0 votes
1 answer
32 views

Interpretation of regression coefficient of logged variable (log X)

I am struggling to see why a one percent change in $X$ is associated with a $\frac{\beta_1}{100}$ change in $Y$ in the following model: $Y = \beta_0 + \beta_1 \ln X + \beta_2 W + ... + u$. It is clear ...
user avatar
1 vote
1 answer
54 views

Derivative of $\nabla_{\theta} f(x, \theta) f(x, \theta)$ (the gradient of the function times the function itself)

I am having troubles computing the derivative of $\nabla_{\theta}f(x, \theta)f(x, \theta) $ (the gradient of the function $f(x, \theta)$ times the function itself) that is \begin{align} D(\nabla_{\...
user avatar
2 votes
1 answer
15 views

While deriving Least Squares Estimators, how to find the derivate of a summation operate?

I'm calculating the Least Squares Estimators. There was one step here: $\frac{d}{d\hat\alpha}{\sum(y_i-\hat\alpha-\hat\beta x_i)}^2=0$ --> $-2{\sum(y_i-\hat\alpha-\hat\beta x_i)}=0$ I know it is ...
user avatar
  • 21
1 vote
1 answer
121 views

Deriving the PDF of the kth order statistic from the CDF

I am trying to understand how to get from the CDF to the PDF of the kth order statistic and I am following this article. I understand that I have to take the derivative of F to get f. I also ...
user avatar
  • 352
1 vote
0 answers
39 views

Is there a general closed-form formula of the derivates of a feedforward network?

I am looking for a general closed-form formula for the derivatives of a Feed-forward Network with respect to the inputs. Mathematically, we can write: $$ \mathbf{y} = f_{FF}(\mathbf{x}) = \mathbf{W}_{...
user avatar
  • 31
0 votes
0 answers
57 views

Best value of x for derivative estimated by linear regression

I am currently evaluating the measurement of the thermal impedance of different semiconductor devices. In order to properly evaluate these measurements, I have to determine the derivative of the ...
user avatar
3 votes
1 answer
38 views

How does the approximate Hessian update in LBFGS work?

Looking at the wikipedia page for BFGS... Wikipedia It looks like a rearranging of Newton's method, but I can't really explain why the update to the approximate Hessian would be given by the following ...
user avatar
  • 619
2 votes
1 answer
38 views

Differentiating a Vector and a Matrix w.r.t. a Vector [Matrix Calculus]

I am studying matrix calculus for linear regression and machine learning and I would like to know exactly if the following calculations are correct: Let $y=\sin(x+yz)$ and $r=\begin{bmatrix}x\\y\\z\...
user avatar
0 votes
0 answers
73 views

How to get the derivative of the cdf of a gamma distribution w.r.t its parameters?

I know the derivative of cdf of the gamma distribution w.r.t the variable is the pdf of the gamma distribution. But I want to find the derivative of cdf of the gamma distribution w.r.t the scale and ...
user avatar
  • 1
1 vote
0 answers
19 views

How can I find the uncertainty of derivatives? [duplicate]

Suppose I have a quadratic (weighted) least-square fit result obtained from a given set of data: $$ f(x) = \underbrace{-0.243(\pm0.3324)}_{quad\_a}x^2\underbrace{-0.921(\pm0.061)}_{quad\_b}x\...
user avatar
  • 111
1 vote
0 answers
41 views

Given a mean, what is the range of variance values that make for possible Beta distribution parameters

The beta distribution can have its parameter estimated via method of moments, which I will be doing. $$\hat\alpha = \bigg(\dfrac{\bar x (1-\bar x)}{var(X)} - 1\bigg)\bar{x}\\ \hat\beta = \bigg(\dfrac{\...
user avatar
  • 31.1k
2 votes
1 answer
67 views

What is $ \frac{d \ E(ln(y)|X)}{d \ y}$ in OLS?

Assume that the true model (DGP) is $ ln(y) \ = \ \beta_0 \ + \ \beta_1 ln(x_1) \ + \ \cdots \ + \ \beta_k x_k \ + \ \varepsilon \hspace{3em} \text{where } \ \begin{bmatrix} x_1\\ \vdots\\ x_k\\ \...
user avatar
  • 173
5 votes
1 answer
122 views

Gradient of the log likelihood for energy based models

Coming from this recent paper Your Classifier is Secretly an Energy Based Model And You Should Treat it Like One, they give the following definition... $$ p_\theta(\mathbf{x}) = \frac{\exp(-E_\theta(\...
user avatar
  • 619
1 vote
1 answer
55 views

How to prove the monotonicity of Softmax when decreasing the weighted inputs?

Softmax function $a^L_j = z^L_j / sum_k(z^L_k)$. When we think about the monotonicity of the Softmax function, $∂a^L_j/∂z^L_k$ is positive if j=k, and negative if j≠k. As a consequence, increasing $z^...
user avatar
  • 79
1 vote
0 answers
16 views

Directional derivative in regression (coefficients, after all, are partial derivatives)

The coefficients in a (let's stick with linear for now) regression are the partial derivatives. A regression equation is a function of several variables, so all of the multivariable calculus tricks ...
user avatar
  • 31.1k
1 vote
1 answer
58 views

Derivative of a Bivariate normal CDF with respect to its variables

Following up on the question (and answers) here, I'm trying to derive $\frac{\partial \Phi(x_1, x_2|\mathbf{\underline{\theta}})}{\partial x_1}$ and $\frac{\partial \Phi(x_1, x_2|\mathbf{\underline{\...
user avatar
  • 124
1 vote
1 answer
892 views

What is the Hessian of the Gaussian likelihood

I am trying to learn the fine differences between different methods of Kronecker factoring for approximate curvature (like [1], and [2]) which require taking the Hessian of the pre-activations of the ...
user avatar
  • 619
0 votes
0 answers
68 views

Why is the Hessian in the Laplace approximation negative

The Laplace approximation builds from the Taylor expansion of the MAP estimate, where the first derivative is 0. The second order Taylor series goes... $$ f(a) + \frac{f'(a)}{1!}(x - a) + \frac{f''(a)}...
user avatar
  • 619
3 votes
1 answer
31 views

How to know the number of dimensions of a Jacobian?

My question comes from a comment in this question Vector Jacobian product in automatic differentiation The question states... $$ t = Wz, \,\,\, z\in \mathbb{R}^{m\times 1}, t \in \mathbb{R}^{n \times ...
user avatar
  • 619
2 votes
1 answer
2k views

Derivation of Hessian for multinomial logistic regression in Böhning (1992)

This question is basically about row/column notation of derivatives and some basic rules. However, I couldn't figure out where I'm wrong. For multinomial logistic regression, I'm trying to get the ...
user avatar
  • 351
1 vote
0 answers
74 views

Matrix dimensions of Derivative of Softmax on Vectorized version

I am trying to get the matrix dimensions right for computing derivative of a two layers network where the last layer is softmax function. For simplicity I am only interested to get derivatives of W2 w....
user avatar
  • 133
0 votes
0 answers
50 views

Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more?

I understand the concept of backpropagation in standard neural networks and backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of ...
user avatar
0 votes
2 answers
102 views

Meaning of θj in equation for partial derivative of MSE

The equation to find the partial derivative of a cost function with respect to a parameter θj is given in the book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: $$ \frac{\partial}...
user avatar
  • 105
0 votes
1 answer
43 views

Meaning of a varaible for calculating the partial derviative of MSE cost function

The equation to find the partial derivative of a cost function with respect to a parameter θj is given in the book 'Hands on Machine Learning with scikit-learn, ...
user avatar
  • 105
3 votes
1 answer
170 views

Difficulties in computing the derivatives of the Dirichlet distribution

I need to compute the first derivatives of the Dirichlet distribution, defined in the following way: $$r(P; \pi, \rho) = \frac{\Gamma(c)}{\prod_{i=1}^{k} \Gamma(c \pi_i)} \cdot \prod_{i=1}^{k} P_i^{c\...
user avatar
  • 57
1 vote
0 answers
23 views

Estimating Logit Interaction Coefficients: the Second cross derivative

I have been working on a mixed effects statistical model that proposes a three-way interaction of continuous predictors on a binary variable in lme4. The general outline of the code is as follows: <...
user avatar
0 votes
0 answers
24 views

Computing the Jacobian $J_F$ with $F = h \circ f$

Let $$ f: \mathbb{R}^l \rightarrow{} \mathbb{R}^m\\[.7ex] h: \mathbb{R}^m \rightarrow{} \mathbb{R}^o$$ and let $$F = h \circ f \quad (F : \mathbb{R}^l \rightarrow{} \mathbb{R}^o)$$ I want to compute ...
user avatar
  • 143
0 votes
1 answer
187 views

How to implement LSTM backpropagation through time?

I'm building a custom LSTM net based on this article. I got questions on how to implement the backpropagation, based on these formulas of the derivatives in an LSTM layer: Question 1: The weights (w.....
user avatar
0 votes
0 answers
90 views

Best way to deal with non-continuously differentiable error functions in machine learning

Suppose you have the following set of $n=9$ numbers x = [1, 2, 3, 4, 5, 6, 7, 8, 9] and the corresponding y values [10, 2.5, 1.1, 0.6, 0.4, 0.2, 0.15, 0.12] which visualized look like this: I'd like ...
user avatar

1
2 3 4 5 6