Questions tagged [derivative]

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

Filter by
Sorted by
Tagged with
1 vote
1 answer
14 views

A question on computational complexity of a numerical differentiation (equation (5.77)) in Bishop's Pattern Recognition and Machine Learning

In page 249 of Christopher M. Bishop's book "Pattern Recognition and Machine Learning", it is said Again, the implementation of such algorithms can be checked by using numerical ...
  • 67
0 votes
0 answers
18 views

Derivation of (5.76) in "Pattern Recognition and Machine Learning"

The book "Pattern Recognition and Machine Learning" by Christopher M. Bishop says in page 248 ... for softmax outputs we have: $$\frac{\partial y_k}{\partial a_l}=\delta_{kl}y_k-y_ky_l.\tag{...
  • 67
1 vote
0 answers
22 views

I need to derivate the maximum likelihood of GLR function [closed]

I am trying to do the partial derivative of this function but I don't know how to include the $\theta$ inside my function to be able to apply the partial derivative in $\theta$. The likelihood ...
0 votes
0 answers
18 views

Second order moment of the Gaussian Distribution [duplicate]

In Bishops book, "Statistical Pattern Recognition", there is one exercise, which states to derive the second order moment of the Gaussian Distribution: $E[x^2] = \int_{-\infty}^{\infty} N(x|...
  • 73
2 votes
0 answers
19 views

Differentiation on the conditional variables of a probability

I have been questioning how to calculate the partial derivatives of a conditional probability function with respect to its parameters. Assume $x$ is data and $\theta$ is a parameter(s). If I have a ...
0 votes
0 answers
19 views

derivative for $-K\log(Z(W,\mathbf{a},\mathbf{b}))+ \sum_k(\mathbf{s^k})^T\mathbf{W}\mathbf{d}^k+\mathbf{a}^T\mathbf{s}^k+\mathbf{b}^T \mathbf{d}^k$?

This is a follow up from this question. Consider a model of diseases and symptoms. $s_i\in\{0,1\}$ is a binary random variable indicating whether the patient is showing the $i$-th symptom and $d_j\in ...
  • 71
1 vote
1 answer
24 views

Derivative of Trace of matrix product

I am trying to compute the gradient with respect to a vector $\mathrm x \in \mathbb{R}^d$ of a complicated expression involving the trace of matrix product. The expression is the following: $$ F(\...
0 votes
2 answers
25 views

Derivative of quadratic form of vector-valued function

This seems like a trivial question but I am currently stuck and cannot see what I am doing wrong. So let us consider a function $f(x) : \mathbb{R}^d \rightarrow \mathbb{R}^d$. I want to compute the ...
1 vote
1 answer
24 views

Best Error Function for Areas with Larger Slopes

I have some nonlinear data (let's say x's and y's) that I would like to perform regression on, and I would like to focus on having the error lower on regions where the graph is more sloped, rather ...
1 vote
1 answer
36 views

Deriving the expression for $p(\mathcal{K})$ where $\mathcal{K} = \{(\mathbf{s}^k,\mathbf{d}^k), k = 1,..., K\}$

This is a follow up from this question. Consider a model of diseases and symptoms. $s_i\in\{0,1\}$ is a binary random variable indicating whether the patient is showing the $i$-th symptom and $d_j\in ...
  • 71
0 votes
1 answer
41 views

Deriving the normal equations' coefficients

Suppose we use the least squares criterion to fit a linear model for the following dataset: $(x_1,y_1),...,(x_m,y_m)\in R \times R$, by solving the following optimisation problem: $$(a^*,b^*) = \text{...
0 votes
0 answers
9 views

How to find the partial derivative of a matrix trace [duplicate]

We need the partial of equation 3 with respect to w, how do we get from equation 3 to equation 4?
  • 1
0 votes
1 answer
24 views

How to compute the derivative of the total loss wrt external trainable parameters?

I was just curious how external trainable parameters are updated. The challenge is to compute the derivative, the rest is handled by the optimiser. I assumed a simple DNN as follows: $$\hat{y} =\sigma(...
1 vote
0 answers
37 views

What is the derivative of a matrix with regard to a vector defined?

I had this question when I read equation (C.20) in Appendix C of "Pattern Recognition and Machine Learning" written by Christopher M. Bishop. Here I copy the equation below for reference: ...
  • 67
1 vote
0 answers
16 views

Standard errors of Monte Carlo plus linear combination

I'm using Monte Carlo to estimate some quantity $V(x)$. To get an approximation of $V'(x)$ I would use the following $$ V'(x)\approx\frac{V(x+h)-V(x-h)}{2h} $$ so I can simply evaluate it with two ...
1 vote
1 answer
44 views

Mismatch between the dimensions of Jacobian matrixes when calculating derivatives during backprop?

I am trying to understand how back propagation works for a linear layer using minibatches by following this post: https://web.eecs.umich.edu/~justincj/teaching/eecs442/notes/linear-backprop.html. ...
  • 237
1 vote
0 answers
36 views

How does the full derivative of softmax + cross entropy have the correct dimensions?

The blog post the softmax function and its derivative explains the following: Imagine that each input has $N$ features / pixels / etc. Imagine each input can be classified into $C$ classes Let the ...
  • 237
0 votes
0 answers
16 views

Finding the derivative of the kSVM classifier funcion with respect to the weight vector?

I would like to try a different approach for defining the kSVM. However to do that at some point I need a derivative of the clasifier function $y(\textbf{x})$ with respect to the weights vector ...
  • 259
1 vote
0 answers
9 views

How does the fixed point interation in invertible resnets work?

I feel like I am missing some easy point about this invertible resnet paper which is making it hard for me to grasp how the fixed point iteration works. stated simply, the residual connection in a ...
  • 639
1 vote
1 answer
26 views

Comparative statics for conditional expectations

Let $f\left(x,y\right)\in\left[0,\frac{1}{2}\right)$ a function such that $\frac{\partial f}{\partial x}>0$, $\frac{\partial f}{\partial y}<0$, and $\frac{\partial^2 f}{\partial x \partial y}<...
1 vote
1 answer
77 views

XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: \begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation} \begin{equation}...
2 votes
1 answer
89 views

How to write the derivative of the inverse gamma function?

I have recently been writing an R program on the inverse of the gamma function and the derivative of the inverse function. Now there is some confusion I would like to ask for advice. I have written ...
  • 43
2 votes
1 answer
51 views

Derivatives of output w.r.t input on a neural network trained with standardized data

I'm using a neural network to model an unknown function for which I would also like to know the derivatives. The nn has four inputs and four outputs, and the training data is preprocessed using scikit-...
  • 25
0 votes
0 answers
25 views

Difference between analytic and numeric gradient in Matlab

I am using matlab to solve a optimization problem. When I check the anlaytic and numeric gradient reported by matlab, they are quite different. So I want to ask if there is a mistake in the analytic ...
  • 105
0 votes
0 answers
138 views

Difference between forward-mode and reverse-mode automatic differentiation?

I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ...
  • 2,041
0 votes
0 answers
29 views

Fisher Information of Weight in Mixture distribution

Let's assume $x$ follows a mixture of two arbitrary continuous probability distributions with probability density functions $p_1(x)$ and $p_2(x)$, respectively. The probability density function of $x$ ...
  • 812
0 votes
0 answers
20 views

How is derivative calculated for Grad-CAM if the final output is multidimensional?

For Grad-CAM, the derivative of the final output is found with respect to the elements of the channel considered Selvaraju et al. 2019. But if the output is a multidimensional matrix how is the ...
14 votes
3 answers
377 views

Proper regression for determining correlations between derivatives of functions

Say we have a permanent-magnet DC motor that roughly obeys the system equation $\ddot{x}(t) = \alpha \dot{x}(t) + \beta u(t) + \gamma$, where $x(t)$ is the displacement of the rotor, and $u(t)$ the ...
1 vote
1 answer
289 views

Why do the derivatives of a function lead towards the extremum of the function?

Is there some theorem in mathematics that formalizes the idea that "for some function, at a given point, moving in the negative direction of the gradient leads you to some (local) extremum point&...
  • 6,068
1 vote
0 answers
74 views

Derivative of a probability

If $Y$ is a discrete random variable, and I define $F(x)=P(Y \leq x),$ where $x \in \mathbb{R},$ can I differentiate $F(x)$?
  • 21
0 votes
0 answers
36 views

Deriving initial weights for IRLS in DESeq2's GLM model

DeSEQ2 is a frequently-used R package for researchers studying differential gene expression via changes in molecular markers such as Poly(A) RNA. Understanding how ...
  • 4,792
1 vote
0 answers
14 views

What does the SVD of the numerical derivative matrix of paired instances tell us about our data? [closed]

Let's say we have real-valued random variables $X$ and $Y$, and that under simple random sampling we obtain paired values $\{(x_1,y_1), \cdots, (x_i,y_i), \cdots, (x_m,y_m) \}$. From this sample we ...
  • 4,792
6 votes
2 answers
616 views

matrix-calculus - Understanding numerator/denominator layouts

Consider the following machine-learning model: Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of training-examples. While performing reverse-mode ...
  • 206
1 vote
1 answer
350 views

reverse sigmoid and its derivative

I wonder, if someone could please check/help me with this simple code: ...
  • 1,346
1 vote
0 answers
50 views

Minimizing Expectation [closed]

I am not entirely sure how the derivative follows from the preceding line in this example. $f(x)$ is a PDF. You are supposed to set the derivative to 0 as the expectation needs to be minimised. ...
2 votes
1 answer
110 views

derivative of the error w.r.t parameters

let's assume my function is as simple as $y = xW + b$ We define an error function as $E = {\frac{(t - y)}{2}}^2$ I wonder if you can help me to write the derivative of the error w.r.t parameters (W ...
  • 1,108
1 vote
1 answer
42 views

Inverse of a noisy derivative

I have a series of samples (x(t), y(t)), where both are noisy and with (assumed) iid errors (sx(t), sy(t)). I need to measure a ...
1 vote
1 answer
789 views

How to calculate derivative of cross entropy loss function?

I have a cross entropy loss function. $$ L = -{1 \over N} \sum_i {y_i \cdot \log {1 \over {1+e^{-\vec x \cdot \vec w}}} + (1-y_i) \cdot \log (1-{1 \over {1+e^{-\vec x \cdot \vec w}}})} $$ I want to ...
  • 113
0 votes
0 answers
20 views

Are numerical solutions appropriate for inference (eg, estimating variance for confidence intervals)?

For a nicely differentiable objective function, we traditionally always derive the gradients to use for e.g. estimating the variance. (1) Is it common nowadays to use numerical rather than analytical ...
  • 5,356
0 votes
1 answer
39 views

Interpretation of regression coefficient of logged variable (log X)

I am struggling to see why a one percent change in $X$ is associated with a $\frac{\beta_1}{100}$ change in $Y$ in the following model: $Y = \beta_0 + \beta_1 \ln X + \beta_2 W + ... + u$. It is clear ...
1 vote
1 answer
67 views

Derivative of $\nabla_{\theta} f(x, \theta) f(x, \theta)$ (the gradient of the function times the function itself)

I am having troubles computing the derivative of $\nabla_{\theta}f(x, \theta)f(x, \theta) $ (the gradient of the function $f(x, \theta)$ times the function itself) that is \begin{align} D(\nabla_{\...
2 votes
1 answer
18 views

While deriving Least Squares Estimators, how to find the derivate of a summation operate?

I'm calculating the Least Squares Estimators. There was one step here: $\frac{d}{d\hat\alpha}{\sum(y_i-\hat\alpha-\hat\beta x_i)}^2=0$ --> $-2{\sum(y_i-\hat\alpha-\hat\beta x_i)}=0$ I know it is ...
  • 21
1 vote
1 answer
241 views

Deriving the PDF of the kth order statistic from the CDF

I am trying to understand how to get from the CDF to the PDF of the kth order statistic and I am following this article. I understand that I have to take the derivative of F to get f. I also ...
  • 352
1 vote
0 answers
57 views

Is there a general closed-form formula of the derivates of a feedforward network?

I am looking for a general closed-form formula for the derivatives of a Feed-forward Network with respect to the inputs. Mathematically, we can write: $$ \mathbf{y} = f_{FF}(\mathbf{x}) = \mathbf{W}_{...
  • 31
0 votes
0 answers
58 views

Best value of x for derivative estimated by linear regression

I am currently evaluating the measurement of the thermal impedance of different semiconductor devices. In order to properly evaluate these measurements, I have to determine the derivative of the ...
3 votes
1 answer
61 views

How does the approximate Hessian update in LBFGS work?

Looking at the wikipedia page for BFGS... Wikipedia It looks like a rearranging of Newton's method, but I can't really explain why the update to the approximate Hessian would be given by the following ...
  • 639
2 votes
1 answer
52 views

Differentiating a Vector and a Matrix w.r.t. a Vector [Matrix Calculus]

I am studying matrix calculus for linear regression and machine learning and I would like to know exactly if the following calculations are correct: Let $y=\sin(x+yz)$ and $r=\begin{bmatrix}x\\y\\z\...
1 vote
0 answers
19 views

How can I find the uncertainty of derivatives? [duplicate]

Suppose I have a quadratic (weighted) least-square fit result obtained from a given set of data: $$ f(x) = \underbrace{-0.243(\pm0.3324)}_{quad\_a}x^2\underbrace{-0.921(\pm0.061)}_{quad\_b}x\...
  • 111
1 vote
0 answers
48 views

Given a mean, what is the range of variance values that make for possible Beta distribution parameters

The beta distribution can have its parameter estimated via method of moments, which I will be doing. $$\hat\alpha = \bigg(\dfrac{\bar x (1-\bar x)}{var(X)} - 1\bigg)\bar{x}\\ \hat\beta = \bigg(\dfrac{\...
  • 35.7k
2 votes
1 answer
69 views

What is $ \frac{d \ E(ln(y)|X)}{d \ y}$ in OLS?

Assume that the true model (DGP) is $ ln(y) \ = \ \beta_0 \ + \ \beta_1 ln(x_1) \ + \ \cdots \ + \ \beta_k x_k \ + \ \varepsilon \hspace{3em} \text{where } \ \begin{bmatrix} x_1\\ \vdots\\ x_k\\ \...
  • 173

1
2 3 4 5 6