# Questions tagged [derivative]

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

243 questions
Filter by
Sorted by
Tagged with
332 views

### matrix-calculus - Understanding numerator/denominator layouts

Consider the following machine-learning model: Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of training-examples. While performing reverse-mode ...
23 views

### reverse sigmoid and its derivative

I wonder, if someone could please check/help me with this simple code: ...
35 views

### Minimizing Expectation [closed]

I am not entirely sure how the derivative follows from the preceding line in this example. $f(x)$ is a PDF. You are supposed to set the derivative to 0 as the expectation needs to be minimised. ...
21 views

### derivative of the error w.r.t parameters

let's assume my function is as simple as $y = xW + b$ We define an error function as $E = {\frac{(t - y)}{2}}^2$ I wonder if you can help me to write the derivative of the error w.r.t parameters (W ...
31 views

### Inverse of a noisy derivative

I have a series of samples (x(t), y(t)), where both are noisy and with (assumed) iid errors (sx(t), sy(t)). I need to measure a ...
38 views

### How to calculate derivative of cross entropy loss function?

I have a cross entropy loss function. $$L = -{1 \over N} \sum_i {y_i \cdot \log {1 \over {1+e^{-\vec x \cdot \vec w}}} + (1-y_i) \cdot \log (1-{1 \over {1+e^{-\vec x \cdot \vec w}}})}$$ I want to ...
17 views

### Are numerical solutions appropriate for inference (eg, estimating variance for confidence intervals)?

For a nicely differentiable objective function, we traditionally always derive the gradients to use for e.g. estimating the variance. (1) Is it common nowadays to use numerical rather than analytical ...
29 views

### Interpretation of regression coefficient of logged variable (log X)

I am struggling to see why a one percent change in $X$ is associated with a $\frac{\beta_1}{100}$ change in $Y$ in the following model: $Y = \beta_0 + \beta_1 \ln X + \beta_2 W + ... + u$. It is clear ...
44 views

### Derivative of $\nabla_{\theta} f(x, \theta) f(x, \theta)$ (the gradient of the function times the function itself)

I am having troubles computing the derivative of $\nabla_{\theta}f(x, \theta)f(x, \theta)$ (the gradient of the function $f(x, \theta)$ times the function itself) that is \begin{align} D(\nabla_{\...
13 views

### While deriving Least Squares Estimators, how to find the derivate of a summation operate?

I'm calculating the Least Squares Estimators. There was one step here: $\frac{d}{d\hat\alpha}{\sum(y_i-\hat\alpha-\hat\beta x_i)}^2=0$ --> $-2{\sum(y_i-\hat\alpha-\hat\beta x_i)}=0$ I know it is ...
50 views

### Deriving the PDF of the kth order statistic from the CDF

I am trying to understand how to get from the CDF to the PDF of the kth order statistic and I am following this article. I understand that I have to take the derivative of F to get f. I also ...
33 views

40 views

37 views

333 views

### What is the Hessian of the Gaussian likelihood

I am trying to learn the fine differences between different methods of Kronecker factoring for approximate curvature (like , and ) which require taking the Hessian of the pre-activations of the ...
34 views

602 views

### Derivation of Hessian for multinomial logistic regression in Böhning (1992)

This question is basically about row/column notation of derivatives and some basic rules. However, I couldn't figure out where I'm wrong. For multinomial logistic regression, I'm trying to get the ...
25 views

### derivative of Matrix normal distribution function

I am going to find the MLE for matrix normal distribution for my mixture model. I did parts of EM algorithm for that. however I am stuck in algebraic part of M step. I owuld appreciate any help with ...
41 views

### Matrix dimensions of Derivative of Softmax on Vectorized version

I am trying to get the matrix dimensions right for computing derivative of a two layers network where the last layer is softmax function. For simplicity I am only interested to get derivatives of W2 w....
27 views

### Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more?

I understand the concept of backpropagation in standard neural networks and backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of ...
95 views

### Help with derivation of gradient for specific filter in CNN

I need help with . I have to compute gradient for the special type of filter in CNN and everytime it comes up to be 0. Either this si correct, or I have some fundamental problem there, any hints are ...
73 views

21 views

### Estimating Logit Interaction Coefficients: the Second cross derivative

I have been working on a mixed effects statistical model that proposes a three-way interaction of continuous predictors on a binary variable in lme4. The general outline of the code is as follows: <...
24 views

### Computing the Jacobian $J_F$ with $F = h \circ f$

Let $$f: \mathbb{R}^l \rightarrow{} \mathbb{R}^m\\[.7ex] h: \mathbb{R}^m \rightarrow{} \mathbb{R}^o$$ and let $$F = h \circ f \quad (F : \mathbb{R}^l \rightarrow{} \mathbb{R}^o)$$ I want to compute ...
292 views

### Comparing gam derivatives with gratia

I'm using gratia (thanks so much Gavin Simpson for that excellent library) to find the first derivative on time series. I cannot post the data here, but I have created a minimal example to illustrate ...
81 views

### Coding gradient descent from scratch - how are fitness functions incorporated into output layer error calculation?

For a project I am currently working on, I'm attempting to implement machine learning for a neural network using backpropagation and gradient descent from scratch. For much of my implementation, I ...
58 views

### Partial derivative of multivariate cdf with respect to coefficients

I want to take the partial derivative of this multivariate gaussian cumulative distribution function with respect to $\beta_1$ (which is a single element of the $\beta$ vector). $X_1$ is a n $\times$ ...
102 views

### How to implement LSTM backpropagation through time?

I'm building a custom LSTM net based on this article. I got questions on how to implement the backpropagation, based on these formulas of the derivatives in an LSTM layer: Question 1: The weights (w.....
50 views

### Best way to deal with non-continuously differentiable error functions in machine learning

Suppose you have the following set of $n=9$ numbers x = [1, 2, 3, 4, 5, 6, 7, 8, 9] and the corresponding y values [10, 2.5, 1.1, 0.6, 0.4, 0.2, 0.15, 0.12] which visualized look like this: I'd like ...
How do you take the derivative of the function $$s(\beta)=\displaystyle\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i?$$ Attempt: $$H(\beta)=\frac\partial{\partial\... 0answers 26 views ### ReLU Neuron Derivative For x \in \mathbb R^d, we define:$$ \text{ReLU}(x) := \max{(x, 0)} \\ \text{Step}(x) := \mathbb{1}[x \ge 0] $$And ‘a neuron of the activation function \psi’ as f_{\psi,w,b}(x) := \psi(w \cdot x ... 0answers 151 views ### For multivariate linear regression, what is the partial derivative for Mean Absolute Error? What is the partial derivative for MAE for multivariate linear regression? I understand that for mean squared error (MSE) the partial derivative with respect to some \theta_1 would be -\theta_1 \... 0answers 74 views ### Canonical LSTM backpropagation equations I'm trying to understand the underlying mechanisms of LSTM from a programming perspective. I am no math person, and a lot of articles and papers look like alphabet soup to me. But I thought that if I ... 0answers 32 views ### How do I take the derivative to \gamma  of (y-x^{\gamma})^T(y-x^{\gamma})? I have to solve the least squares for \gamma in the following problem. The model is described as y_i = \beta x_i^{\gamma} + u_i, where u_i  is i.i.d. normal with mean zero and variance \sigma^2... 1answer 139 views ### How to derivate the following loss function? How can I derivate the following optimization function?$$L=\sum_{u,i}(y_{u,i}-v_ix_u)^2+\lambda\left(\sum_i\|v_i\|_2^2+\sum_u\|x_u\|_2^2\right)$$I just want to get the equations of the gradient ... 1answer 265 views ### How to calculate and interpret a marginal treatment effect (local instrumental variable)? (Intuition through simple example.) I am working on the intuition behind local instrumental variables (LIV), also known as the marginal treatment effect (MTE), developed by Heckman & Vytlacil. I have worked some time on this and ... 1answer 78 views ### Deriving Logit Maximum Likelihood Estimator According to Verbeek, we can obtain the logit model by simplifying the first order condition of the log-likelihood function. Where,$$logL(\beta) = \Sigma^N_{i=1} y_i logF(x^{'}_i\beta)+ \Sigma^N_{i=...
Suppose I have the following function, representing a linear model with an interaction term: $$f(x, y) = \beta_{1} x + \beta_{2} y + \beta_{3} xy.$$ Now I want to see how the function changes if ...