Questions tagged [derivative]

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

Filter by
Sorted by
Tagged with
5
votes
2answers
332 views

matrix-calculus - Understanding numerator/denominator layouts

Consider the following machine-learning model: Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of training-examples. While performing reverse-mode ...
1
vote
1answer
23 views

reverse sigmoid and its derivative

I wonder, if someone could please check/help me with this simple code: ...
1
vote
0answers
35 views

Minimizing Expectation [closed]

I am not entirely sure how the derivative follows from the preceding line in this example. $f(x)$ is a PDF. You are supposed to set the derivative to 0 as the expectation needs to be minimised. ...
1
vote
1answer
21 views

derivative of the error w.r.t parameters

let's assume my function is as simple as $y = xW + b$ We define an error function as $E = {\frac{(t - y)}{2}}^2$ I wonder if you can help me to write the derivative of the error w.r.t parameters (W ...
1
vote
1answer
31 views

Inverse of a noisy derivative

I have a series of samples (x(t), y(t)), where both are noisy and with (assumed) iid errors (sx(t), sy(t)). I need to measure a ...
1
vote
1answer
38 views

How to calculate derivative of cross entropy loss function?

I have a cross entropy loss function. $$ L = -{1 \over N} \sum_i {y_i \cdot \log {1 \over {1+e^{-\vec x \cdot \vec w}}} + (1-y_i) \cdot \log (1-{1 \over {1+e^{-\vec x \cdot \vec w}}})} $$ I want to ...
0
votes
0answers
17 views

Are numerical solutions appropriate for inference (eg, estimating variance for confidence intervals)?

For a nicely differentiable objective function, we traditionally always derive the gradients to use for e.g. estimating the variance. (1) Is it common nowadays to use numerical rather than analytical ...
0
votes
1answer
29 views

Interpretation of regression coefficient of logged variable (log X)

I am struggling to see why a one percent change in $X$ is associated with a $\frac{\beta_1}{100}$ change in $Y$ in the following model: $Y = \beta_0 + \beta_1 \ln X + \beta_2 W + ... + u$. It is clear ...
2
votes
1answer
44 views

Derivative of $\nabla_{\theta} f(x, \theta) f(x, \theta)$ (the gradient of the function times the function itself)

I am having troubles computing the derivative of $\nabla_{\theta}f(x, \theta)f(x, \theta) $ (the gradient of the function $f(x, \theta)$ times the function itself) that is \begin{align} D(\nabla_{\...
2
votes
1answer
13 views

While deriving Least Squares Estimators, how to find the derivate of a summation operate?

I'm calculating the Least Squares Estimators. There was one step here: $\frac{d}{d\hat\alpha}{\sum(y_i-\hat\alpha-\hat\beta x_i)}^2=0$ --> $-2{\sum(y_i-\hat\alpha-\hat\beta x_i)}=0$ I know it is ...
1
vote
1answer
50 views

Deriving the PDF of the kth order statistic from the CDF

I am trying to understand how to get from the CDF to the PDF of the kth order statistic and I am following this article. I understand that I have to take the derivative of F to get f. I also ...
1
vote
0answers
33 views

Is there a general closed-form formula of the derivates of a feedforward network?

I am looking for a general closed-form formula for the derivatives of a Feed-forward Network with respect to the inputs. Mathematically, we can write: $$ \mathbf{y} = f_{FF}(\mathbf{x}) = \mathbf{W}_{...
0
votes
0answers
55 views

Best value of x for derivative estimated by linear regression

I am currently evaluating the measurement of the thermal impedance of different semiconductor devices. In order to properly evaluate these measurements, I have to determine the derivative of the ...
3
votes
1answer
22 views

How does the approximate Hessian update in LBFGS work?

Looking at the wikipedia page for BFGS... Wikipedia It looks like a rearranging of Newton's method, but I can't really explain why the update to the approximate Hessian would be given by the following ...
2
votes
1answer
31 views

Differentiating a Vector and a Matrix w.r.t. a Vector [Matrix Calculus]

I am studying matrix calculus for linear regression and machine learning and I would like to know exactly if the following calculations are correct: Let $y=\sin(x+yz)$ and $r=\begin{bmatrix}x\\y\\z\...
0
votes
0answers
53 views

How to get the derivative of the cdf of a gamma distribution w.r.t its parameters?

I know the derivative of cdf of the gamma distribution w.r.t the variable is the pdf of the gamma distribution. But I want to find the derivative of cdf of the gamma distribution w.r.t the scale and ...
1
vote
0answers
19 views

How can I find the uncertainty of derivatives? [duplicate]

Suppose I have a quadratic (weighted) least-square fit result obtained from a given set of data: $$ f(x) = \underbrace{-0.243(\pm0.3324)}_{quad\_a}x^2\underbrace{-0.921(\pm0.061)}_{quad\_b}x\...
1
vote
0answers
40 views

Given a mean, what is the range of variance values that make for possible Beta distribution parameters

The beta distribution can have its parameter estimated via method of moments, which I will be doing. $$\hat\alpha = \bigg(\dfrac{\bar x (1-\bar x)}{var(X)} - 1\bigg)\bar{x}\\ \hat\beta = \bigg(\dfrac{\...
2
votes
1answer
65 views

What is $ \frac{d \ E(ln(y)|X)}{d \ y}$ in OLS?

Assume that the true model (DGP) is $ ln(y) \ = \ \beta_0 \ + \ \beta_1 ln(x_1) \ + \ \cdots \ + \ \beta_k x_k \ + \ \varepsilon \hspace{3em} \text{where } \ \begin{bmatrix} x_1\\ \vdots\\ x_k\\ \...
5
votes
1answer
106 views

Gradient of the log likelihood for energy based models

Coming from this recent paper Your Classifier is Secretly an Energy Based Model And You Should Treat it Like One, they give the following definition... $$ p_\theta(\mathbf{x}) = \frac{\exp(-E_\theta(\...
1
vote
1answer
37 views

How to prove the monotonicity of Softmax when decreasing the weighted inputs?

Softmax function $a^L_j = z^L_j / sum_k(z^L_k)$. When we think about the monotonicity of the Softmax function, $∂a^L_j/∂z^L_k$ is positive if j=k, and negative if j≠k. As a consequence, increasing $z^...
1
vote
0answers
15 views

Directional derivative in regression (coefficients, after all, are partial derivatives)

The coefficients in a (let's stick with linear for now) regression are the partial derivatives. A regression equation is a function of several variables, so all of the multivariable calculus tricks ...
1
vote
1answer
34 views

Derivative of a Bivariate normal CDF with respect to its variables

Following up on the question (and answers) here, I'm trying to derive $\frac{\partial \Phi(x_1, x_2|\mathbf{\underline{\theta}})}{\partial x_1}$ and $\frac{\partial \Phi(x_1, x_2|\mathbf{\underline{\...
2
votes
1answer
333 views

What is the Hessian of the Gaussian likelihood

I am trying to learn the fine differences between different methods of Kronecker factoring for approximate curvature (like [1], and [2]) which require taking the Hessian of the pre-activations of the ...
0
votes
0answers
34 views

Why is the Hessian in the Laplace approximation negative

The Laplace approximation builds from the Taylor expansion of the MAP estimate, where the first derivative is 0. The second order Taylor series goes... $$ f(a) + \frac{f'(a)}{1!}(x - a) + \frac{f''(a)}...
3
votes
1answer
24 views

How to know the number of dimensions of a Jacobian?

My question comes from a comment in this question Vector Jacobian product in automatic differentiation The question states... $$ t = Wz, \,\,\, z\in \mathbb{R}^{m\times 1}, t \in \mathbb{R}^{n \times ...
2
votes
1answer
602 views

Derivation of Hessian for multinomial logistic regression in Böhning (1992)

This question is basically about row/column notation of derivatives and some basic rules. However, I couldn't figure out where I'm wrong. For multinomial logistic regression, I'm trying to get the ...
0
votes
0answers
25 views

derivative of Matrix normal distribution function

I am going to find the MLE for matrix normal distribution for my mixture model. I did parts of EM algorithm for that. however I am stuck in algebraic part of M step. I owuld appreciate any help with ...
1
vote
0answers
41 views

Matrix dimensions of Derivative of Softmax on Vectorized version

I am trying to get the matrix dimensions right for computing derivative of a two layers network where the last layer is softmax function. For simplicity I am only interested to get derivatives of W2 w....
0
votes
0answers
27 views

Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more?

I understand the concept of backpropagation in standard neural networks and backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of ...
0
votes
0answers
95 views

Help with derivation of gradient for specific filter in CNN

I need help with . I have to compute gradient for the special type of filter in CNN and everytime it comes up to be 0. Either this si correct, or I have some fundamental problem there, any hints are ...
0
votes
2answers
73 views

Meaning of θj in equation for partial derivative of MSE

The equation to find the partial derivative of a cost function with respect to a parameter θj is given in the book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: $$ \frac{\partial}...
0
votes
1answer
35 views

Meaning of a varaible for calculating the partial derviative of MSE cost function

The equation to find the partial derivative of a cost function with respect to a parameter θj is given in the book 'Hands on Machine Learning with scikit-learn, ...
3
votes
1answer
138 views

Difficulties in computing the derivatives of the Dirichlet distribution

I need to compute the first derivatives of the Dirichlet distribution, defined in the following way: $$r(P; \pi, \rho) = \frac{\Gamma(c)}{\prod_{i=1}^{k} \Gamma(c \pi_i)} \cdot \prod_{i=1}^{k} P_i^{c\...
1
vote
0answers
21 views

Estimating Logit Interaction Coefficients: the Second cross derivative

I have been working on a mixed effects statistical model that proposes a three-way interaction of continuous predictors on a binary variable in lme4. The general outline of the code is as follows: <...
0
votes
0answers
24 views

Computing the Jacobian $J_F$ with $F = h \circ f$

Let $$ f: \mathbb{R}^l \rightarrow{} \mathbb{R}^m\\[.7ex] h: \mathbb{R}^m \rightarrow{} \mathbb{R}^o$$ and let $$F = h \circ f \quad (F : \mathbb{R}^l \rightarrow{} \mathbb{R}^o)$$ I want to compute ...
0
votes
0answers
292 views

Comparing gam derivatives with gratia

I'm using gratia (thanks so much Gavin Simpson for that excellent library) to find the first derivative on time series. I cannot post the data here, but I have created a minimal example to illustrate ...
0
votes
0answers
81 views

Coding gradient descent from scratch - how are fitness functions incorporated into output layer error calculation?

For a project I am currently working on, I'm attempting to implement machine learning for a neural network using backpropagation and gradient descent from scratch. For much of my implementation, I ...
0
votes
0answers
58 views

Partial derivative of multivariate cdf with respect to coefficients

I want to take the partial derivative of this multivariate gaussian cumulative distribution function with respect to $\beta_1$ (which is a single element of the $\beta$ vector). $X_1$ is a n $\times$ ...
0
votes
1answer
102 views

How to implement LSTM backpropagation through time?

I'm building a custom LSTM net based on this article. I got questions on how to implement the backpropagation, based on these formulas of the derivatives in an LSTM layer: Question 1: The weights (w.....
0
votes
0answers
50 views

Best way to deal with non-continuously differentiable error functions in machine learning

Suppose you have the following set of $n=9$ numbers x = [1, 2, 3, 4, 5, 6, 7, 8, 9] and the corresponding y values [10, 2.5, 1.1, 0.6, 0.4, 0.2, 0.15, 0.12] which visualized look like this: I'd like ...
1
vote
1answer
109 views

$\frac\partial{\partial\beta}\left(\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i\right)$

How do you take the derivative of the function $$s(\beta)=\displaystyle\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i?$$ Attempt: $$H(\beta)=\frac\partial{\partial\...
1
vote
0answers
26 views

ReLU Neuron Derivative

For $x \in \mathbb R^d$, we define: $$ \text{ReLU}(x) := \max{(x, 0)} \\ \text{Step}(x) := \mathbb{1}[x \ge 0] $$ And ‘a neuron of the activation function $\psi$’ as $f_{\psi,w,b}(x) := \psi(w \cdot x ...
0
votes
0answers
151 views

For multivariate linear regression, what is the partial derivative for Mean Absolute Error?

What is the partial derivative for MAE for multivariate linear regression? I understand that for mean squared error (MSE) the partial derivative with respect to some $\theta_1$ would be $-\theta_1 \...
3
votes
0answers
74 views

Canonical LSTM backpropagation equations

I'm trying to understand the underlying mechanisms of LSTM from a programming perspective. I am no math person, and a lot of articles and papers look like alphabet soup to me. But I thought that if I ...
0
votes
0answers
32 views

How do I take the derivative to $\gamma $ of $(y-x^{\gamma})^T(y-x^{\gamma})$?

I have to solve the least squares for $\gamma$ in the following problem. The model is described as $y_i = \beta x_i^{\gamma} + u_i$, where $u_i $ is i.i.d. normal with mean zero and variance $\sigma^2$...
0
votes
1answer
139 views

How to derivate the following loss function?

How can I derivate the following optimization function? $$L=\sum_{u,i}(y_{u,i}-v_ix_u)^2+\lambda\left(\sum_i\|v_i\|_2^2+\sum_u\|x_u\|_2^2\right)$$ I just want to get the equations of the gradient ...
6
votes
1answer
265 views

How to calculate and interpret a marginal treatment effect (local instrumental variable)? (Intuition through simple example.)

I am working on the intuition behind local instrumental variables (LIV), also known as the marginal treatment effect (MTE), developed by Heckman & Vytlacil. I have worked some time on this and ...
3
votes
1answer
78 views

Deriving Logit Maximum Likelihood Estimator

According to Verbeek, we can obtain the logit model by simplifying the first order condition of the log-likelihood function. Where,  $$logL(\beta) = \Sigma^N_{i=1} y_i logF(x^{'}_i\beta)+ \Sigma^N_{i=...
0
votes
1answer
28 views

Total differential of a linear model with interaction terms

Suppose I have the following function, representing a linear model with an interaction term: $$ f(x, y) = \beta_{1} x + \beta_{2} y + \beta_{3} xy. $$ Now I want to see how the function changes if ...

1
2 3 4 5