Questions tagged [derivative]

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

Filter by
Sorted by
Tagged with
1
vote
0answers
12 views

Matrix form of elementwise derivations

The elementwise derivations w.r.t e of $$ J = \frac{1}{2}[\Sigma_{r,s=1}^{R}a_{rt}a_{st}k(e_r,e_s) - 2\Sigma_{r=1}^{R}a_{rt}k(e_r, x_t)]$$ can be given by: $$ \frac{\partial J}{\partial e_r} = \Sigma_{...
2
votes
1answer
33 views

Covariance of derivative of Gaussian Process Regression

There are a quite a few questions and answers which discuss how to calculate the gradients/derivatives of the posterior of Gaussian Process Regression (see here, here). These include the equations for ...
0
votes
0answers
14 views

calculate the gradient of cross-entropy(A ReLU(AXW0) W1) w.r.t A [closed]

I've been trying to calculate the gradient of loss of my neural network with respect to a matrix A. The loss is calculated as loss = cross-entropy(A ReLU(AXW0) W1) ...
1
vote
1answer
28 views

Derivative of a Function $ln(1+\exp(-y.w^T.\phi_W(x_i) ))$

I want to take derivative of $ln(1+\exp(-y.w^T.\phi_{W}(x_i) ))$ with respect to $w$. So far what i have done is Let $u=1+\exp(-y.w^T.\phi_{W}(x_i) )$, The above expression will become $ln(u)$ $\frac{...
0
votes
0answers
18 views

What are the derivatives of Squared Exponential kernel function w.r.t. characteristic length scale (Gauss Process)

I'm writing a matlab code to implement Gaussian process. In the book: Gaussian Process for machine learning by Carl Edward Rasmussen and Christopher K. I. Williams, the authors define the squared ...
0
votes
0answers
17 views

Derivation of matrix derivative [duplicate]

In the ML lecture notes of cs229, I am not able to derive the equation 3 in matrix derivatives. I applied the equation 1 to derive it, considering the product after A as another matrix, I am not ...
0
votes
0answers
21 views

Is my step by step derivation of quadratic cost function's (Neural Networks) partial derivative with respect to some weights matrix correct?

I am trying to revise the details of a Multi-layer Perceptron with a set of weight matrices $\mathcal W$ and a set of bias vectors $\mathbf b$. Here is the quadratic cost function I am using, $$C(\...
8
votes
4answers
187 views

Where does the logistic function come from?

I first learned the logistic function in machine learning course, where it is just a function that map a real number to 0 to 1. We can use calculus to get its derivative and use the derivative for ...
0
votes
1answer
25 views

Derivative of all the parameters in Logistic Regression

$\mathcal{L}$ is the loss function, $\mathcal{L} = y_i \text{log} \sigma(z) + (1-y_i) \text{log} (1-\sigma(z))$, where $z = \sum_i w_ix_i$, with $w_i$ representing the weights and $x_i$ the features. ...
1
vote
1answer
19 views

n'th cumulant (of a CGF) for exponential family / exponential dispersion model

The n'th cumulant is defined to be the n'th derivative of the CGF (cumulant generating function). $$\kappa_n = \frac{d^n K(t)}{dt^n} |_{t=0} $$ But I'm reading in a book (p.215, chapter5, eq. 5.8) ...
1
vote
1answer
268 views

For back propagation in neural networks , how do we calculate vector by matrix derivative?

I am following the course deep learning ai by Andrew NG. In course1 week4, 04-06-Forward and Backward Propagation, he calculates backward propagation for layer $l$ in neural networks as follows (a ...
2
votes
1answer
155 views

Can we calculate the predictive process of the derivative of a stochastic process from the data?

I asked this question, Calculating the expression for the derivative of a Gaussian process, some time ago, and now I am interested in an extension to the question. So originally I wanted to know the ...
31
votes
1answer
12k views

Step-by-step example of reverse-mode automatic differentiation

Not sure if this question belongs here, but it's closely related to gradient methods in optimization, which seems to be on-topic here. Anyway, feel free to migrate if you think some other community ...
49
votes
5answers
97k views

Backpropagation with Softmax / Cross Entropy

I'm trying to understand how backpropagation works for a softmax/cross-entropy output layer. The cross entropy error function is $$E(t,o)=-\sum_j t_j \log o_j$$ with $t$ and $o$ as the target and ...
1
vote
0answers
40 views

Posterior distribution for a Gaussian Process with a transformation in a gaussian likelihood

Suppose we are modelling observations y as follows. Our likelihood is normal $ y \sim \mathcal{N}(g(f(x)), \mathcal{I}\sigma^2)$, where $\mathcal{I}$ is the identity matrix and $g$ is some function ...
0
votes
0answers
21 views

Relation between dyi/dei and its leverage (hii)

Althought I have tried in different ways, I have not been able to show that $$\frac{\mathrm{d}Y_i}{\mathrm{d}e_i} = \frac{1}{1 - l_{ii}}$$ $e_i$ is equal to $l_{ii}$ is the $i-th$ element of the ...
1
vote
0answers
20 views

Is there a statistical test to determine correlation between the first derivative of two time based series?

I have two time series. They are normally distributed, parametric, exactly the same size and evenly spaced. I am seeking to determine if the rate of change of these two time series are correlated ...
3
votes
0answers
13 views

Meaning of a notation regarding mean square derivative

I'm reading a paper (On Differentiable Functionals, Van der Vaart, 1991, Annals of Statistics), and I've got a question regarding a notation in the following part: My Question: Does $dP^{1/2}$ mean $...
1
vote
1answer
27 views

Partial derivative of a linear regression with correlated predictors

Let's set up the situation of having some $Y$ that I think depends on a linear combination of $X_1$ and $X_2$. I could fit a regression model: $$y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2}$$ We ...
4
votes
1answer
40 views

Normal distribution “stable” under derivative?

Suppose that $\theta(t)\sim\mathcal N(\mu(t),\Sigma(t))$ where $t$ is some parameter. Then it holds that $$\theta(t) = \mu(t) + \Sigma(t)^{0.5}\xi$$ for $\xi\sim\mathcal N(0, I)$. I am interested in ...
0
votes
1answer
51 views

What do I miss in this derivation?

The school is closed due to the ongoing pandemic. And I am interested in the application of the Bayes Theorem in COVID-19. Here is what I thought. The total population in U.S. is approximately 327,...
0
votes
2answers
601 views

derivative from regression model in R

My regression model is this: R2 <- lm(lnwage ~ educ + exper + hrswk + educ*exper + educ2 + exper2) and I want to estimate: $\frac{\partial ln(wage)}{\partial ...
9
votes
4answers
8k views

How to find derivative of softmax function for the purpose of gradient descent?

I'm trying to understand back propagation algorithm for multiclass classification using gradient descent. I'm using https://www.cs.toronto.edu/~graves/phd.pdf . The output layer is a softmax layer, in ...
2
votes
2answers
244 views

Is computing natural gradient equivalent to deriving directional derivative?

It seems to me that natural gradient is simply derived from directional derivative. For example, for a vector $v$, $\tilde{\nabla} f \cdot v = G^{-1} \nabla f \cdot v = \lim_{h\to0} \frac{f(x+hv)-f(x)...
0
votes
1answer
28 views

How to derive the gradient of the reparameterized score function estimator?

In the paper Evolution Strategies as a Scalable Alternative to Reinforcement Learning, the authors derive the following gradient of the score function estimator $$ \begin{align} \nabla_\psi\mathbb E_{...
1
vote
0answers
13 views

error propagation for derivatives

I have the following problem: I have some data of a function f(x) with a set of 300 values of it associated to the same number of values of x including corresponding standard deviation σ(f) for each ...
0
votes
0answers
39 views

Finding Derivative of a spline

I have received data values for a spline (which was already fit to some ndvi data). I just have only the data points of the spline and do not know the function that the spline follows. My goal is to ...
1
vote
0answers
10 views

rate of convergence for cross derivative estimation in local linear regression

Suppose $Y_{i}=m(X_{1i},X_{2i})+\epsilon_{i}$, with $E(Y_{i}|X_{1i},X_{2i})=m(X_{1i},X_{2i})$ where $m(\cdot,\cdot)$ is an unknown smooth function. If the estimator $\widehat{m}(x_{1},x_{2})$ is ...
0
votes
0answers
40 views

Finding the gradient $\nabla$ of the logistic regression cost function

I want to use vector calculus to derive the gradient $\nabla_wJ(w)$ of the logistic regression cost function $J(w) = -\textbf{y}\cdot ln\textbf{ s} - (\mathbf{1} - \textbf{y}) \cdot ln( \mathbf{1} - \...
3
votes
2answers
76 views

When using OLS on $\ln(y) = \beta_1 \ln(x) + \epsilon$, is $\beta_1$ the elasticity of $E[y\vert x]$, or the $y$ in the data (or both)?

Specifically, suppose we are estimating $$ \ln(y)=\beta_1\ln(x) + \epsilon $$ I understand that $\beta_1 = \frac{\partial \ln(y)}{\partial \ln(x)}$ which is the elasticity of $y$ with respect to $x$ ...
0
votes
0answers
39 views

Back-propagation through cross entropy or logistic loss function

I have neural network which ends with softmax function and I want to minimize cross-entropy cost function which takes output of this network and one-hot labels as arguments. To calculate partial ...
1
vote
1answer
46 views

Expression for Derivative of Hyperparameter of Kernel with respect to New Data

I would like to determine how the hyperparameter will change when a new data is observed and the GP is updated with this new data. Considering the following predictive distribution of the GP: $$\mu(...
0
votes
1answer
53 views

Derivation of gradient-bandit algorithm, Why is the sum of the derivatives is zero?

https://www.cs.mcgill.ca/~dprecup/courses/RL/Lectures/2-bandits-2019.pdf In above pdf document, page 19, they explain by formula: $$\sum _{ b }^{ }{ \frac { \nabla { \Pi }_{ t }(b) }{ \nabla { H }_{ ...
0
votes
0answers
9 views

Computing custom gradient for LSTM equations

Consider an LSTM that takes in as input a sequence of N words $X_1,\cdots,X_N$. Each word is a vector $\in R^D$. The dimension of the LSTM neuron is $H$. Suppose we are doing sentiment classification ...
3
votes
1answer
226 views

Derivative of Gaussian Process (continued)

This is to extend the discussion of the derivative of the GP. The formulation provided in the previous post describes the gradient of GP as derivative of kernel function as follows with respect to $(x^...
0
votes
0answers
78 views

Gradient of multivariate normal distribution function?

Let $X\sim\mathcal{N}_J(\mu,\Sigma)$ be a multivariate normal with PDF $f_X$ and CDF $F_X$. Taking derivatives of $f_X$ wrt $X$, $\mu$ and $\Sigma$ is easy as shown here. However, I am interested in ...
1
vote
0answers
45 views

Relating Two Derivatives (and Elasticities) of a Log-Log Regression

Consider a standard "log-log" linear regression model like this: $\log(y_i) = \log(a_i + b_i)\delta + \epsilon_i$, where $y$ is the dependent variable, $a$ and $b$ are two independent variables, and ...
1
vote
1answer
70 views

Why does gradient descent HAVE to find the minimum as oppose to a change in the opposite direction

I have a question about the gradient descent step in neural networks. I fully understand the derivative step and taking the steps required to move in the direction that reduces the loss (finding the ...
0
votes
1answer
38 views

is it a good idea to take the derivative or integral of some features and add them as new features in machine learning?

I'm learning how to do feature Engineering and come across some ideas in my head that's why I want to ask if I had some dataset with some features let's say 2 features and I have a timestamp column ...
9
votes
1answer
10k views

Differentiation of Cross Entropy

I have been trying to create a program for training Neural Networks on my computer. For the Network in question, I have decided to use the Cross Entropy Error function: $$E = -\sum_jt_j\ln o_j$$ ...
1
vote
2answers
4k views

Matrix Representation of Softmax Derivatives in Backpropagation

I have a simple multilayer fully connected neural network for classification. At the last layer I have used softmax activation function. So I have to propagate the error through the softmax layer. ...
11
votes
3answers
2k views

Can a neural network learn a functional, and its functional derivative?

I understand that neural networks (NNs) can be considered universal approximators to both functions and their derivatives, under certain assumptions (on both the network and the function to ...
3
votes
1answer
59 views

Derivation of score vector

Can anyone explain the process of this derivation, step by step? This derivation is from Joint Models for Longitudinal and Time-to Event Data by Dimitris Rizopoulos. \begin{equation} \begin{aligned} ...
0
votes
2answers
44 views

Source of vanishing/exploding gradients in RNN

Problem I am trying to understand the source of vanishing/exploding gradients in vanilla RNN. The update rule of vanilla RNN is $$ \begin{aligned} &\mathbf{a}^{\left<t\right>}=\...
0
votes
0answers
184 views

Deriving Gradient from negative log-likelihood function

I have been having some difficulty deriving a gradient of an equation. I have a Negative log likelihood function, from which i have to derive its gradient function. Negative log likelihood function ...
0
votes
0answers
30 views

Show that $\int^b_a\phi'''(z)dz$ lies between $\pm[\phi(0)+2\phi(\sqrt3)]$ for every $a<b$

Show that $\int^b_a\phi'''(z)dz$ lies between $\pm[\phi(0)+2\phi(\sqrt3)]$ for every $a<b$. $$\phi(z)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}z^2}$$ I've shown that: $$\phi''(z)=(z^2-1)\phi(z)$$ $$\...
4
votes
1answer
2k views

Computing gradients via Gaussian Process Regression

I have a set of noisy data that I am fitting using Gaussian Process Regression via Python's sklearn package. The posterior mean of the GP is essentially my output with an associated error. Based on ...
14
votes
3answers
428 views

How can I fit a spline to data that contains values and 1st/2nd derivatives?

I have a dataset that contains, let's say, some measurements for position, speed and acceleration. All come from the same "run". I could construct a linear system and fit a polynomial to all of those ...
1
vote
0answers
62 views

Computing gradients w.r.t input / Gaussian Process (RBF Kernel)

my intent is to write down an illustrative example of the derivatives w.r.t input from the MSE -> GP (with an RBF Kernel) -> Inputs $\frac{\partial MSE }{\partial x} (GP(x))$ Could anyone help me ...
0
votes
1answer
88 views

Derivative of reparametrization

I am stuck in one equation when reading the paper. Regardless of the context, here is the question: Let's say given an equation L($\sigma_g^2$, $\sigma_\epsilon^2$, $\beta$) = .... Now I want to use $\...