Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [derivative]

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

0
votes
0answers
8 views

Error when computing the gradient of a derivative with Gaussian process regression in scikit

Gaussian process regression in scikit-learn provides a conditional mean and variance, $(y_*,\sigma_*^2)$, based upon the observed data, $(y,\sigma^2)$. But given $(y_*,\sigma_*^2)$, how can you ...
2
votes
1answer
28 views

Derivative of expectation where the variable appears in the integration limit and in the integrand?

I want to calculate the derivative of $$\varphi(\mu) = \int_{-\infty}^{\mu} r(x-\mu) f(x)dx,$$ wrt to $\mu$, where $r$ is a function and $f$ is a density function. How can I account for the presence ...
0
votes
1answer
21 views

Calculate the derivative of the CDF with respect to the mean value [duplicate]

I want to derive the cumulative density function (cdf) for variables following normal distribution with respect to the parameters of the cdf (such as the mean or the standard deviation)
2
votes
1answer
25 views

Question about the gradient of weight normalization

In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$ \mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v $$ Then they ...
0
votes
0answers
8 views

notation of derivative probability function

I am reading article about bayesian predictive function. In the article it denote posterior distribution $\pi_n(d\theta) = \frac {\prod^n_{i=1}f(y_i|\theta) \pi(d\theta)}{\int \prod^n_{i=1}f(y_i|\...
0
votes
0answers
12 views

CCA on feature maps: Gradient w.r.t to Jacobian

Assume I have two neural networks, abstracted as two feature maps, parametrized by $\theta_x,\theta_y$ respectively. $\phi_x(x;\theta_x) \in \mathbb{R}^{h_1}$, $\phi_y(x;\theta_y) \in \mathbb{R}^{h_2}$...
0
votes
0answers
24 views

Gradient of marginal likelihood of Gaussian Process w.r.t likelihood parameters with Laplace approximation

The derivation of gradient of the marginal likelihood w.r.t covariance function hyperparameters $\theta$ is given in http://www.gaussianprocess.org/gpml/chapters/RW5.pdf, page 125. However, the ...
10
votes
1answer
629 views

Second order approximation of the loss function (Deep learning book, 7.33)

In Goodfellow's (2016) book on deep learning, he talked about equivalence of early stopping to L2 regularisation (https://www.deeplearningbook.org/contents/regularization.html page 247). Quadratic ...
2
votes
1answer
20 views

understanding derivatives of a regression spline

I am trying to understand why regression splines are continuous at their knots Suppose I am fitting a regression spline $$ E[Y|X] = \alpha + \beta_1 x + \beta_2 (x - t)^+ $$ where $(x - t)^+ = \...
0
votes
1answer
71 views

Find the derivative w.r.t. matrix normal distribution pdf

We have the pdf of matrix normal distribution for the random matrix $X$ (https://en.wikipedia.org/wiki/Matrix_normal_distribution): However here in my case, $X$ is of a parameter, say $\theta$. So my ...
0
votes
1answer
30 views

Optimisation by using directional derivative

So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise ...
0
votes
1answer
35 views

Derivative with Reparameterisation Trick

Below is some steps for differentiating a function wrt a set of parameters $\phi$ using the "reparameterisation trick" (Kingma & Welling 2013). However after applying the derivative as follows I ...
1
vote
0answers
16 views

gradient vs derivative: defintions of [closed]

According to wikipedia: In mathematics, the gradient is a multi-variable generalization of the derivative. Like the derivative, the gradient represents the slope of the tangent of the graph of ...
2
votes
0answers
36 views

Incorrect computation in Knight and Fu (2000)?

I'm currently reading Knight and Fu's 2000 paper on the asymptotics of "Bridge" estimators with a particular focus on LASSO as a special case. In the proof of theorem 2, they make the claim that under ...
2
votes
1answer
61 views

Finding expression of $n$-th derivative, when $n$ is large

For completeness, assume $C$ is an Archimedean copula with some generator function $\varphi$, which is usually assumed to have nice properties. It is known that $$ C(u_1, u_2, \ldots, u_n)=\varphi^{-1}...
2
votes
0answers
32 views

Derivatives of quantile loss function [duplicate]

I'm reading a text - Roger Koenker (2005) Quantile Regression [page 8] - that goes like this: Consider the function $$R(\xi) = \sum_{i=1}^n \rho_\tau(y_i-\xi)$$ where $$\rho_\tau(y_i-\xi) =(y_i-\xi)(\...
0
votes
1answer
25 views

Small calculation in Max entropy

I am wondering where the -1 comes from in the derivatives with respect to the probability. For example, these notes. For the basic example with the uniform distribution. We want to minimze $[-\int ...
0
votes
0answers
15 views

Derivative of Time-Transformed Stochastic Process

Given a continuous time stochastic process X(t), we can define the functional transformation, $$f(X)(t)=(X(t))^2−2X(t)$$ and evaluate the Hadamard derivative. Given a transformation on the real ...
4
votes
2answers
54 views

Slope of Curve with Unknown Functional Form

I have a monotonically-increasing curve whose functional form is not known a priori and would like to compute the curve's slope at the rightmost endpoint. Typically, when the functional form is known, ...
2
votes
1answer
329 views

Computing gradients via Gaussian Process Regression

I have a set of noisy data that I am fitting using Gaussian Process Regression via Python's sklearn package. The posterior mean of the GP is essentially my output with an associated error. Based on ...
0
votes
1answer
147 views

How are biases updated when 'batch size' > 1?

This is my network represented in matrices: (a dot represents an arbitrary number) Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity) Backpropagation The ...
0
votes
2answers
707 views

How to calculate the derivative of crossentropy error function?

I'm reading this tutorial (presented below) on computing derivative of crossentropy. The author used the loss function of logistic regression I think. https://www.dropbox.com/s/rxrtz3auu845fuy/...
2
votes
1answer
52 views

Gaussian log-density variational derivative

In Appendix C.1 of 'Taming VAEs' paper, the authors need to compute the functional derivative $$\frac{\delta}{\delta g\left( z \right)} \mathbb{E}_{q\left(z\mid x\right)} \left[(g \left( z \right) - ...
-1
votes
0answers
26 views

Linear Regression Update Rule

Given the following quadratic loss function: $$\frac{1}{2n}\sum_{i \in [n]} \left[y^{(i)} - \sigma(w^\top x^{(i)})\right]^2$$ I'm trying to figure out the update rule for the weight $w$. $w:=w - \...
1
vote
0answers
16 views

How to obtain the functional derivative of variational distribution?

Referring to , I want to know how to derive the parameters for the variational distribution, in the Bayesian inference section of the paper. I know how to derive, but I don't know how to deduct the ...
1
vote
0answers
112 views

Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
1
vote
0answers
45 views

How to analytically solve the probability of improvement acquisition function in Bayesian Optimization with Vector inputs?

I have been using the probability of improvement acquisition function in my Bayesian Optimization program, but I've run into a problem because I am not optimizing the acquisition function that quickly....
3
votes
2answers
57 views

When are the linear regression parameters of Y and X the same as the parameters of Y' and X'?

I am working on some simple linear modeling of a physical system and assumed that taking the derivative of an equation $$Y = \beta_1 + \beta_2 X + \varepsilon$$ would give me $$\frac{dY}{dt} = \...
1
vote
0answers
100 views

Get partial derivative in pytorch [closed]

coords[i] is a list containing 3 elements x,y,z and I want to get the derivative of G[i] w.r....
0
votes
1answer
53 views

What is the expression for derivative of the signum function one should use in the BP training method

The back propagation learning method requires knowing of derivatives of activation functions. But what expression one should use for signum activation function $$ \mathrm{Signum}( x ) = \...
1
vote
1answer
51 views

For back propagation in neural networks , how do we calculate vector by matrix derivative?

I am following the course deep learning ai by Andrew NG. In course1 week4, 04-06-Forward and Backward Propagation, he calculates backward propagation for layer $l$ in neural networks as follows (a ...
2
votes
1answer
44 views

Minimizing a least square [duplicate]

I'm a bit confused with matrix calculus. Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$ I don't see how this works. My plan was to ...
1
vote
1answer
54 views

What does $\mathcal{L}$ stands for in back propagation?

I am trying to learn how to do deep neural networks with this Ipython notebook. I'm puzzled about notations in linear backward learning section. For layer $l$, the linear part is: $Z^{[l]} = W^{[l]} ...
3
votes
0answers
34 views

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian. For the time being, I obtain the derivative as $\frac{\...
3
votes
1answer
42 views

Correlation between the j-feature and partial residual

I am reading a book for the lasso-regression. However, I think that this is a more general question regarding the derivative of the OLS-term: Can someone explain me why this $c_j$ term is ...
0
votes
0answers
112 views

Deriving the gradient vector of a Probit model

Consider the probit regression model where the pdf of $y_{i}$ is $$f(y_{i};\mathbf{\beta}) = \mu_{i}^{y_{i}}\left ( 1 - \mu_{i} \right )^{1 - y_{i}},$$ where $y_{i} = \left \{ 0,1 \right \}$ and $\...
1
vote
0answers
12 views

Comparing Importance of Values to Importance in Change of Values [closed]

I am working with some hourly weather data and I am trying to find how strongly variables correlate and which variables are more important that others. Along with the actual weather data (temperatures ...
1
vote
0answers
133 views

LASSO: All KKT-Conditions

I'm trying to code the lasso algorithm via coordinate descent by myself. However, I want to implement a check for the KKT-conditions as in this blog. But what are all KKT conditions for the LASSO ? $...
1
vote
1answer
61 views

Derivative equal to zero in PCA

Source: presentation In PCA, we want to maximize variance $var(\alpha'_kX)$, i.e. $\alpha'_k \Sigma \alpha_k$ with respect to $\alpha' \alpha = 1$, where $\Sigma = X'X$. After introducing Lagrange ...
0
votes
1answer
176 views

Optimal neural net weights when using cross entropy loss

I'm trying to understand how cross-entropy works for finding the optimal weights in neural networks. According to Eli Bendersky's website and neural networks and deeplearning tutorial, we can find the ...
1
vote
1answer
117 views

Differentiation of RSS

I have the equation $RSS\left ( \beta \right )=\left ( y-X\beta \right )^{T}\left ( y-X\beta \right )$ that I'm trying to differentiate w.r.t. $\beta$. I used chain rule to get to $\frac{\partial ...
1
vote
0answers
181 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
3
votes
0answers
59 views

Second derivative test for machine learning algorithms [duplicate]

I have a question on second derivative test for most "modern" machine learning algorithms. I learned that in calculus but never seen it in real applications. Most machine learning algorithms ...
2
votes
0answers
94 views

Derivative of a conditional CDF w.r.t. the condition [closed]

I would like to ask you for a help with my problem. I have random variable $$X = X_1 + \sum_{i = 2}^{n}(X_i - y_i)^{+}, \ \ \ (.)^{+} = \max(0, . ), $$ where $X_i$ are independent, continuous with ...
0
votes
1answer
55 views

Backpropagation - partial derivatives

I am currently reading the Neural networks and deep learning book by Michael Nielsen. I have a question regarding the backpropagation chapter: Background: He explains the influence of a neuron on ...
1
vote
1answer
251 views

Derivative of bivariate normal CDF with common mean parameters

I am trying to calculate a derivative of the form $\frac{d}{dz}\Phi_2(\mu_1(z),\mu_2(z),\rho)$ where $\Phi_2$ is the standard bivariate normal CDF. I am thinking it might be an application of a ...
1
vote
0answers
408 views

Gradient boosting for binary outcome - terminal nodes estimate (using R gbm)

I have been searching for the answer for the below query quite a long time and found a few answers (see: interpretation of gbm single tree prediction in pretty.gbm.tree or R Package GBM - Bernoulli ...
0
votes
0answers
180 views

Derivatives of the multivariate normal CDF w.r.t. its parameters

Let $\boldsymbol X \sim \mathcal N_d(\boldsymbol\mu, \boldsymbol\Sigma)$ and let $f_{\boldsymbol X}(\boldsymbol x)$ be its PDF. The CDF is $$ F_{\boldsymbol X}(\boldsymbol q) = \int_{-\infty}^{q_1} \...
2
votes
1answer
1k views

Working for Logistic regression partial derivatives

In Andrew Ng's Neural Networks and Deep Learning course on Coursera the logistic regression loss function for a single training example is given as: $$ \mathcal L(a,y) = - \Big(y\log a + (1 - y)\log (...
1
vote
2answers
161 views

Is computing natural gradient equivalent to deriving directional derivative?

It seems to me that natural gradient is simply derived from directional derivative. For example, for a vector $v$, $\tilde{\nabla} f \cdot v = G^{-1} \nabla f \cdot v = \lim_{h\to0} \frac{f(x+hv)-f(x)...