The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [derivative]

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

Filter by
Sorted by
Tagged with
1
vote
0answers
24 views

Computing gradients w.r.t input / Gaussian Process (RBF Kernel)

my intent is to write down an illustrative example of the derivatives w.r.t input from the MSE -> GP (with an RBF Kernel) -> Inputs $\frac{\partial MSE }{\partial x} (GP(x))$ Could anyone help me ...
0
votes
1answer
41 views

Derivative of reparametrization

I am stuck in one equation when reading the paper. Regardless of the context, here is the question: Let's say given an equation L($\sigma_g^2$, $\sigma_\epsilon^2$, $\beta$) = .... Now I want to use $\...
13
votes
3answers
259 views

How can I fit a spline to data that contains values and 1st/2nd derivatives?

I have a dataset that contains, let's say, some measurements for position, speed and acceleration. All come from the same "run". I could construct a linear system and fit a polynomial to all of those ...
0
votes
0answers
25 views

Meaning of the derivative of cross entropy wrt $p(x)$

Lets define the cross entropy between 2 probability distributions $p(x)$ and $q(x)$ as $$H(p,q) = -\sum p(x) \log{q(x)} $$ What would be the meaning of derivative of $H(p,q)$ when taking the ...
1
vote
0answers
28 views

First derivative of splines in bayesian model

Inspired by this post https://www.fromthebottomoftheheap.net/2014/05/15/identifying-periods-of-change-with-gams/, I'm trying to identify periods of change in a GAM model, using bayesian inference. ...
2
votes
1answer
66 views

how to derive eq 21.2.3 in BRML, Factor Analysis (Eigen-approach likelihood optimization)

how to derive eq 21.2.3 in BRML, chapter21 Factor Analysis? Log Likelihood function (eq. 21.1.13): $$ \log{p(\mathcal{V} | \mathbf{F}, \mathbf{\Psi})} = -\frac{N}{2}\left( \mathrm{trace}(\mathbf{\...
0
votes
1answer
43 views

What is the formula for the conditional variance when taking the derivative of a Gaussian process?

The formulae for the conditional mean and variance of a Gaussian process is given by equations (2.23) and (2.24): Also, the formula for the covariance of the derivative of a Gaussian process is given ...
0
votes
0answers
33 views

Quadratic Approximation of the binary logistic regression

I am using https://web.stanford.edu/~hastie/Papers/glmnet.pdf package to solve my optimization problem for the Binary Logistic Regression. On page 10 it is stated that the quadratic approximation of ...
1
vote
1answer
18 views

Updating Prediction Errors in Gradient Ascent (Friston's Free-energy)

Background In Rafal Bogacz's tutorial on the free-energy framework for modelling perception and learning, section 2.3 we have: $$\dot{\phi} = \frac{\partial F}{\partial\phi} = \varepsilon_u g'(\phi) ...
1
vote
0answers
16 views

What does this assumption mean regarding equal marginal densities?

Suppose that we have a random variable $\epsilon$ with density $q(\epsilon)$ and $w = t(\theta, \epsilon)$, where $t$ is a deterministic function of a constant $\theta$ and random variable $\epsilon$. ...
1
vote
1answer
44 views

Why do cubic splines need to be continuous at the first and second derivative, but discontinuous at the third?

I'm working through Introduction to Statistical Learning and came upon this: One can show that adding a term of the form $ \beta_4h(x,\xi)$ to the model (7.8) $$y_i = \beta_0 + \beta_1x_i + \...
1
vote
0answers
43 views

Propagating error when taking the derivative

I have a function that corresponds to a set of $(X,Y)$ coordinates with a Gaussian uncertainty ($\sigma_Y$) for each point. What I want to do is now compute the gradient of this function and the ...
0
votes
1answer
31 views

Backpropagation gradient of the average

In the Pytorch Udacity course, the following is said at one point: To calculate the gradients, you need to run the .backward method on a Variable, ...
2
votes
1answer
32 views

Derivative of expectation where the variable appears in the integration limit and in the integrand?

I want to calculate the derivative of $$\varphi(\mu) = \int_{-\infty}^{\mu} r(x-\mu) f(x)dx,$$ wrt to $\mu$, where $r$ is a function and $f$ is a density function. How can I account for the presence ...
0
votes
1answer
47 views

Calculate the derivative of the CDF with respect to the mean value [duplicate]

I want to derive the cumulative density function (cdf) for variables following normal distribution with respect to the parameters of the cdf (such as the mean or the standard deviation)
2
votes
1answer
47 views

Question about the gradient of weight normalization

In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$ \mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v $$ Then they ...
0
votes
0answers
8 views

notation of derivative probability function

I am reading article about bayesian predictive function. In the article it denote posterior distribution $\pi_n(d\theta) = \frac {\prod^n_{i=1}f(y_i|\theta) \pi(d\theta)}{\int \prod^n_{i=1}f(y_i|\...
0
votes
0answers
14 views

CCA on feature maps: Gradient w.r.t to Jacobian

Assume I have two neural networks, abstracted as two feature maps, parametrized by $\theta_x,\theta_y$ respectively. $\phi_x(x;\theta_x) \in \mathbb{R}^{h_1}$, $\phi_y(x;\theta_y) \in \mathbb{R}^{h_2}$...
0
votes
0answers
39 views

Gradient of marginal likelihood of Gaussian Process w.r.t likelihood parameters with Laplace approximation

The derivation of gradient of the marginal likelihood w.r.t covariance function hyperparameters $\theta$ is given in http://www.gaussianprocess.org/gpml/chapters/RW5.pdf, page 125. However, the ...
11
votes
1answer
706 views

Second order approximation of the loss function (Deep learning book, 7.33)

In Goodfellow's (2016) book on deep learning, he talked about equivalence of early stopping to L2 regularisation (https://www.deeplearningbook.org/contents/regularization.html page 247). Quadratic ...
3
votes
1answer
29 views

understanding derivatives of a regression spline

I am trying to understand why regression splines are continuous at their knots Suppose I am fitting a regression spline $$ E[Y|X] = \alpha + \beta_1 x + \beta_2 (x - t)^+ $$ where $(x - t)^+ = \...
0
votes
1answer
77 views

Find the derivative w.r.t. matrix normal distribution pdf

We have the pdf of matrix normal distribution for the random matrix $X$ (https://en.wikipedia.org/wiki/Matrix_normal_distribution): However here in my case, $X$ is of a parameter, say $\theta$. So my ...
0
votes
1answer
32 views

Optimisation by using directional derivative

So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise ...
0
votes
1answer
39 views

Derivative with Reparameterisation Trick

Below is some steps for differentiating a function wrt a set of parameters $\phi$ using the "reparameterisation trick" (Kingma & Welling 2013). However after applying the derivative as follows I ...
1
vote
0answers
25 views

gradient vs derivative: defintions of [closed]

According to wikipedia: In mathematics, the gradient is a multi-variable generalization of the derivative. Like the derivative, the gradient represents the slope of the tangent of the graph of ...
2
votes
0answers
48 views

Incorrect computation in Knight and Fu (2000)?

I'm currently reading Knight and Fu's 2000 paper on the asymptotics of "Bridge" estimators with a particular focus on LASSO as a special case. In the proof of theorem 2, they make the claim that under ...
2
votes
1answer
67 views

Finding expression of $n$-th derivative, when $n$ is large

For completeness, assume $C$ is an Archimedean copula with some generator function $\varphi$, which is usually assumed to have nice properties. It is known that $$ C(u_1, u_2, \ldots, u_n)=\varphi^{-1}...
2
votes
0answers
34 views

Derivatives of quantile loss function [duplicate]

I'm reading a text - Roger Koenker (2005) Quantile Regression [page 8] - that goes like this: Consider the function $$R(\xi) = \sum_{i=1}^n \rho_\tau(y_i-\xi)$$ where $$\rho_\tau(y_i-\xi) =(y_i-\xi)(\...
0
votes
1answer
25 views

Small calculation in Max entropy

I am wondering where the -1 comes from in the derivatives with respect to the probability. For example, these notes. For the basic example with the uniform distribution. We want to minimze $[-\int ...
4
votes
2answers
71 views

Slope of Curve with Unknown Functional Form

I have a monotonically-increasing curve whose functional form is not known a priori and would like to compute the curve's slope at the rightmost endpoint. Typically, when the functional form is known, ...
3
votes
1answer
700 views

Computing gradients via Gaussian Process Regression

I have a set of noisy data that I am fitting using Gaussian Process Regression via Python's sklearn package. The posterior mean of the GP is essentially my output with an associated error. Based on ...
0
votes
1answer
212 views

How are biases updated when 'batch size' > 1?

This is my network represented in matrices: (a dot represents an arbitrary number) Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity) Backpropagation The ...
0
votes
2answers
2k views

How to calculate the derivative of crossentropy error function?

I'm reading this tutorial (presented below) on computing derivative of crossentropy. The author used the loss function of logistic regression I think. https://www.dropbox.com/s/rxrtz3auu845fuy/...
2
votes
1answer
55 views

Gaussian log-density variational derivative

In Appendix C.1 of 'Taming VAEs' paper, the authors need to compute the functional derivative $$\frac{\delta}{\delta g\left( z \right)} \mathbb{E}_{q\left(z\mid x\right)} \left[(g \left( z \right) - ...
1
vote
0answers
18 views

How to obtain the functional derivative of variational distribution?

Referring to , I want to know how to derive the parameters for the variational distribution, in the Bayesian inference section of the paper. I know how to derive, but I don't know how to deduct the ...
1
vote
0answers
147 views

Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
1
vote
0answers
67 views

How to analytically solve the probability of improvement acquisition function in Bayesian Optimization with Vector inputs?

I have been using the probability of improvement acquisition function in my Bayesian Optimization program, but I've run into a problem because I am not optimizing the acquisition function that quickly....
3
votes
2answers
57 views

When are the linear regression parameters of Y and X the same as the parameters of Y' and X'?

I am working on some simple linear modeling of a physical system and assumed that taking the derivative of an equation $$Y = \beta_1 + \beta_2 X + \varepsilon$$ would give me $$\frac{dY}{dt} = \...
0
votes
1answer
97 views

What is the expression for derivative of the signum function one should use in the BP training method

The back propagation learning method requires knowing of derivatives of activation functions. But what expression one should use for signum activation function $$ \mathrm{Signum}( x ) = \...
1
vote
1answer
81 views

For back propagation in neural networks , how do we calculate vector by matrix derivative?

I am following the course deep learning ai by Andrew NG. In course1 week4, 04-06-Forward and Backward Propagation, he calculates backward propagation for layer $l$ in neural networks as follows (a ...
2
votes
1answer
48 views

Minimizing a least square [duplicate]

I'm a bit confused with matrix calculus. Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$ I don't see how this works. My plan was to ...
1
vote
1answer
57 views

What does $\mathcal{L}$ stands for in back propagation?

I am trying to learn how to do deep neural networks with this Ipython notebook. I'm puzzled about notations in linear backward learning section. For layer $l$, the linear part is: $Z^{[l]} = W^{[l]} ...
3
votes
0answers
38 views

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian. For the time being, I obtain the derivative as $\frac{\...
3
votes
1answer
43 views

Correlation between the j-feature and partial residual

I am reading a book for the lasso-regression. However, I think that this is a more general question regarding the derivative of the OLS-term: Can someone explain me why this $c_j$ term is ...
1
vote
0answers
158 views

LASSO: All KKT-Conditions

I'm trying to code the lasso algorithm via coordinate descent by myself. However, I want to implement a check for the KKT-conditions as in this blog. But what are all KKT conditions for the LASSO ? $...
1
vote
1answer
79 views

Derivative equal to zero in PCA

Source: presentation In PCA, we want to maximize variance $var(\alpha'_kX)$, i.e. $\alpha'_k \Sigma \alpha_k$ with respect to $\alpha' \alpha = 1$, where $\Sigma = X'X$. After introducing Lagrange ...
0
votes
1answer
188 views

Optimal neural net weights when using cross entropy loss

I'm trying to understand how cross-entropy works for finding the optimal weights in neural networks. According to Eli Bendersky's website and neural networks and deeplearning tutorial, we can find the ...
1
vote
1answer
189 views

Differentiation of RSS

I have the equation $RSS\left ( \beta \right )=\left ( y-X\beta \right )^{T}\left ( y-X\beta \right )$ that I'm trying to differentiate w.r.t. $\beta$. I used chain rule to get to $\frac{\partial ...
1
vote
0answers
210 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
3
votes
0answers
61 views

Second derivative test for machine learning algorithms [duplicate]

I have a question on second derivative test for most "modern" machine learning algorithms. I learned that in calculus but never seen it in real applications. Most machine learning algorithms ...