Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [derivative]

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

0
votes
0answers
13 views

Why are the non-diagonals in a Jacobian zeros? [migrated]

From the Matrix Calculus for Deep Learning, in the "Derivatives of vector element-wise binary operators" section, it says Any time the general function is a vector, we know that $f_i(w)$ reduces to ...
0
votes
0answers
4 views

need help with matrix calculus [migrated]

I am trying to find $\frac{\partial (x'Ax)}{\partial x}$ where x is a vector (2 x 1 vector) and A is a matrix (say 2x2 dimensions). When I looked up in http://www.matrixcalculus.org/ I found the ...
2
votes
1answer
49 views

Finding expression of $n$-th derivative, when $n$ is large

For completeness, assume $C$ is an Archimedean copula with some generator function $\varphi$, which is usually assumed to have nice properties. It is known that $$ C(u_1, u_2, \ldots, u_n)=\varphi^{-1}...
2
votes
0answers
25 views

Derivatives of quantile loss function [duplicate]

I'm reading a text - Roger Koenker (2005) Quantile Regression [page 8] - that goes like this: Consider the function $$R(\xi) = \sum_{i=1}^n \rho_\tau(y_i-\xi)$$ where $$\rho_\tau(y_i-\xi) =(y_i-\xi)(\...
0
votes
1answer
19 views

Small calculation in Max entropy

I am wondering where the -1 comes from in the derivatives with respect to the probability. For example, these notes. For the basic example with the uniform distribution. We want to minimze $[-\int ...
-1
votes
0answers
15 views

In optimization, what is the purpose of Gateaux and Frechet derivatives?

When reading Optimization related papers, I sometimes come across terms such as "Gateaux" or "Frechet" derivatives. I'm often puzzled as to why we need these definitions, when we already know the ...
0
votes
0answers
13 views

Derivative of Time-Transformed Stochastic Process

Given a continuous time stochastic process X(t), we can define the functional transformation, $$f(X)(t)=(X(t))^2−2X(t)$$ and evaluate the Hadamard derivative. Given a transformation on the real ...
4
votes
2answers
43 views

Slope of Curve with Unknown Functional Form

I have a monotonically-increasing curve whose functional form is not known a priori and would like to compute the curve's slope at the rightmost endpoint. Typically, when the functional form is known, ...
2
votes
1answer
93 views

Computing gradients via Gaussian Process Regression

I have a set of noisy data that I am fitting using Gaussian Process Regression via Python's sklearn package. The posterior mean of the GP is essentially my output with an associated error. Based on ...
0
votes
1answer
56 views

How are biases updated when 'batch size' > 1?

This is my network represented in matrices: (a dot represents an arbitrary number) Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity) Backpropagation The ...
0
votes
2answers
80 views

How to calculate the derivative of crossentropy error function?

I'm reading this tutorial (presented below) on computing derivative of crossentropy. The author used the loss function of logistic regression I think. https://www.dropbox.com/s/rxrtz3auu845fuy/...
2
votes
1answer
41 views

Gaussian log-density variational derivative

In Appendix C.1 of 'Taming VAEs' paper, the authors need to compute the functional derivative $$\frac{\delta}{\delta g\left( z \right)} \mathbb{E}_{q\left(z\mid x\right)} \left[(g \left( z \right) - ...
0
votes
0answers
79 views

Fast RCNN ROI backpropagation

In the Fast R-CNN paper, in Section 2.3 - "Back-propagation through ROI pooling layers", the following formula is being used for calculating the partial derivative of the activation input to the ROI ...
0
votes
0answers
19 views

Linear Regression Update Rule

Given the following quadratic loss function: $$\frac{1}{2n}\sum_{i \in [n]} \left[y^{(i)} - \sigma(w^\top x^{(i)})\right]^2$$ I'm trying to figure out the update rule for the weight $w$. $w:=w - \...
1
vote
0answers
13 views

How to obtain the functional derivative of variational distribution?

Referring to , I want to know how to derive the parameters for the variational distribution, in the Bayesian inference section of the paper. I know how to derive, but I don't know how to deduct the ...
1
vote
0answers
47 views

Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
1
vote
0answers
23 views

How to analytically solve the probability of improvement acquisition function in Bayesian Optimization with Vector inputs?

I have been using the probability of improvement acquisition function in my Bayesian Optimization program, but I've run into a problem because I am not optimizing the acquisition function that quickly....
3
votes
2answers
56 views

When are the linear regression parameters of Y and X the same as the parameters of Y' and X'?

I am working on some simple linear modeling of a physical system and assumed that taking the derivative of an equation $$Y = \beta_1 + \beta_2 X + \varepsilon$$ would give me $$\frac{dY}{dt} = \...
1
vote
0answers
47 views

Get partial derivative in pytorch [closed]

coords[i] is a list containing 3 elements x,y,z and I want to get the derivative of G[i] w.r....
0
votes
1answer
23 views

What is the expression for derivative of the signum function one should use in the BP training method

The back propagation learning method requires knowing of derivatives of activation functions. But what expression one should use for signum activation function $$ \mathrm{Signum}( x ) = \...
0
votes
0answers
68 views

pseudo R2 as xgboost objective function

I want to use a custom objective function with xgboost: 1 - (log(y) - log(p)) / (log(y) - log(q)) y = true value, p = my probabilities, q = some other base ...
1
vote
0answers
36 views

For back propagation in neural networks , how do we calculate vector by matrix derivative?

I am following the course deep learning ai by Andrew NG. In course1 week4, 04-06-Forward and Backward Propagation, he calculates backward propagation for layer $l$ in neural networks as follows (a ...
2
votes
1answer
37 views

Minimizing a least square [duplicate]

I'm a bit confused with matrix calculus. Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$ I don't see how this works. My plan was to ...
1
vote
1answer
51 views

What does $\mathcal{L}$ stands for in back propagation?

I am trying to learn how to do deep neural networks with this Ipython notebook. I'm puzzled about notations in linear backward learning section. For layer $l$, the linear part is: $Z^{[l]} = W^{[l]} ...
3
votes
0answers
34 views

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian. For the time being, I obtain the derivative as $\frac{\...
3
votes
1answer
38 views

Correlation between the j-feature and partial residual

I am reading a book for the lasso-regression. However, I think that this is a more general question regarding the derivative of the OLS-term: Can someone explain me why this $c_j$ term is ...
0
votes
0answers
25 views

Deriving the gradient vector of a Probit model

Consider the probit regression model where the pdf of $y_{i}$ is $$f(y_{i};\mathbf{\beta}) = \mu_{i}^{y_{i}}\left ( 1 - \mu_{i} \right )^{1 - y_{i}},$$ where $y_{i} = \left \{ 0,1 \right \}$ and $\...
1
vote
0answers
11 views

Comparing Importance of Values to Importance in Change of Values [closed]

I am working with some hourly weather data and I am trying to find how strongly variables correlate and which variables are more important that others. Along with the actual weather data (temperatures ...
1
vote
0answers
97 views

LASSO: All KKT-Conditions

I'm trying to code the lasso algorithm via coordinate descent by myself. However, I want to implement a check for the KKT-conditions as in this blog. But what are all KKT conditions for the LASSO ? $...
1
vote
1answer
41 views

Derivative equal to zero in PCA

Source: presentation In PCA, we want to maximize variance $var(\alpha'_kX)$, i.e. $\alpha'_k \Sigma \alpha_k$ with respect to $\alpha' \alpha = 1$, where $\Sigma = X'X$. After introducing Lagrange ...
0
votes
1answer
149 views

Optimal neural net weights when using cross entropy loss

I'm trying to understand how cross-entropy works for finding the optimal weights in neural networks. According to Eli Bendersky's website and neural networks and deeplearning tutorial, we can find the ...
1
vote
1answer
69 views

Differentiation of RSS

I have the equation $RSS\left ( \beta \right )=\left ( y-X\beta \right )^{T}\left ( y-X\beta \right )$ that I'm trying to differentiate w.r.t. $\beta$. I used chain rule to get to $\frac{\partial ...
1
vote
0answers
154 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
3
votes
0answers
58 views

Second derivative test for machine learning algorithms [duplicate]

I have a question on second derivative test for most "modern" machine learning algorithms. I learned that in calculus but never seen it in real applications. Most machine learning algorithms ...
2
votes
0answers
64 views

Derivative of a conditional CDF w.r.t. the condition [closed]

I would like to ask you for a help with my problem. I have random variable $$X = X_1 + \sum_{i = 2}^{n}(X_i - y_i)^{+}, \ \ \ (.)^{+} = \max(0, . ), $$ where $X_i$ are independent, continuous with ...
0
votes
1answer
47 views

Backpropagation - partial derivatives

I am currently reading the Neural networks and deep learning book by Michael Nielsen. I have a question regarding the backpropagation chapter: Background: He explains the influence of a neuron on ...
0
votes
0answers
30 views

Partial derivatives to interpret regressions with interaction terms

I'm working in this model: glm(sells ~ clicks.sqrt + day + day.sqrt + clicks.sqrt * day.sqrt, family = poisson) I chose poisson reg because I need to count (...
1
vote
1answer
155 views

Derivative of bivariate normal CDF with common mean parameters

I am trying to calculate a derivative of the form $\frac{d}{dz}\Phi_2(\mu_1(z),\mu_2(z),\rho)$ where $\Phi_2$ is the standard bivariate normal CDF. I am thinking it might be an application of a ...
1
vote
0answers
309 views

Gradient boosting for binary outcome - terminal nodes estimate (using R gbm)

I have been searching for the answer for the below query quite a long time and found a few answers (see: interpretation of gbm single tree prediction in pretty.gbm.tree or R Package GBM - Bernoulli ...
0
votes
0answers
117 views

Derivatives of the multivariate normal CDF w.r.t. its parameters

Let $\boldsymbol X \sim \mathcal N_d(\boldsymbol\mu, \boldsymbol\Sigma)$ and let $f_{\boldsymbol X}(\boldsymbol x)$ be its PDF. The CDF is $$ F_{\boldsymbol X}(\boldsymbol q) = \int_{-\infty}^{q_1} \...
2
votes
1answer
888 views

Working for Logistic regression partial derivatives

In Andrew Ng's Neural Networks and Deep Learning course on Coursera the logistic regression loss function for a single training example is given as: $$ \mathcal L(a,y) = - \Big(y\log a + (1 - y)\log (...
1
vote
2answers
144 views

Is computing natural gradient equivalent to deriving directional derivative?

It seems to me that natural gradient is simply derived from directional derivative. For example, for a vector $v$, $\tilde{\nabla} f \cdot v = G^{-1} \nabla f \cdot v = \lim_{h\to0} \frac{f(x+hv)-f(x)...
0
votes
1answer
97 views

Why do derivative of weights for sigmoid binary output classification layer depend on the previous layer?

I've seen the derivative of the weights for a sigmoid binary output classification layer written as: $\frac{\partial \mathcal{J}}{\partial W^{[i]}} = \frac{1}{m}\frac{\partial \mathcal{J}}{\partial Z^...
11
votes
1answer
745 views

Interpretation of Radon-Nikodym derivative between probability measures?

I have seen at some points the use of the Radon-Nikodym derivative of one probability measure with respect to another, most notably in the Kullback-Leibler divergence, where it is the derivative of ...
0
votes
0answers
74 views

Calculating derivatives of log-likelihood function (for new GAMLSS distribution)

I have a dataset that is very closely fit by the discrete 3-parameter Burr Type XII distribution (outlined at http://medcraveonline.com/BBIJ/BBIJ-04-00092.pdf). I want to use the GAMLSS package in R ...
0
votes
1answer
153 views

Partial Derivative of Standard Trivariate Normal CDF

I am trying to get partial derivative of standard trivariate normal cdf with respect to $x_1$. i.e. I would like to get $ \frac{\partial}{\partial x_1}\Phi(x_1,x_2,x_3;r_{12},r_{13},r_{23})$. $\Phi(...
3
votes
0answers
2k views

Finding PDF from CDF

I just got really unsure, can someone confirm/rectify? I have the CDF defined as $F(x)= \begin{cases}0, &\text{if}~x < 0,\\ 4x^2 &\text{if}~ 0 \leq x < \frac{1}{4} \\ 1-\frac{4}{3}(1-x)^...
4
votes
1answer
116 views

Second directional derivate and Hessian matrix

I am reading the following from the book Deep Learning, and I have the following questions. I don't quite understand second directional derivatives. The first directional derivative of a function $f:\...
1
vote
1answer
140 views

Maximizing (and derivating) log-likelihood of penalized logistic regression

I'm trying to solve Exercise 18.3 of "Elements of Statistical learning" by Hastie et al. and I'd be really grateful for any hints. Show that the fitted coefficients for the regularized multiclass ...
2
votes
0answers
52 views

Working out the derivatives in backproagation

So I have no calculus experience what so ever and I've been tasked to build a neural network so finding the derivatives is proving quite problematic with my limited calculus experience. I've got the ...