Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

0
votes
0answers
18 views

Linear Regression Update Rule

Given the following quadratic loss function: $$\frac{1}{2n}\sum_{i \in [n]} \left[y^{(i)} - \sigma(w^\top x^{(i)})\right]^2$$ I'm trying to figure out the update rule for the weight $w$. $w:=w - \...
0
votes
0answers
10 views

How to obtain the functional derivative of variational distribution's parameters?

Referring to Collaborative variational autoencoder for recommender systems, I want to know how to derive the parameters for the variational distribution, in the Bayesian inference section of the paper....
1
vote
0answers
9 views

How to obtain the functional derivative of variational distribution?

Referring to , I want to know how to derive the parameters for the variational distribution, in the Bayesian inference section of the paper. I know how to derive, but I don't know how to deduct the ...
0
votes
0answers
13 views

Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
1
vote
0answers
12 views

How to analytically solve the probability of improvement acquisition function in Bayesian Optimization with Vector inputs?

I have been using the probability of improvement acquisition function in my Bayesian Optimization program, but I've run into a problem because I am not optimizing the acquisition function that quickly....
3
votes
2answers
53 views

When are the linear regression parameters of Y and X the same as the parameters of Y' and X'?

I am working on some simple linear modeling of a physical system and assumed that taking the derivative of an equation $$Y = \beta_1 + \beta_2 X + \varepsilon$$ would give me $$\frac{dY}{dt} = \...
1
vote
0answers
9 views

Get partial derivative in pytorch [closed]

coords[i] is a list containing 3 elements x,y,z and I want to get the derivative of G[i] w.r....
0
votes
1answer
12 views

What is the expression for derivative of the signum function one should use in the BP training method

The back propagation learning method requires knowing of derivatives of activation functions. But what expression one should use for signum activation function $$ \mathrm{Signum}( x ) = \...
0
votes
0answers
20 views

pseudo R2 as xgboost objective function

I want to use a custom objective function with xgboost: 1 - (log(y) - log(p)) / (log(y) - log(q)) y = true value, p = my probabilities, q = some other base ...
1
vote
0answers
27 views

For back propagation in neural networks , how do we calculate vector by matrix derivative?

I am following the course deep learning ai by Andrew NG. In course1 week4, 04-06-Forward and Backward Propagation, he calculates backward propagation for layer $l$ in neural networks as follows (a ...
2
votes
1answer
29 views

Minimizing a least square [duplicate]

I'm a bit confused with matrix calculus. Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$ I don't see how this works. My plan was to ...
1
vote
1answer
47 views

What does $\mathcal{L}$ stands for in back propagation?

I am trying to learn how to do deep neural networks with this Ipython notebook. I'm puzzled about notations in linear backward learning section. For layer $l$, the linear part is: $Z^{[l]} = W^{[l]} ...
3
votes
0answers
28 views

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian. For the time being, I obtain the derivative as $\frac{\...
3
votes
1answer
35 views

Correlation between the j-feature and partial residual

I am reading a book for the lasso-regression. However, I think that this is a more general question regarding the derivative of the OLS-term: Can someone explain me why this $c_j$ term is ...
0
votes
0answers
16 views

Deriving the gradient vector of a Probit model

Consider the probit regression model where the pdf of $y_{i}$ is $$f(y_{i};\mathbf{\beta}) = \mu_{i}^{y_{i}}\left ( 1 - \mu_{i} \right )^{1 - y_{i}},$$ where $y_{i} = \left \{ 0,1 \right \}$ and $\...
1
vote
0answers
11 views

Comparing Importance of Values to Importance in Change of Values [closed]

I am working with some hourly weather data and I am trying to find how strongly variables correlate and which variables are more important that others. Along with the actual weather data (temperatures ...
1
vote
0answers
52 views

LASSO: All KKT-Conditions

I'm trying to code the lasso algorithm via coordinate descent by myself. However, I want to implement a check for the KKT-conditions as in this blog. But what are all KKT conditions for the LASSO ? $...
1
vote
1answer
36 views

Derivative equal to zero in PCA

Source: presentation In PCA, we want to maximize variance $var(\alpha'_kX)$, i.e. $\alpha'_k \Sigma \alpha_k$ with respect to $\alpha' \alpha = 1$, where $\Sigma = X'X$. After introducing Lagrange ...
0
votes
1answer
100 views

Optimal neural net weights when using cross entropy loss

I'm trying to understand how cross-entropy works for finding the optimal weights in neural networks. According to Eli Bendersky's website and neural networks and deeplearning tutorial, we can find the ...
1
vote
1answer
42 views

Differentiation of RSS

I have the equation $RSS\left ( \beta \right )=\left ( y-X\beta \right )^{T}\left ( y-X\beta \right )$ that I'm trying to differentiate w.r.t. $\beta$. I used chain rule to get to $\frac{\partial ...
1
vote
0answers
96 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
3
votes
0answers
53 views

Second derivative test for machine learning algorithms [duplicate]

I have a question on second derivative test for most "modern" machine learning algorithms. I learned that in calculus but never seen it in real applications. Most machine learning algorithms ...
2
votes
0answers
37 views

Derivative of a conditional CDF w.r.t. the condition [closed]

I would like to ask you for a help with my problem. I have random variable $$X = X_1 + \sum_{i = 2}^{n}(X_i - y_i)^{+}, \ \ \ (.)^{+} = \max(0, . ), $$ where $X_i$ are independent, continuous with ...
0
votes
1answer
43 views

Backpropagation - partial derivatives

I am currently reading the Neural networks and deep learning book by Michael Nielsen. I have a question regarding the backpropagation chapter: Background: He explains the influence of a neuron on ...
0
votes
0answers
26 views

Partial derivatives to interpret regressions with interaction terms

I'm working in this model: glm(sells ~ clicks.sqrt + day + day.sqrt + clicks.sqrt * day.sqrt, family = poisson) I chose poisson reg because I need to count (...
1
vote
1answer
91 views

Derivative of bivariate normal CDF with common mean parameters

I am trying to calculate a derivative of the form $\frac{d}{dz}\Phi_2(\mu_1(z),\mu_2(z),\rho)$ where $\Phi_2$ is the standard bivariate normal CDF. I am thinking it might be an application of a ...
1
vote
0answers
215 views

Gradient boosting for binary outcome - terminal nodes estimate (using R gbm)

I have been searching for the answer for the below query quite a long time and found a few answers (see: interpretation of gbm single tree prediction in pretty.gbm.tree or R Package GBM - Bernoulli ...
0
votes
0answers
67 views

Derivatives of the multivariate normal CDF w.r.t. its parameters

Let $\boldsymbol X \sim \mathcal N_d(\boldsymbol\mu, \boldsymbol\Sigma)$ and let $f_{\boldsymbol X}(\boldsymbol x)$ be its PDF. The CDF is $$ F_{\boldsymbol X}(\boldsymbol q) = \int_{-\infty}^{q_1} \...
1
vote
1answer
454 views

Working for Logistic regression partial derivatives

In Andrew Ng's Neural Networks and Deep Learning course on Coursera the logistic regression loss function for a single training example is given as: $$ \mathcal L(a,y) = - \Big(y\log a + (1 - y)\log (...
1
vote
2answers
112 views

Is computing natural gradient equivalent to deriving directional derivative?

It seems to me that natural gradient is simply derived from directional derivative. For example, for a vector $v$, $\tilde{\nabla} f \cdot v = G^{-1} \nabla f \cdot v = \lim_{h\to0} \frac{f(x+hv)-f(x)...
0
votes
1answer
44 views

Why do derivative of weights for sigmoid binary output classification layer depend on the previous layer?

I've seen the derivative of the weights for a sigmoid binary output classification layer written as: $\frac{\partial \mathcal{J}}{\partial W^{[i]}} = \frac{1}{m}\frac{\partial \mathcal{J}}{\partial Z^...
11
votes
1answer
616 views

Interpretation of Radon-Nikodym derivative between probability measures?

I have seen at some points the use of the Radon-Nikodym derivative of one probability measure with respect to another, most notably in the Kullback-Leibler divergence, where it is the derivative of ...
0
votes
0answers
53 views

Calculating derivatives of log-likelihood function (for new GAMLSS distribution)

I have a dataset that is very closely fit by the discrete 3-parameter Burr Type XII distribution (outlined at http://medcraveonline.com/BBIJ/BBIJ-04-00092.pdf). I want to use the GAMLSS package in R ...
2
votes
0answers
18 views

Please help me with this calculus problem involving use of differentiation under integral [closed]

I am really confused in this problem. I know how to take derivatives of single integral. You plug in the upper limit in the function and then take the derivatives of upper limit and subtract it from ...
0
votes
1answer
125 views

Partial Derivative of Standard Trivariate Normal CDF

I am trying to get partial derivative of standard trivariate normal cdf with respect to $x_1$. i.e. I would like to get $ \frac{\partial}{\partial x_1}\Phi(x_1,x_2,x_3;r_{12},r_{13},r_{23})$. $\Phi(...
3
votes
0answers
862 views

Finding PDF from CDF

I just got really unsure, can someone confirm/rectify? I have the CDF defined as $F(x)= \begin{cases}0, &\text{if}~x < 0,\\ 4x^2 &\text{if}~ 0 \leq x < \frac{1}{4} \\ 1-\frac{4}{3}(1-x)^...
4
votes
1answer
92 views

Second directional derivate and Hessian matrix

I am reading the following from the book Deep Learning, and I have the following questions. I don't quite understand second directional derivatives. The first directional derivative of a function $f:\...
1
vote
1answer
114 views

Maximizing (and derivating) log-likelihood of penalized logistic regression

I'm trying to solve Exercise 18.3 of "Elements of Statistical learning" by Hastie et al. and I'd be really grateful for any hints. Show that the fitted coefficients for the regularized multiclass ...
2
votes
0answers
49 views

Working out the derivatives in backproagation

So I have no calculus experience what so ever and I've been tasked to build a neural network so finding the derivatives is proving quite problematic with my limited calculus experience. I've got the ...
0
votes
1answer
30 views

Linear regression: Why derviative = 0 is the minimum for OLS

I am new in this field, but I wanna advance fast and wan't to have a complete picture of everything. I have a very simple question, but I guess I am missing something in imagining the whole picture. ...
0
votes
0answers
66 views

Show Nadaraya-Watson Kernel Estimator with Gaussian Kernel is Differentiable

I asked this question on math.stack.exchange but thought it might be more appropriate on here. I am wondering how to prove the Nadaraya-Watson Kernel Estimator with a Gaussian Kernel is differentiable....
3
votes
0answers
92 views

Fitting a Multivariate ARMA-GARCH model

I am considering a multivariate time series. I denote the general term of this multivariate time series by $Y_t = \left[y_t^{(1)}, \ldots, y_t^{(k)}\right]^T \in \mathbb{R}^{k \times 1}$. For some ...
1
vote
1answer
1k views

Mean Absolute Error (MAE) derivative

$MAE=|y_{pred} - y_{true}|$ $\dfrac{dMAE}{dy_{pred}} = ?$ I'm trying to understand how MAE works as a loss function in neural networks using backpropogation. I know it can be used directly in some ...
0
votes
0answers
90 views

simultaneous confidence bands for derivative of density

There is lots of attention on confidence bands for regression functions, but I see not many for densities and even less for derivative of density. In particular, I would like to know some theory and ...
0
votes
0answers
71 views

Relating coefficients: partial derivative of ratio vs. ratio of partial derivatives, with logs

Consider the following regression: $$y_{ij}=\beta_a \times \log(A_{ij}) + \beta_b \times \log(B_{j}) + \tau_j + \mu_{ij}$$ where y, A and B are variables that vary at the $ij$ and $j$ levels ...
2
votes
1answer
60 views

How to simplify the derivation of the inverse cdf yielded from l'Hospital rule?

I am currently dealing with a proof of Pauline Barrieu`s Paper "Assessing Financial Model Risk" (page 19). At one point she applys the l'Hospital rule on a limit equation. We have some cumulative ...
1
vote
1answer
52 views

MLE estimation case

Consider a case that we see {H,H} in two coin tossing. We model this with binomial distribution as: $P(D|x) = x^2*(1-x)^0 = x^2$ Now, if we want to compute x based on MLE: $Log(P(D|x)) = 2Logx$. and ...
0
votes
1answer
57 views

Method to compute empirical derivative about some point

I have black-box access to some function and I want to compute the derivative about the point X. Is there a method that does this?
0
votes
1answer
258 views

What is the difference in “weight update process” in gradient descent vs Stochastic gradient descent?

Question In normal GD the weights are updated for every row in the training dataset while in SGD the weights are updated only once for the mini batch based on cummulative dLoss/dw1, dLoss/dw2 . Is my ...
0
votes
1answer
2k views

How L2 Regularization changes backpropogation formulas

I am going through online deep learning book and trying to recreate Neural Network that was written there with a bit of different class designs. However, I've run into a problem, where when using L2 ...