Questions tagged [derivative]

For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/

Filter by
Sorted by
Tagged with
0
votes
0answers
22 views

Backpropagation in an MLP with 1-hidden layer - What am I missing?

I want to do the math behind backpropagation of an MLP with 1 hidden layer and softmax output layer for a simple classification problem, and I can't for the life of me figure out what do I do wrong. ...
0
votes
0answers
16 views

What are the derivatives of Squared Exponential kernel function w.r.t. characteristic length scale (Gauss Process)

I'm writing a matlab code to implement Gaussian process. In the book: Gaussian Process for machine learning by Carl Edward Rasmussen and Christopher K. I. Williams, the authors define the squared ...
0
votes
0answers
16 views

Derivation of matrix derivative [duplicate]

In the ML lecture notes of cs229, I am not able to derive the equation 3 in matrix derivatives. I applied the equation 1 to derive it, considering the product after A as another matrix, I am not ...
0
votes
0answers
19 views

Is my step by step derivation of quadratic cost function's (Neural Networks) partial derivative with respect to some weights matrix correct?

I am trying to revise the details of a Multi-layer Perceptron with a set of weight matrices $\mathcal W$ and a set of bias vectors $\mathbf b$. Here is the quadratic cost function I am using, $$C(\...
0
votes
1answer
25 views

Derivative of all the parameters in Logistic Regression

$\mathcal{L}$ is the loss function, $\mathcal{L} = y_i \text{log} \sigma(z) + (1-y_i) \text{log} (1-\sigma(z))$, where $z = \sum_i w_ix_i$, with $w_i$ representing the weights and $x_i$ the features. ...
1
vote
1answer
16 views

n'th cumulant (of a CGF) for exponential family / exponential dispersion model

The n'th cumulant is defined to be the n'th derivative of the CGF (cumulant generating function). $$\kappa_n = \frac{d^n K(t)}{dt^n} |_{t=0} $$ But I'm reading in a book (p.215, chapter5, eq. 5.8) ...
1
vote
0answers
38 views

Posterior distribution for a Gaussian Process with a transformation in a gaussian likelihood

Suppose we are modelling observations y as follows. Our likelihood is normal $ y \sim \mathcal{N}(g(f(x)), \mathcal{I}\sigma^2)$, where $\mathcal{I}$ is the identity matrix and $g$ is some function ...
0
votes
0answers
21 views

Relation between dyi/dei and its leverage (hii)

Althought I have tried in different ways, I have not been able to show that $$\frac{\mathrm{d}Y_i}{\mathrm{d}e_i} = \frac{1}{1 - l_{ii}}$$ $e_i$ is equal to $l_{ii}$ is the $i-th$ element of the ...
1
vote
0answers
20 views

Is there a statistical test to determine correlation between the first derivative of two time based series?

I have two time series. They are normally distributed, parametric, exactly the same size and evenly spaced. I am seeking to determine if the rate of change of these two time series are correlated ...
3
votes
0answers
13 views

Meaning of a notation regarding mean square derivative

I'm reading a paper (On Differentiable Functionals, Van der Vaart, 1991, Annals of Statistics), and I've got a question regarding a notation in the following part: My Question: Does $dP^{1/2}$ mean $...
8
votes
4answers
178 views

Where does the logistic function come from?

I first learned the logistic function in machine learning course, where it is just a function that map a real number to 0 to 1. We can use calculus to get its derivative and use the derivative for ...
1
vote
1answer
25 views

Partial derivative of a linear regression with correlated predictors

Let's set up the situation of having some $Y$ that I think depends on a linear combination of $X_1$ and $X_2$. I could fit a regression model: $$y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2}$$ We ...
4
votes
1answer
40 views

Normal distribution “stable” under derivative?

Suppose that $\theta(t)\sim\mathcal N(\mu(t),\Sigma(t))$ where $t$ is some parameter. Then it holds that $$\theta(t) = \mu(t) + \Sigma(t)^{0.5}\xi$$ for $\xi\sim\mathcal N(0, I)$. I am interested in ...
0
votes
1answer
51 views

What do I miss in this derivation?

The school is closed due to the ongoing pandemic. And I am interested in the application of the Bayes Theorem in COVID-19. Here is what I thought. The total population in U.S. is approximately 327,...
1
vote
0answers
12 views

error propagation for derivatives

I have the following problem: I have some data of a function f(x) with a set of 300 values of it associated to the same number of values of x including corresponding standard deviation σ(f) for each ...
0
votes
0answers
35 views

Finding Derivative of a spline

I have received data values for a spline (which was already fit to some ndvi data). I just have only the data points of the spline and do not know the function that the spline follows. My goal is to ...
0
votes
1answer
26 views

How to derive the gradient of the reparameterized score function estimator?

In the paper Evolution Strategies as a Scalable Alternative to Reinforcement Learning, the authors derive the following gradient of the score function estimator $$ \begin{align} \nabla_\psi\mathbb E_{...
1
vote
0answers
10 views

rate of convergence for cross derivative estimation in local linear regression

Suppose $Y_{i}=m(X_{1i},X_{2i})+\epsilon_{i}$, with $E(Y_{i}|X_{1i},X_{2i})=m(X_{1i},X_{2i})$ where $m(\cdot,\cdot)$ is an unknown smooth function. If the estimator $\widehat{m}(x_{1},x_{2})$ is ...
0
votes
0answers
37 views

Finding the gradient $\nabla$ of the logistic regression cost function

I want to use vector calculus to derive the gradient $\nabla_wJ(w)$ of the logistic regression cost function $J(w) = -\textbf{y}\cdot ln\textbf{ s} - (\mathbf{1} - \textbf{y}) \cdot ln( \mathbf{1} - \...
3
votes
2answers
73 views

When using OLS on $\ln(y) = \beta_1 \ln(x) + \epsilon$, is $\beta_1$ the elasticity of $E[y\vert x]$, or the $y$ in the data (or both)?

Specifically, suppose we are estimating $$ \ln(y)=\beta_1\ln(x) + \epsilon $$ I understand that $\beta_1 = \frac{\partial \ln(y)}{\partial \ln(x)}$ which is the elasticity of $y$ with respect to $x$ ...
0
votes
0answers
37 views

Back-propagation through cross entropy or logistic loss function

I have neural network which ends with softmax function and I want to minimize cross-entropy cost function which takes output of this network and one-hot labels as arguments. To calculate partial ...
1
vote
1answer
43 views

Expression for Derivative of Hyperparameter of Kernel with respect to New Data

I would like to determine how the hyperparameter will change when a new data is observed and the GP is updated with this new data. Considering the following predictive distribution of the GP: $$\mu(...
0
votes
0answers
8 views

Computing custom gradient for LSTM equations

Consider an LSTM that takes in as input a sequence of N words $X_1,\cdots,X_N$. Each word is a vector $\in R^D$. The dimension of the LSTM neuron is $H$. Suppose we are doing sentiment classification ...
0
votes
1answer
47 views

Derivation of gradient-bandit algorithm, Why is the sum of the derivatives is zero?

https://www.cs.mcgill.ca/~dprecup/courses/RL/Lectures/2-bandits-2019.pdf In above pdf document, page 19, they explain by formula: $$\sum _{ b }^{ }{ \frac { \nabla { \Pi }_{ t }(b) }{ \nabla { H }_{ ...
0
votes
0answers
72 views

Gradient of multivariate normal distribution function?

Let $X\sim\mathcal{N}_J(\mu,\Sigma)$ be a multivariate normal with PDF $f_X$ and CDF $F_X$. Taking derivatives of $f_X$ wrt $X$, $\mu$ and $\Sigma$ is easy as shown here. However, I am interested in ...
1
vote
0answers
45 views

Relating Two Derivatives (and Elasticities) of a Log-Log Regression

Consider a standard "log-log" linear regression model like this: $\log(y_i) = \log(a_i + b_i)\delta + \epsilon_i$, where $y$ is the dependent variable, $a$ and $b$ are two independent variables, and ...
2
votes
1answer
194 views

Derivative of Gaussian Process (continued)

This is to extend the discussion of the derivative of the GP. The formulation provided in the previous post describes the gradient of GP as derivative of kernel function as follows with respect to $(x^...
0
votes
1answer
36 views

is it a good idea to take the derivative or integral of some features and add them as new features in machine learning?

I'm learning how to do feature Engineering and come across some ideas in my head that's why I want to ask if I had some dataset with some features let's say 2 features and I have a timestamp column ...
1
vote
1answer
68 views

Why does gradient descent HAVE to find the minimum as oppose to a change in the opposite direction

I have a question about the gradient descent step in neural networks. I fully understand the derivative step and taking the steps required to move in the direction that reduces the loss (finding the ...
3
votes
1answer
57 views

Derivation of score vector

Can anyone explain the process of this derivation, step by step? This derivation is from Joint Models for Longitudinal and Time-to Event Data by Dimitris Rizopoulos. \begin{equation} \begin{aligned} ...
0
votes
2answers
42 views

Source of vanishing/exploding gradients in RNN

Problem I am trying to understand the source of vanishing/exploding gradients in vanilla RNN. The update rule of vanilla RNN is $$ \begin{aligned} &\mathbf{a}^{\left<t\right>}=\...
0
votes
0answers
178 views

Deriving Gradient from negative log-likelihood function

I have been having some difficulty deriving a gradient of an equation. I have a Negative log likelihood function, from which i have to derive its gradient function. Negative log likelihood function ...
0
votes
0answers
30 views

Show that $\int^b_a\phi'''(z)dz$ lies between $\pm[\phi(0)+2\phi(\sqrt3)]$ for every $a<b$

Show that $\int^b_a\phi'''(z)dz$ lies between $\pm[\phi(0)+2\phi(\sqrt3)]$ for every $a<b$. $$\phi(z)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}z^2}$$ I've shown that: $$\phi''(z)=(z^2-1)\phi(z)$$ $$\...
1
vote
0answers
60 views

Computing gradients w.r.t input / Gaussian Process (RBF Kernel)

my intent is to write down an illustrative example of the derivatives w.r.t input from the MSE -> GP (with an RBF Kernel) -> Inputs $\frac{\partial MSE }{\partial x} (GP(x))$ Could anyone help me ...
0
votes
1answer
82 views

Derivative of reparametrization

I am stuck in one equation when reading the paper. Regardless of the context, here is the question: Let's say given an equation L($\sigma_g^2$, $\sigma_\epsilon^2$, $\beta$) = .... Now I want to use $\...
14
votes
3answers
410 views

How can I fit a spline to data that contains values and 1st/2nd derivatives?

I have a dataset that contains, let's say, some measurements for position, speed and acceleration. All come from the same "run". I could construct a linear system and fit a polynomial to all of those ...
1
vote
0answers
31 views

Meaning of the derivative of cross entropy wrt $p(x)$

Lets define the cross entropy between 2 probability distributions $p(x)$ and $q(x)$ as $$H(p,q) = -\sum p(x) \log{q(x)} $$ What would be the meaning of derivative of $H(p,q)$ when taking the ...
1
vote
0answers
55 views

First derivative of splines in bayesian model

Inspired by this post https://www.fromthebottomoftheheap.net/2014/05/15/identifying-periods-of-change-with-gams/, I'm trying to identify periods of change in a GAM model, using bayesian inference. ...
2
votes
1answer
68 views

how to derive eq 21.2.3 in BRML, Factor Analysis (Eigen-approach likelihood optimization)

how to derive eq 21.2.3 in BRML, chapter21 Factor Analysis? Log Likelihood function (eq. 21.1.13): $$ \log{p(\mathcal{V} | \mathbf{F}, \mathbf{\Psi})} = -\frac{N}{2}\left( \mathrm{trace}(\mathbf{\...
0
votes
1answer
123 views

What is the formula for the conditional variance when taking the derivative of a Gaussian process?

The formulae for the conditional mean and variance of a Gaussian process is given by equations (2.23) and (2.24): Also, the formula for the covariance of the derivative of a Gaussian process is given ...
0
votes
0answers
112 views

Quadratic Approximation of the binary logistic regression

I am using https://web.stanford.edu/~hastie/Papers/glmnet.pdf package to solve my optimization problem for the Binary Logistic Regression. On page 10 it is stated that the quadratic approximation of ...
1
vote
1answer
28 views

Updating Prediction Errors in Gradient Ascent (Friston's Free-energy)

Background In Rafal Bogacz's tutorial on the free-energy framework for modelling perception and learning, section 2.3 we have: $$\dot{\phi} = \frac{\partial F}{\partial\phi} = \varepsilon_u g'(\phi) ...
1
vote
0answers
16 views

What does this assumption mean regarding equal marginal densities?

Suppose that we have a random variable $\epsilon$ with density $q(\epsilon)$ and $w = t(\theta, \epsilon)$, where $t$ is a deterministic function of a constant $\theta$ and random variable $\epsilon$. ...
1
vote
1answer
129 views

Why do cubic splines need to be continuous at the first and second derivative, but discontinuous at the third?

I'm working through Introduction to Statistical Learning and came upon this: One can show that adding a term of the form $ \beta_4h(x,\xi)$ to the model (7.8) $$y_i = \beta_0 + \beta_1x_i + \...
1
vote
0answers
45 views

Propagating error when taking the derivative

I have a function that corresponds to a set of $(X,Y)$ coordinates with a Gaussian uncertainty ($\sigma_Y$) for each point. What I want to do is now compute the gradient of this function and the ...
0
votes
1answer
47 views

Backpropagation gradient of the average

In the Pytorch Udacity course, the following is said at one point: To calculate the gradients, you need to run the .backward method on a Variable, ...
2
votes
1answer
39 views

Derivative of expectation where the variable appears in the integration limit and in the integrand?

I want to calculate the derivative of $$\varphi(\mu) = \int_{-\infty}^{\mu} r(x-\mu) f(x)dx,$$ wrt to $\mu$, where $r$ is a function and $f$ is a density function. How can I account for the presence ...
0
votes
1answer
112 views

Calculate the derivative of the CDF with respect to the mean value [duplicate]

I want to derive the cumulative density function (cdf) for variables following normal distribution with respect to the parameters of the cdf (such as the mean or the standard deviation)
2
votes
1answer
130 views

Question about the gradient of weight normalization

In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$ \mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v $$ Then they ...
11
votes
1answer
854 views

Second order approximation of the loss function (Deep learning book, 7.33)

In Goodfellow's (2016) book on deep learning, he talked about equivalence of early stopping to L2 regularisation (https://www.deeplearningbook.org/contents/regularization.html page 247). Quadratic ...