Questions tagged [derivative]
For on-topic questions involving the mathematical concept of a derivative, i.e. $\frac{d}{dx} f(x)$. For purely mathematical questions about the derivative it is better to ask on math SE https://math.stackexchange.com/
301
questions
0
votes
0
answers
20
views
Neural network learns to mimic distribution of classes in dataset instead of using signal from input
I'm trying to implement example from a classic AI paper named "Learning representations by back-propagating errors" by Hinton et al.
Example aims at training network able to predict third ...
1
vote
1
answer
23
views
Why does the score test work for values longer in the tail that have a small log-likelihood derivative?
The score test says that we take the derivative of the log-likelihood at $H_0$ and divide it by the fisher information at $H_0$.
$U(\theta )={\frac {\partial \log L(\theta \mid x)}{\partial \theta }}.$...
3
votes
0
answers
72
views
Derivative-based effect size for Gaussian GAMs
It is often the realistic advice I have seen here that Gaussian GAMs are not regressions with which you can easily approximate an effect size for, as the effects are of course non-linear. However, we ...
- 8,655
0
votes
0
answers
10
views
How to normalize d1 and d2 when using backward finite difference approximation?
My goal is to normalize the first and/or second derivative when approximated from a backward finite difference, where the normalized value is made distinct for some segment of the line (such as ${f'(x)...
- 109
0
votes
0
answers
44
views
Understanding relation between axis of least and maximum second moment
I was going through computer vision lecture video. You can find the pdf of this lecture here.
I was trying to understand how orientation of object corresponds to axis of least second moment aka ...
- 215
0
votes
0
answers
69
views
Is the equation of the derivative of a loss function relative to the input to the Sigmoid (z) the same whether computed backward or forward?
I am referring to the derivative of the binary cross-entropy loss function for logistic regression. Using back-propagation, the derivative of the loss function L ...
- 133
1
vote
0
answers
19
views
Monotonicity of softmax (considering updates from all variables)
There's a relevant question here that doesn't quite answer my question, but I'm unable to comment.
Define softmax to be $$a_i = \text{softmax}(u_i)= \frac{e^{u_i}}{\sum_j{e^{u_j}}}$$
As the linked ...
- 11
1
vote
0
answers
78
views
Understanding Backpropagation with Softmax and Quadratic Error
I'm trying to understand how to compute the derivative of the Softmax activation function in order to compute the gradient of the quadratic cost function w.r.t. the weights of the last layer.
...
0
votes
0
answers
29
views
Problem for a math formula in Weight Uncertainty in Neural Network
I am studying the paper "Weight Uncertainty in Neural Networks" by Blundell et al (2015, on arXiv), and there is a formula I don't get page 4, namely formula (3) in step 5:
I don't ...
- 1
0
votes
0
answers
27
views
Derivation of Truncated Derivatives in Hochreiter's original LSTM Paper
I was trying to understand the derivation of the truncated derivatives in Hochreiter's origin LSTM paper. Spent a bunch of time without avail. I suppose I am missing a way to map my intuition of ...
4
votes
1
answer
183
views
Rewriting the expectation of f(x) by means of its derivative
I have a question regarding this proposition.
Let $f: \mathbb{R} \rightarrow \mathbb{R}$ be an a.e. differentiable function so that $\int \frac{\left|f^{\prime}(x)\right|}{(1+|x|)^s} d x<\infty$ ($...
- 309
1
vote
2
answers
102
views
First derivative of multivariate normal density with exchangeable correlation structure
As part of a proof, I need to take the first derivative of the log of the following multivariate normal density: $(2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left(\frac{-1}{2} x'\Sigma^{-1}x\right)$.
In this ...
- 714
0
votes
0
answers
31
views
Gradient vs differences to remove non-stationarity in time series?
When dealing with non-stationary time series (for instance, in auto-correlation analysis), differencing (computing absolute differences between consecutive samples/observations) is often regarded as ...
- 141
1
vote
0
answers
20
views
Elasticity estimates for zero-truncated negative binomial part in the hurdle model
I estimated a hurdle negative binomial regression model with zero-truncated negative binomial model as the count component in R using the pscl package. I wish to present elasticities for the count ...
- 11
0
votes
1
answer
23
views
Useful value for rate of change
I want to create a simple representation of "rate of change" for a number of different metrics, which aren't really comparable between each-other. To illustrate my problem, let's say I have ...
- 121
1
vote
0
answers
39
views
Propagation of uncertainty in a derivative of a function [duplicate]
I've performed an ordinary least squares on a data set with one variable. For simplicity, let's say I've fitted a polynomial function
$$f(x)=a+bx+cx^2+dx^3.$$
I obtain the best fit and the standard ...
- 11
2
votes
2
answers
151
views
Probability Generating Function for The Difference of Two Binomially Distributed Random Variables?
Suppose I have 2 random variables:
$X\sim \textrm{Bin}(m,p_1)$ and $Y\sim \textrm{Bin}(n,p_2).$
I want to find the distribution of $S=X-Y$ using the probability generating function ($PGF$) treating $S$...
- 41
0
votes
0
answers
35
views
Trying to understand the derivation in the Information Bottleneck Method
I'm trying to understand the proofs on the The information bottleneck method paper by Tishby, Pereira and Bialek without luck. In particular, the second term in the functional derivative of the ...
- 103
1
vote
0
answers
64
views
Taking derivative of a function containing random variable wrt the variance of that variable [closed]
Say, I have a function containing a random variable such as $ f(X)$, where $X $ is the random variable that comes from a family of random variables that differ only in the first and second moments (e....
- 11
2
votes
1
answer
118
views
When are $f(X)$ and $f'(X)$ independent?
I want to know the effect of differentiation on the independence of random variables. For a random variable $X$, when are $f^{(n)}(X)$ and $f^{(n+k)}(X)$ independent?, $\forall n\geq0\;, k\geq 1$.
- 21
0
votes
0
answers
15
views
Are some gradient weights equal?
I want to create a 3 layers neural network from scratch to perform linear regression.
The first and the second layer have 2 neurons, and the last layer has one neuron.
Feature vector x is divided into ...
- 1
2
votes
1
answer
73
views
How to express backpropagation dE/dV using matrix
I'm new in NN and my math is not that good. I try to do manual calculation using NN model. I already know and try to calculate the feedforward and backward one by one using the formula. but when I try ...
- 21
1
vote
1
answer
75
views
The confusing derivation in the book 'Machine Learning: A Probabilistic Perspective' by Kevin P. Murphy
In the section 15.5 of the book 'Machine Learning: A Probabilistic Perspective' by Kevin P. Murphy, it discusses the Gaussian Process Latent Variable Model. The log-likelihood objective function is ...
- 11
0
votes
1
answer
48
views
Derivative conditional moments binary random variable
Let $e$ be a continuously distributed RV with pdf $f$ and let $q( x )$ be a binary RV that depends on the former through the relation $q ( x ) = 1[h( x ) \geq e ]$, where $h$ is a well-behaved ...
- 70
1
vote
1
answer
68
views
What is the derivative of a set or a string? [closed]
Neural networks operate on numbers, and it's well-known what the derivative of numeric functions are, as well as what the derivative of matrix functions are.
What about functions that operate on maps ...
- 111
0
votes
1
answer
126
views
Is the Inverse Mills Ratio Strictly Decreasing?
As far as I know, the Inverse Mills ratio, $\lambda(x)=\phi(x)/\Phi(x)$, is decreasing in $x$. Thus, I am curious now whether $\lambda(x)$ is in fact strictly decreasing in $x$.
To see this, I derived ...
- 303
0
votes
1
answer
76
views
Derivative of an integral of a random normal variable
Let $x$ be a random normal variable with pdf $h(x)$ and CDF $H(x)$. Also let $\alpha$ be a constant, and $x^\star$ a variable.
I am trying to take the following derivative:
$$\frac{d}{d x^\star} \bigg ...
- 105
0
votes
0
answers
36
views
Is it possible to find an explicit equation for this maximum likelihood for a particular variable of mean function
I have a log-likelihood equation that involves multivariate normal. Let's say, $le = \sum_{i=1}^n logf(y_i)$ and
$f(y)=(2\pi)^{-\dfrac{n}{2}}|\sigma^2I_n|^{-\dfrac{1}{2}}exp[-\dfrac{1}{2}(y-x(t))^T|\...
0
votes
0
answers
57
views
Why are ranking loss functions hard to optimize?
Currently I am learning about ranking loss functions, in particular the normalized cumulative gain, which can be written as:
$NDCG = \frac{DCG}{IDCG}$
where DCG is defined as
$DCG = \sum_{i=1}^N \frac{...
- 505
0
votes
0
answers
17
views
Neural Network training as non stationary stateless continuous reinforcement learning problem
Say I have a neural network denoted as f(\theta), and we want to optimize $\theta$.
What I thought is that $\theta$ can be seen as an action sampled from a ...
- 859
0
votes
0
answers
91
views
derivatives and distribution of a 3-dimensional copula in R
I am looking for a way to calculate in the R software, the distribution, the density and the derivatives (of order 1, 2) partial of a Gaussian copula of dimension 3.
Indeed, I have three variables (u1,...
- 1
0
votes
1
answer
109
views
How did we derive the least square estimator using OLS?
How does multiplying a matrix with its transpose equal "minimizing" it?
When calculating the partial derivative, where does the X' come from?
Why setting the value of third equation to 0 is ...
- 1
2
votes
1
answer
252
views
Derivative error with respect to bias in binary cross entropy
I will do research using NN with 1 hidden layer. To calculate loss using binary cross entropy and for the activation function using sigmoid. I found the derivative formula from Sadowski, 2016 (link: ...
- 57
2
votes
1
answer
110
views
What happens if we follow the gradient of a softmax activation
Given a softmax output layer, what does it mean to "follow the gradient"? Usually that would consist in "increasing the output" but obviously the softmax has no notion of "...
- 859
0
votes
0
answers
48
views
Ask for help understanding a sentence in Bishop's PRML book on soft weight sharing
It's about section 5.5.7 of Christopher M. Bishop's "Pattern Recognition and Machine Learning" on soft weight sharing. The sentence is the first three lines of page 271. First the author ...
- 303
0
votes
0
answers
97
views
four-point forward-difference formula using Newton's form for first order derivative [closed]
We know that ${f'(x) \approx \frac{f(x+h)- f(x)}{h}}$. If we have three points ${x_0 = x-h}$, ${x_1 = x}$, ${x_2 = x + h}$, we can compute the 3-point centered-difference formula using the Newton's ...
- 217
0
votes
0
answers
26
views
Derivative of multivariate normal cdf with respect to it’s arguments [duplicate]
I'm using a result from the dissertation of Poddar(2016, link) and he states the following in his appendix A1: We will use the well known property, stated here for completeness, of the multivariate ...
- 65
1
vote
0
answers
59
views
Please help me to understand the Taylor’s theorem when transiting from Gradient Boost to XGboost
I am reading this article, which explains how the algorithm replaces the actual loss function with so-called 2nd order Taylor expansion. I can understand til Step 4, and can't understand step 5.
I ...
- 109
1
vote
1
answer
26
views
A question on computational complexity of a numerical differentiation (equation (5.77)) in Bishop's Pattern Recognition and Machine Learning
In page 249 of Christopher M. Bishop's book "Pattern Recognition and Machine Learning", it is said
Again, the implementation of such algorithms can be checked by using
numerical ...
- 303
0
votes
0
answers
53
views
Derivation of (5.76) in "Pattern Recognition and Machine Learning"
The book "Pattern Recognition and Machine Learning" by Christopher M. Bishop says in page 248
... for softmax outputs we have:
$$\frac{\partial y_k}{\partial a_l}=\delta_{kl}y_k-y_ky_l.\tag{...
- 303
1
vote
0
answers
34
views
I need to derivate the maximum likelihood of GLR function [closed]
I am trying to do the partial derivative of this function but I don't know how to include the $\theta$ inside my function to be able to apply the partial derivative in $\theta$.
The likelihood ...
- 11
0
votes
0
answers
26
views
Second order moment of the Gaussian Distribution [duplicate]
In Bishops book, "Statistical Pattern Recognition", there is one exercise, which states to derive the second order moment of the Gaussian Distribution:
$E[x^2] = \int_{-\infty}^{\infty} N(x|...
- 505
2
votes
0
answers
26
views
Differentiation on the conditional variables of a probability
I have been questioning how to calculate the partial derivatives of a conditional probability function with respect to its parameters.
Assume $x$ is data and $\theta$ is a parameter(s). If I have a ...
1
vote
2
answers
104
views
Derivative of Trace of matrix product
I am trying to compute the gradient with respect to a vector $\mathrm x \in \mathbb{R}^d$ of a complicated expression involving the trace of matrix product.
The expression is the following:
$$ F(\...
2
votes
2
answers
57
views
Derivative of quadratic form of vector-valued function
This seems like a trivial question but I am currently stuck and cannot see what I am doing wrong.
So let us consider a function $f(x) : \mathbb{R}^d \rightarrow \mathbb{R}^d$.
I want to compute the ...
1
vote
1
answer
31
views
Best Error Function for Areas with Larger Slopes
I have some nonlinear data (let's say x's and y's) that I would like to perform regression on, and I would like to focus on having the error lower on regions where the graph is more sloped, rather ...
- 11
1
vote
1
answer
43
views
Deriving the expression for $p(\mathcal{K})$ where $\mathcal{K} = \{(\mathbf{s}^k,\mathbf{d}^k), k = 1,..., K\}$
This is a follow up from this question.
Consider a model of diseases and symptoms. $s_i\in\{0,1\}$ is a binary random variable indicating whether the patient is showing the $i$-th symptom and $d_j\in ...
- 71
0
votes
1
answer
46
views
Deriving the normal equations' coefficients
Suppose we use the least squares criterion to fit a linear model for the following dataset: $(x_1,y_1),...,(x_m,y_m)\in R \times R$, by solving the following optimisation problem:
$$(a^*,b^*) = \text{...
- 167
0
votes
1
answer
126
views
How to compute the derivative of the total loss wrt external trainable parameters?
I was just curious how external trainable parameters are updated. The challenge is to compute the derivative, the rest is handled by the optimiser.
I assumed a simple DNN as follows:
$$\hat{y} =\sigma(...
- 103
1
vote
0
answers
70
views
What is the derivative of a matrix with regard to a vector defined?
I had this question when I read equation (C.20) in Appendix C of "Pattern Recognition and Machine Learning" written by Christopher M. Bishop. Here I copy the equation below for reference:
...
- 303