55
votes
Accepted
How is the cost function from Logistic Regression differentiated
Adapted from the notes in the course, which I don't see available (including this derivation) outside the notes contributed by students within the page of Andrew Ng's Coursera Machine Learning course.
...
- 26.9k
33
votes
Backpropagation with Softmax / Cross Entropy
While @GeoMatt22's answer is correct, I personally found it very useful to reduce the problem to a toy example and draw a picture:
I then defined the operations each node was computing, treating the $...
- 3,263
22
votes
Accepted
What justifies this calculation of the derivative of a matrix function?
There is a subtle but heavy abuse of the notation that renders many of the steps confusing. Let's address this issue by going back to the definitions of matrix multiplication, transposition, traces, ...
- 334k
21
votes
Accepted
Mean Absolute Error (MAE) derivative
The mae, as a function of $y_{\text{pred}}$, is not differentiable at $y_{\text{pred}}=y_{\text{true}}$. Elsewhere, the derivative is $\pm 1$ by a straightforward application of the chain rule:
$$\...
- 131k
19
votes
Accepted
Computing gradients via Gaussian Process Regression
Gaussian process regression (GPR) gives a posterior distribution over functions mapping input to output. We can differentiate to obtain a distribution over the gradient. Below, I'll derive an ...
- 33.2k
17
votes
Derivative of Softmax with respect to weights
The last hidden layer produces output values forming a vector $\vec x = \mathbf x$. The output neuronal layer is meant to classify among $K=1,\dots,k$ categories with a SoftMax activation function ...
- 26.9k
15
votes
Accepted
Second order approximation of the loss function (Deep learning book, 7.33)
They talk about the weights at optimum:
We can model the cost function $J$ with a quadratic approximation in the neighborhood of the empirically optimal value of the weights $w^∗$
At that point, ...
- 11.5k
14
votes
How is the cost function from Logistic Regression differentiated
To avoid impression of excessive complexity of the matter, let us just see the structure of solution.
With simplification and some abuse of notation, let $G(\theta)$ be a term in sum of $J(\theta)$, ...
- 359
14
votes
Interpretation of Radon-Nikodym derivative between probability measures?
First, we don't need probability measures, just $\sigma$-finiteness. So let $\mathcal M = (\Omega, \mathscr F)$ be a measurable space and let $\mu$ and $\nu$ be $\sigma$-finite measures on $\mathcal M$...
- 20.8k
13
votes
Neural network softmax activation
The internet has told me that when using Softmax combined with cross entropy, Step 1 simply becomes $\frac{\partial E} {\partial z_j} = o_j - t_j$ where $t$ is a one-hot encoded target output vector. ...
- 26.9k
12
votes
Accepted
Derivation of Group Lasso
It took me some time to understand this derivation. As usual, once you get the trick, it's actually straightforward.
To solve the group LASSO via block coordinate descent, we solve for each group of ...
- 2,137
11
votes
Accepted
How to calculate the derivative of crossentropy error function?
There is indeed a mistake in slide with title "Crossentropy Error Function":\begin{align}
\frac{\partial E_x}{\partial o_j^x} &=\frac{\partial }{\partial o_j^x} \left( - \sum_{k}[t_k^x \...
- 7,031
11
votes
Accepted
Where does the logistic function come from?
I read from Strogatz's book that it was originated from modeling the human populations by Verhulst in 1838. Assume the population size is $N(t)$, then the per capita growth rate is $\dot N(t)/N(t)$. ...
- 1,928
10
votes
Accepted
How can I fit a spline to data that contains values and 1st/2nd derivatives?
We will describe how a spline can be used through Kalman Filtering
(KF) techniques in relation with a State-Space Model (SSM). The fact
that some spline models can be represented by SSM and computed ...
- 5,706
9
votes
Derivative of Softmax with respect to weights
I got a different result. Also $\sigma(j)$ depends on $\mathbf{w}_i$ inside the denominator of the softmax, so not sure Antoni's result is correct.
$$\begin{align}\frac{\partial}{\partial \mathbf{w}...
- 91
9
votes
Accepted
Differentiation of Cross Entropy
Your $\frac{\partial E}{\partial o_j}$ is correct, but $\frac{\partial E}{\partial z_j}$ should be
$$\frac{\partial E}{\partial z_j}=\sum_i\frac{\partial E}{\partial o_i}\frac{\partial o_i}{\partial ...
- 16.8k
9
votes
Proper regression for determining correlations between derivatives of functions
If you have long-enough time-series you can try to tackle it in Fourier domain. If you do that it may be a good idea to apply a window function, which would force your signal $x=x\left(t\right)$ to ...
- 773
9
votes
Proper regression for determining correlations between derivatives of functions
If your measurements are at equidistant intervals, you can try converting this into ARIMAX(1,1,0) model, which can be estimated with OLS. Despite similarity to OLS that you described in the question, ...
- 62.3k
8
votes
How can I fit a spline to data that contains values and 1st/2nd derivatives?
You can do spectacularly well with a standard least-squares routine, provided you have a reasonable idea of the relative sizes of the random errors made for each derivative. There is no restriction ...
- 334k
8
votes
Accepted
Gradient of the log likelihood for energy based models
The issue emerges in the evaluation of the second term in line $(3)$ and $(4)$ of your derivation. Note that
$$\nabla_{\theta} Z(\theta)^{-1} = \nabla_{\theta} \frac{1}{\int_x \exp(-E_{\theta}(x))\, ...
- 2,630
8
votes
Deriving the PDF of the kth order statistic from the CDF
The $r$-th order statistic $X_{(r)}$ has cdf
$$F_{X_{(r)}}(x) = \sum_{j=r}^{n} \binom nj [ F_{X}(x) ]^{j} [ 1 - F_{X}(x) ]^{n-j}$$ because
\begin{align}\mathbb P(X_{(r)}\le x) &= \mathbb P(\exists~...
- 108k
8
votes
Accepted
matrix-calculus - Understanding numerator/denominator layouts
If you think of $L$ as a column vector, then I think both your sources agree that $\frac{dJ}{dL}$ should be a row vector.
But what if you really want $L$ as a row vector. Surely, the math shouldn't &...
- 26.5k
7
votes
Where does the logistic function come from?
I don't know about its history, but logistic function has a property which makes it attractive for machine learning and logistic regression:
If you have two normally distributed classes with equal ...
- 9,678
7
votes
Accepted
Derivation of Hessian for multinomial logistic regression in Böhning (1992)
The source of the issue in my view comes from a confusion concerning dimensionality, and because the Hessian departs from the usual context in that there is sub-partitioning going on. Other than a ...
- 2,630
7
votes
Accepted
Is the Inverse Mills Ratio Strictly Decreasing?
Theorem $1.$ (Sampford) If $\lambda(x):=\frac{\varphi(x)}{\Phi(x)},$ then $\lambda^\prime(x)\in(-1,0).$
$\frac{e^{-\frac{1}{2} z^2}}{\int^{x}_{-\infty}e^{-\frac{1}{2} z^2}\, \mathrm dz}$ is nothing ...
- 10.3k
7
votes
Accepted
2nd derivative of spline
I believe one can achieve this with the splines2 package in R, which essentially takes the original splines package and adds ...
- 18.4k
6
votes
Accepted
What is the second derivative of a B-spline?
This document gives (with a corrected typo)
\begin{equation}
\frac{\text{d}^{(n)}B_{i,j}(x)}{\text{d}x^{(n)}}
=
(j-1)
\left(
\frac{- \text{d}^{(n-1)} B_{i+1,j-1}(x) / \text{d}...
- 3,132
6
votes
Derivative of bivariate normal CDF with common mean parameters
We can obtain a nice closed-form answer simply by applying definitions and the most basic result of linear regression theory: no calculation is needed.
First consider more generally what happens to $\...
- 334k
6
votes
matrix-calculus - Understanding numerator/denominator layouts
Unfortunately, I didn't come across a resource that doesn't leave gaps. It's a disputed area. Even the chain rule may sometimes not make a lot sense, e.g. some terms might be 3D tensors that the ...
- 58.2k
6
votes
Proper regression for determining correlations between derivatives of functions
The differential equation can be rewritten
$$x_{t+1}=(\alpha+1)x_t+\beta\int^t u(x)\,\mathrm{d}x+\gamma t+C.$$
This eliminates many of the problematic aspects of the direct formulation and computing ...
- 334k
Only top scored, non community-wiki answers of a minimum length are eligible