20
votes
Accepted
Under the 0-1 loss function, the Bayesian estimator is the mode of the posterior distribution
You need to be a bit careful with this kind of problem because the definition of the zero-one loss function will depend on whether you are dealing with a discrete or continuous parameter. For a ...
14
votes
Accepted
Finding the slope at different points in a sigmoid curve
Your question is very broad. There are many ways to do this, even without assuming a specific function. For the following I assume that you have a good reason to use the Gompertz model.
First let's ...
13
votes
Accepted
Limit of Integration of continuous function
Clearly, the integral can be rewritten as $E[f(Y_n)]$, where $Y_n = \frac{1}{n}(X_1 + \cdots + X_n)$, and $X_1, \ldots, X_n \text{ i.i.d.} \sim U(0, 1)$. By (weak) law of large numbers, we have $Y_n \...
10
votes
gradient versus partial derivatives
Gradient is the partial derivatives :
$$\nabla f = \left(\frac{\partial f}{\partial x_1};\frac{\partial f}{\partial x_2};...;\frac{\partial f}{\partial x_n}\right)$$
Eg : $f=x^2y$
$$\nabla f =(2xy;...
10
votes
How do you obtain the standard error for a slope at a given data point, for curvilinear regression?
The model is
$$\mathbb{E}(y) = \beta_0 + \beta_1 x + \beta_2 x^2.$$
Adding a fixed (usually small) quantity $\delta x$ to $x$ and comparing gives the difference
$$\eqalign{
\frac{\delta\,\mathbb{E}(...
8
votes
Accepted
Why does Judea Pearl call his causal graphs Markovian?
He is referring to the Parental Markov Condition (see theorems 1.2.7 and 1.4.1 of Causality). Given a graph $G$, we say a distribution $P$ is Markov relative to $G$ if every variable is independent of ...
6
votes
Accepted
Calculating t-SNE gradient (a mistake in the original t-SNE paper)
I just signed up for this forum due to your question :)
Nice question! It shows someone is indeed trying to follow & derive the nitty gritty. Your question is totally valid, (28) is indeed ...
6
votes
Accepted
What is the second derivative of a B-spline?
This document gives (with a corrected typo)
\begin{equation}
\frac{\text{d}^{(n)}B_{i,j}(x)}{\text{d}x^{(n)}}
=
(j-1)
\left(
\frac{- \text{d}^{(n-1)} B_{i+1,j-1}(x) / \text{d}...
5
votes
Why is optimisation solved with gradient descent rather than with an analytical solution?
The system of equations you get setting the derivatives equal to zero cannot generally be solved analytically. For instance suppose I want choose $10>x>0$ to minimize $xln(x)-\sqrt{x}$ (which ...
5
votes
gradient versus partial derivatives
If a function $f$ takes the parameters $x_1, \ldots, x_n$, then the partial derivatives w.r.t. the $x_i$ determine the gradient:
\begin{equation}
\nabla f = \frac{\partial f}{\partial x_1 }\mathbf{e}...
5
votes
Accepted
Correlation between normal and log-normal variables
As is often the case, precisely formulating the question helped me work out the answer.
My approach makes use of the marginal expectation of the bivariate normal:
$$E_X(y) = E(X|Y=y) = \mu_x + \rho\...
5
votes
Applying Leibniz's integral rule to the Gaussian distribution's normalization condition
Leibniz' rule can be extended to infinite regions of integration with an extra condition on the function being integrated. Let's write out the basic form:
$${d \over dy}\int_a^b f(x,y)dx = \int_a^b {...
4
votes
Accepted
Differentiating $ (y-X\beta)^T(y - X \beta) $ with respect to $\beta$
Let us assume that you are working in a setup where $y$ is $N \times 1$ and $X$ is $N \times K$ and $\beta$ is $K \times 1$. I prefer to define $e(\beta) := (y - X\beta)$ and similarly the $i$'th ...
4
votes
Accepted
Is the sample mean of the gradient the same as the gradient of the sample mean?
The derivative and expectation have the associative property (you can exchange the order) by Leibniz integral rule (computing the expectation is just some sort of integration)
$$\frac{\partial}{\...
4
votes
Accepted
Why is the formula for the density of a transformed random variable expressed in terms of the derivative of the inverse?
It seems that the heuristic described by @whuber in their answer to the linked problem can be modified slightly to yield the change of variables formula for the density in its more familiar form. ...
4
votes
Calculus for Statistics
Learn how to differentiate and integrate, and what each of these concepts mean when applied to multi variate functions. I mean Riemann integrals. This alone will get you a long way in statistics. It’s ...
Community wiki
4
votes
Accepted
Applying Leibniz's integral rule to the Gaussian distribution's normalization condition
Nice question!
option 1
Derivatives are limits (of fractions), and moving limits inside an expectation often triggers an invocation of a theorem such as the monotone convergence theorem, Lebesgue's ...
3
votes
How does one compute a variational derivative?
I will carry out the steps in a way I hope is clear enough to indicate what assumptions must be made about $y,$ $p,$ and $\delta$ to justify the steps.
When a function $y$ is a local minimum of a ...
3
votes
Accepted
Derivative of expectation where the variable appears in the integration limit and in the integrand?
Consider any differentiable function $f:\mathbb{R}^2\to \mathbb{R}.$ It is a theorem that the derivative of $f$, written $Df,$ is given by its partial derivatives,
$$Df(x,y) = \left(\frac{\partial f}...
3
votes
Accepted
Expected Loss Calculation
Here are two methods, a nested numerical integration and a Monte Carlo integration, in R. The nested numerical integration has an outer integration that goes from 0 to 1 with respect to $\theta_B$ ...
3
votes
Simple Log-likelihood question
By rewriting
$$\eqalign{
l(\lambda_1, \lambda_2) &= y_1\log(\lambda_1F_1) - \lambda_1 F_1 - \log(y_1!)+y_2\log(\lambda_2F_2)-\lambda_2F_2-\log(y_2!) \\
&= y_1\left(\log(\lambda_1)+\log(F_1)\...
3
votes
Calculating t-SNE gradient (a mistake in the original t-SNE paper)
Let our cost function be $C=KL(P||Q)=\sum_k\sum_{l\neq k}p_{kl}\log(p_{kl})-p_{kl}\log(q_{kl})$, so that we are trying to find the gradient $\frac{\partial C}{\partial y_i}$.
We will define some ...
3
votes
Neural Networks: How to get the gradient vector for the xOr problem?
You have several mistakes.
You're missing the minus sign due to $-\hat y_i$ term
If you have $n$ samples, $x_1$ and $x_2$ should also have indices
Derivative wrt $b_3$ can't be $0$ because it's ...
3
votes
Accepted
derivation of coordinate ascent variational inference
I'll start with a quick derivation of the Euler-Lagrange formula, then I'll show how you can use it, then I'll show how it applies to your problem.
Background
Equation 22 is a functional -- a function ...
2
votes
Accepted
Help with derivation of Mean Field Variational Inference
The main issue is that the integrals involved are multivariate. A confusing thing about Bishop's notation is that, inside those integrals, $q_i$ should actually be $q_i(\mathbf{Z}_i)$.
So we want to ...
2
votes
Accepted
Expectation / Summation inequality with multiple indices
Assuming the $I_i$'s are identically distributed,\begin{align*}\sum_{i\ge1}\mathbb P\left(I_i \ge i \right) &= \sum_{i\ge1}\sum_{j \ge i}\mathbb P\left(I_i = j \right)\\&= \sum_{\underbrace{j \...
2
votes
Why does Judea Pearl call his causal graphs Markovian?
These graphs do satisfy the Markov property - once you condition on the parent node, from which the causal arrow comes, the variable is independent of earlier ancestors that causally affect that ...
2
votes
Accepted
Symmetry between integrals including absolute value
In the left hand you have the integral of a function of $|v|$ (considering the quadratic term too), hence that function is symmetric on $v$'s domain. This comes simply from the fact that $f(|-v|) = f(|...
2
votes
Accepted
Simple Appplication of Law of Iterated Expectation
I believe, you can; we can think over $E[Y_i|T_i=1,D_i=1]$ first. By the law of iterated expectations, we have $$E[Y_i|T_i=1,D_i=1] = E[ E[Y_i|T_i=1,D_i=1,M_i] ]
\\ =\int{E[Y_i|T_i=1,D_i=1,M_i=m]\ \ ...
2
votes
Accepted
Derivation of M step for Gaussian mixture model
TL;DR, we have that
$$\mu^*_k = \frac{\sum_{i=1}^n W_{ik}x_i}{\sum_{i=1}^n W_{ik}}$$
$$\Sigma^*_k = \frac{\sum_{i=1}^{n} W_{ik}(x_i -\mu^*_k)(x_i - \mu^*_k)'}{\sum_{i=1}^n W_{ik}}$$
In particular, ...
Only top scored, non community-wiki answers of a minimum length are eligible
Related Tags
calculus × 110machine-learning × 18
mathematical-statistics × 16
probability × 13
self-study × 11
neural-networks × 11
expected-value × 11
backpropagation × 10
integral × 10
regression × 8
derivative × 8
distributions × 6
optimization × 6
gradient-descent × 5
linear-algebra × 5
r × 4
time-series × 4
matrix × 4
gradient × 4
normal-distribution × 3
least-squares × 3
references × 3
quantiles × 3
lognormal-distribution × 3
variance × 2