51
votes
Accepted
Derivation of change of variables of a probability density function?
Suppose $X$ is a continuous random variable with pdf $f$.
Let $Y=g(X)$, where $g$ is a monotonic function.
The function $g$ could be either monotonically increasing or monotonically decreasing. If $g$ ...
- 2,127
27
votes
Accepted
Generating random points uniformly on a disk
The problem is due to the fact that the radius is not uniformly distributed. Namely, if $(X,Y)$ is uniformly distributed over
$$\left\{ (x,y);\ x^2+y^2\le 1\right\}$$
then the (polar coordinates) ...
- 108k
17
votes
Accepted
Suppose $Y_1, \dots, Y_n \overset{\text{iid}}{\sim} \text{Exp}(1)$. Show $\sum_{i=1}^{n}(Y_i - Y_{(1)}) \sim \text{Gamma}(n-1, 1)$
The proof is given in the Mother of All Random Generation Books, Devroye's Non-uniform Random Variate Generation, on p.211 (and it is a very elegant one!):
Theorem 2.3 (Sukhatme, 1937) If we define ...
- 108k
17
votes
Generating random points uniformly on a disk
The simplest and least error-prone approach would be rejection sampling: generate uniformly distributed points in the square around your circle, and only keep those that are in the circle.
...
- 131k
16
votes
Accepted
How can I obtain a Cauchy distribution from two standard normal distributions?
This can be done with a minimum of computation, relying only on (a) simple algebra and (b) basic knowledge of distributions associated with statistical tests. As such, the demonstration may have ...
- 334k
14
votes
How can I obtain a Cauchy distribution from two standard normal distributions?
Correction: the Jacobian of the transform is $|V|$, not $V$, which implies that
$$f_{U,V}(u,v)=f_{X,Y}(uv,v-uv)|J|=\frac{|v|}{2\pi}\exp\left\{\frac{-v^2}{2}(2u^2-2u+1)\right\}$$
Hence
\begin{align}f_U(...
- 108k
12
votes
What is the jacobian for a neural network
A Jacobian is quite a general term indeed. Lets take this simple, single-hidden-layer network
$$\hat{\boldsymbol{y}} = g(\mathbf{W}^{(1)} \cdot f(\mathbf{W}^{(0)} \cdot \boldsymbol{x} + \boldsymbol{b}...
9
votes
Accepted
Sample random variables conditional on their sum
If you seek the conditional density of $(X_1,...,X_{n-1})$ given $$S=\sum_{k=1}^n X_k$$ a change of variable from $$(X_1,...,X_{n})\sim\prod_{i=1}^n f(x_i)$$ to $$\left(X_1,...,X_{n-1},S\right)\sim\...
- 108k
9
votes
Accepted
Reason for absolute value of Jacobian determinant in change-of-variable formula?
For a specific example, in addition to @whuber's advice, let $y=f(x)=-2x$, and $x=g(y)=-y/2$; and $x \in [0,1]$, i.e. the support. Then, $y$ would be in the range $[-2,0]$. Also, we have $g'(y)=-1/2, ...
- 58.2k
8
votes
Generating random points uniformly on a disk
You can find the mathematics of this situation in a related question here. The method is set out in Xi'an's excellent answer, and it can be summarised by the following requirements:
$$\begin{matrix}
...
- 133k
8
votes
Accepted
Inverse Gaussian chi square connection
The proof is not exactly standard, although it relates to the "law of the unconscious statistician" [an expression I cannot fathom and do not find amusing] :
First, define $Y=\min\{X,\mu^2/X\...
- 108k
6
votes
Higher Order of Vectorization in Backpropagation in Neural Network
You're right that that doesn't make sense as the Jacobian. Furthermore if multiplying jacobians was really how autodiff worked, any pointwise function applied on vector of length $n$ would result in a ...
- 26.5k
6
votes
Accepted
Jacobian for transformation of discrete random variables (intuition)
Here's an attempt at an intuitive explanation for the transformation $f(x) = 2x$.
Discrete case: a discrete random variable is like a collection of point masses. Imagine a collection of rocks of ...
- 4,159
6
votes
Accepted
Finding the pdf of Y from that of X, linear transformation
There are several standard approaches for deriving the density of a transform $g(X)$ of a random variable, including:
the "push-forward" technique, when looking at
$$\int_A f_Y(y)\text dy=\...
- 108k
5
votes
Accepted
Weighted Least Squares weights not changing Jacobian matrix
This is simply because $A$ is a square and non-singular matrix. Therefore,
\begin{align}
H = (A^TWA)^{-1}A^TW = A^{-1}(A^TW)^{-1}A^TW = A^{-1}.
\end{align}
In practice, $A$ is typically slim ...
- 21.2k
5
votes
Suppose $Y_1, \dots, Y_n \overset{\text{iid}}{\sim} \text{Exp}(1)$. Show $\sum_{i=1}^{n}(Y_i - Y_{(1)}) \sim \text{Gamma}(n-1, 1)$
I lay out here what has been suggested in comments by @jbowman.
Let a constant $a\geq 0$. Let $Y_i$ follow an $\text{Exp(1)}$ and consider $Z_i = Y_i-a$. Then
$$\Pr(Z_i\leq z_i \mid Y_i \geq a) =...
- 60.8k
5
votes
Accepted
Vector-Jacobian Product Computational Cost
This is a well known result from automatic differentiation literature. Specifically, the result is that reverse mode differentiation can
calculate the gradient $\frac{\partial \hat{f}} {\partial \...
- 380
4
votes
How to calculate the Jacobian of the transformation ( for covariance matrix)
The algebra of differential forms makes short work of this.
I take it that $\Sigma = (\sigma_{ij})$ is coordinatized by means of one of its triangles, say the upper triangle with components $\sigma_{...
- 334k
4
votes
Computing directly the pdf of $Y=X^2$ for the pdf $f_X(x) = \frac{2}{9}(x+1)$
The third condition is important in that
\begin{align*}
\mathbb{P}(Y\in B) &= \int_B f_Y(y)\text{d}y\\
&= \mathbb{P}(g(X)\in B)\\
&= \sum_{i=1}^k \mathbb{P}(g(X)\in B,\,X\in A_i)\\
&= \...
- 108k
4
votes
Accepted
IS $\int_{-\infty}^\infty e^{-\beta\cdot g(x)}g(x)^{\alpha-1}\text{d}x={\Gamma(\alpha)\over \beta^\alpha}\ \ ?$
As stated, the relation is not true: consider $g(x) = 1$, in which case the integral is infinite.
Using the general framework of integration by substitution, the following conditions are sufficient:
...
- 25.2k
4
votes
What is the jacobian for a neural network
Classical approach for neural network is to take a batch of samples and calculate average gradient over these samples. For the Jacobian instead of calculating average gradient - you calculate gradient ...
- 7,919
4
votes
Accepted
Jacobian of transformation
For the parameter transform$$\eta\longmapsto\sigma^2=\exp\{2\eta\}$$the Jacobian is
$$\frac{\text d\sigma^2}{\text d\eta}=2\exp\{2\eta\}=2\sigma^2$$and the posterior changes from $p(\mu,\eta)$ into$$p^...
- 108k
4
votes
Accepted
Derivation of ELBO in ADVI Paper, Jacobian of Elliptical Transformation
The transformation of the formula from (5) in the paper to the formula at the bottom of page 9 is a consequence of the Law of the Unconscious Statistician (LOTUS) -see (28) in Monte Carlo Gradient ...
- 168
4
votes
Accepted
Transformation of a Random Variable
As this is a strictly monotonic transformation ($Y$ increases as $X$ increases), you can obtain the answer immediately using the change-of-variable formula.
To find the PDF of $Y$ from first ...
- 3,416
3
votes
What does this terminology mean in this introduction to likelihood ratios?
This is from the change of variable-formula for (multiple) integrals. That strange extra factor $\left| \frac{\partial x}{\partial y} \right|$ is the Jacobian.
- 82.8k
3
votes
Accepted
How do the derivatives of the loss function with respect to a layer's inputs form a Jacobian?
When there is only one function to evaluate, you'll have one row in the Jacobian matrix, i.e. a vector. For completeness, the following quote is from wikipedia:
Suppose $f : ℝ^n → ℝ^m$ is a function ...
- 58.2k
3
votes
Derivation of ELBO in ADVI Paper, Jacobian of Elliptical Transformation
I don't think this has anything to do with LOTUS, instead it's just a quality of the Normal distribution:
Suppose $x\sim N(\mu, \Sigma)$. And suppose $g(x)$ is some function of $x$ we want to take the ...
- 3,680
3
votes
Accepted
Is the Jacobian term needed if the prior is on the transformation parameter?
The Jacobian is used when you convert back and forth between a distribution for $\sigma$ and a distribution for $\beta$. This is true of the conversion whether you are talking about the prior or ...
- 133k
2
votes
Computing directly the pdf of $Y=X^2$ for the pdf $f_X(x) = \frac{2}{9}(x+1)$
To provide an alternative series of steps to arrive at the density of $Y$, using the "distribution function" method as @Xi'an answer does, we have
$$P(Y \leq y) = P(X^2 \leq y)$$
For the support of ...
- 60.8k
2
votes
Accepted
For a posterior $p(\theta |y)$, if I specify a one-to-one transformation $\phi = g(\theta)$, how can I apply the transformation?
Just take the conditional distribution as the distribution of interest:
$$ q(\theta) = p(\theta \vert y),$$
since $y$ is "given". This is justified by the definition of the conditional ...
- 36
Only top scored, non community-wiki answers of a minimum length are eligible