8

I'll try to explain it in linear case. Consider the linear model $$Y_i=\sum_{j=1}^{p} \beta_jX_{i}^{(j)}+\epsilon_i, i=1,...,n. $$ When $p \leq n$ (number of independent variables less or equal then number of observation) and design matrix has full rank, the least squared estimator of $b$ is $$\hat{b}=(X^TX)^{-1}X^TY$$ and prediction error is $$ \dfrac{\| X(\...


8

Assuming independent $X_i$, the mean $\frac{1}{n}\sum X_i$ is also normal, i.e. $N(0,1/n)$. Absolute value of it is Half-normal, which has mean $E[Y_1]=\frac{\sigma\sqrt{2}}{\sqrt{\pi}}=\sqrt{\frac{2}{n\pi}}$. For $Y_2$ we can find the expected value directly: $$E[Y_2]=\frac{1}{n}\sum_{i=1}^n E[|X_i|]=E[|X_i|]=\sqrt\frac{2}{\pi}$$ This means $\sqrt n E[Y_1]=...


6

Let's try the usual preliminaries: simplify by choosing appropriate units of measurement and exploiting the symmetry assumption. Reframing the question Change the units of $X$ so that its mean is $m=1$: this will not alter the truth of the inequality. Thus the distribution $F$ of $X$ is symmetric about $1$ and the range of $X$ is within the interval $[0,2]...


6

For a single covariance you only need the bottom equation. The bounds are that the covariance cannot be greater than the product of the standard deviations (and cannot be less than the negative of the same value). However for a covariance matrix of more than 2 terms there is an additional limit, the matrix has to be positive semi-definite (or positive ...


6

Answer: Whatever the distribution of $X_1,...,X_n$, $$\mathbb{E} Y_2 \geq \mathbb{E} Y_1.$$ Details: For any $n$ numbers $X_1,..., X_n$ it is true that $$ \sum_i |X_i| \geq |\sum_i X_i|$$ and dividing both sides by $n$: $$ \frac{1}{n}\sum_i |X_i| \geq \frac{1}{n}|\sum_i X_i| = |\frac{1}{n}\sum_i X_i|.$$ Now, the key word is 'any', that means that ...


5

Remark: the sign seems to be flipped. Here is how we can prove if $X> 0$, then $$E(X \ln X) \ge E(X) E(\ln X)$$ We can apply Jensen's inequality twice. $x\ln x$ is convex, hence we have,$$E[X] \ln (E[X]) \le E(X\ln X)$$ $\ln x$ is concave, hence we have $$\ln (E(X)) \ge E(\ln X).$$ Combining the two inequalities, $$E(X \ln X) \ge E[X] \ln(E(X)) ...


5

You automatically get $||X + Y|| \le ||X|| + ||Y||$ by Minowski's inequality. The last part is simple arithmetic showing that: $\sqrt{A + B} \le \sqrt{A} + \sqrt{B}$ when $A, B > 0$. which is a simple direct proof. In fact, I think the whole thing is a direct consequence of Minowski's inequality.


5

One way to do it is to proceed from the answer you linked in comments, at the second last line (at this point no properties of the normal have been used). Note that $Y$ is just a standardized $X$; i.e. $X=\mu+\sigma Y$: \begin{align}\operatorname{Cov}(X,X^2)&=\sigma^3\operatorname{Cov}(Y,Y^2)+2\mu\sigma^2\operatorname{Cov}(Y,Y)\\ &=\sigma^3\gamma_1+...


4

\begin{align} EX &= \int x\cdot \mathbf{1}_{x< a} + x\cdot\mathbf{1}_{x\geq a} \ dP \\ &= \int x\cdot \mathbf{1}_{x< a}\ dP\\ &< \int a\cdot \mathbf{1}_{x< a}\ dP \\ &= a, \end{align} so yes.


4

Here is an answer (to a related question) which provides a valid generalization of $\mathbb{E}\left[\frac{1}{X}\right] \geq \frac{1}{\mathbb{E}[X]}$. What is the relation between $\mathbb{E}X^r$ and $\mathbb{[E}X]^r$, for all possible values of $r$, when $X$ is a positive random variable? This can be answered by applying Jensens's inequality, based on the ...


4

First, @Dilip Sarwate comments, for such ratio variables mean and variance often do not exist and then there is little to expect. For a detailed discussion of this see I've heard that ratios or inverses of random variables often are problematic, in not having expectations. Why is that?. But, let us assume a case where expectation and variance exists. ...


4

I will focus on the case where $|\rho| < 1$. We are given that $\mathbb{E}[X_1] = \mathbb{E}[X_2]=0$, $\operatorname{Var}[X_1]=\operatorname{Var}[X_2]=1, $ and the correlation is $\rho$. From there we can deduce that $\mathbb{E}[X_1^2]= \mathbb{E}[X_2^2]=1$ and $\mathbb{E}[X_1X_2]=\rho$. Let $g(x_1, x_2; t)=x_1^2+2tx_1x_2+x_2^2$ where $|t| < 1$. We ...


4

Your second form of the inequality is wrong. The initial form of Chebychev's inequality (for a random variable $X$ with zero mean) is: $$\mathbb{P}(|X| \geqslant k \sigma) \leqslant \frac{1}{k^2}.$$ Letting $k = k_* / \sigma$ and substituting into this initial form gives the alternative form: $$\mathbb{P}(|X| \geqslant k_*) \leqslant \frac{\sigma^2}{k_*^...


4

A bivariate copula is the distribution function $C$ of a random variable $(X,Y)$ on $[0,1]\times[0,1]$ where the marginal distributions of $X$ and $Y$ are both uniform (Sklar's Theorem). That is, there exists a random variable $(X,Y)$ for which (for all $0\le u,v\le 1$) $$C(u,v) = \Pr(X\le u,\, Y\le v) \tag{1}$$ and $$\Pr(X\le u) = u,\ \Pr(Y\le v)=v.$$ ...


3

If you have the joint distribution of $(X,Y)$ at hand, then by definition $\Pr(X>Y)=$ $\begin{cases}\displaystyle\sum_{\{(i,j):i>j\}}\Pr(X=i,Y=j)&,\text{ if } (X,Y)\text{ is jointly discrete}\\\displaystyle \iint_{\{(x,y):x>y\}}f_{X,Y}(x,y)\,dx\,dy&,\text{ if } (X,Y)\text{ is jointly continuous having pdf }f_{X,Y}\end{cases}$ Assumption ...


3

Okay so thanks to my friend pointing it out, it turns out there WAS a typo in the paper. It should have said $EF(Y) = EF(Y+EZ)\leq EF(Y+Z)$. So the RHS follows from applying conditional Jensen's (conditioning on Y). $E[F(Y+EZ)] = E[F(E[(Y+Z)|Y])] \leq E(E(F(Y+Z)|Y)) = E[F(Y+Z)].$


3

$\mbox{KL}(f||g)\geq 0$. But seriously, this is actually a really hard problem. Relevant to this topic is the area of Large Deviations Theory, specifically Rate Functions. You'll find a compendum of bounds here for example: https://en.wikipedia.org/wiki/Inequalities_in_information_theory#Lower_bounds_for_the_Kullback.E2.80.93Leibler_divergence with ...


3

The KL divergence isn't symmetric, so the answer might not be the same depending on how exactly you set up the "local triangle inequality" For one set-up, the inequality can never be right: $$ KL(p,q) + KL(p,r) \geq_? A KL(q,r)$$ because here is a counter-example: $p$ a gaussian centered at 0, $q_n$ a mixture of a gaussian centered at 0, and a gaussian ...


3

First note that the Markov inequality applies to non-negative random variables; you can't apply it directly to $Z$. Let $Y=|Z|^k$ for $k>0$, which is non-negative. Now apply the Markov inequality to $Y$, then relate that back to a probability statement about $|Z|$. (Edit: as whuber already noted in comments; not sure how I missed that) As for ...


2

In the multivariate case, you can use what is called the multivariate Cauchy-Schwarz inequality: $$ \newcommand{\Var}{Var} \newcommand{\Cov}{Cov} \Var(z) \ge \Cov(z,y) \Var(y)^{-1} \Cov(y,z) $$ Where here the inequality sign must be interpreted in the sense of the partial order on the cone of positive-definite matrices: $A \le B$ means that $B-A$ is ...


2

No, I'm afraid that's not possible. Consider the simpler problem of only two random variables. You want to evaluate $$ P(A\leq a|B\leq b) $$ but only have access to $P(A\leq a|B=b)$ which does not contain enough information to evaluate the expression you are after. However, \begin{align} P(A\leq a|B\leq b)&=\frac{P(B\leq b|A\leq a)P(A\leq a)}{P(B\leq b)}\...


2

Your friend is the Cauchy-Schwarz inequality. Let $H=X (X^T X)^{-1} X^T$ where $X$ is the design matrix in the linear model $Y_i = X_i^T \beta + \epsilon_i$, $X_i$ is the $i$th row of the design matrix. Since $(X^TX)^{-1}$ is positive definite, we can define an inner product by $$ h_{ij} =\langle X_i, X_j \rangle = X_i^T (X^TX)^{-1} X_j $$ and by the ...


2

This is false. Take $x = 0$ and $y = -\frac{1}{\alpha}$. Then $$\frac{2}{\alpha^2} \left( e^{\alpha y} - e^{\alpha x} \right) + e^{\alpha x} \left( x^2 - y^2 \right) = \frac{2}{\alpha^2} \left( e^{-1} - 1\right) + 1 \left(0 - \frac{1}{\alpha^2}\right)$$ Which simplifies to $$ \frac{1}{\alpha^2} \left( 2 e^{-1} - 3 \right) $$ Which is negative >&...


2

I also bumped into this passage recently! I'm not very familiar with probability/information theory, so I hope this makes sense and my notation is understandable; I tried for precision at the expense of brevity, but there's some notation in the book that I just don't know how to use precisely. As far as I can tell, the "data-processing inequality" for KL ...


2

The inequality depends on the fact that for any $y \in \mathbb{R}$, $$ \chi_{y^2 \geq 1} \leq y^2. $$ The proof is by cases. Suppose $y^2 < 1$. Then $$ \chi_{y^2 \geq 1} = 0 \leq y^2. $$ Now suppose $y^2 \geq 1$. Then $$ \chi_{y^2 \geq 1} = 1 \leq y^2. $$ To obtain the inequality in your question, simply substitute $y = \frac{X - \mu}{k\sigma}$. Since ...


2

Cantelli's inequality gives a better bound in many cases. Simply stated, for $k>0,$ $$P \left[X \geq \mu + k \sigma \right] \leq \frac{1}{k^2+1} $$ For a thorough treatment, see B.K. Ghosh's "Probability Inequalities Related to Markov's Theorem," $\it{The \ American \ Statistician},$ August 2002, Vol. 56, No. 3, pp. 186-190.


2

Let $p(t):=at^2+bt+c$ be a second degree polynomial. Then the roots of $p(t)$ are: $t_{1,2}:=\frac{-b\pm \sqrt{D}}{2a}$ where $D:=b^2-4ac$. This defines the discriminant $D$ of a polynomial. if $D\geq 0$, this implies that both roots (possibly repeated) are real. In particular if $D>0$, there are two distinct real roots, and you can easily prove that this ...


2

The quadratic in question must have complex roots, or at best two identical real roots; else there would be an interval $I$ such that for all $t\in I$, the random variable $tX-Y$ has negative mean-square value $E[(tX-Y)^2]$ which is impossible. If the discriminant is negative, then the roots of the quadratic are complex-valued (remember $\frac{-b \pm \sqrt{b^...


2

You can use the Cauchy-Schwarz inegality. $E\left(|YZ|^r\right) \leq \sqrt{E\left(|Y|^{2r}\right)E\left(|Z|^{2r}\right)}$. Then $$E\left(|XYZ|^r\right) \leq \sqrt{E\left(|X|^{2r}\right)\sqrt{E\left(|Y|^{2r}\right)E\left(|Z|^{2r}\right)}}$$ By the way $E\left(|X|^{2r}\right)\leq C$ if $r\leq \tfrac{4+\delta}{2} \geq \tfrac{4}{3}$ After all, $$E\left(|XYZ|...


2

$$E|X|^{r-1} = E\left[|X|^{r-1} \mathbb{1}_{(|X|\leq 1)} \right]+E\left[|X|^{r-1} \mathbb{1}_{(|X|> 1)} \right]$$ $$\leq E\left[1 \times \mathbb{1}_{(|X|\leq 1)} \right]+E\left[|X|^{r-1} \mathbb{1}_{(|X|> 1)} \right]$$ $$\leq E\left[1 \times \mathbb{1}_{(|X|\leq 1)} \right]+E\left[|X|^{r} \mathbb{1}_{(|X|> 1)} \right]$$ $$\leq 1 + E|X|^r$$ Where ...


Only top voted, non community-wiki answers of a minimum length are eligible