Taylor
  • Member for 10 years
  • Last seen this week
Which loss function is correct for logistic regression?
Accepted answer
41 votes

The relationship is as follows: $l(\beta) = \sum_i L(z_i)$. Define a logistic function as $f(z) = \frac{e^{z}}{1 + e^{z}} = \frac{1}{1+e^{-z}}$. They possess the property that $f(-z) = 1-f(z)$. Or ...

View answer
Softmax overflow
Accepted answer
24 votes

Observe that $$ \frac{e^{x_i}}{\sum_j e^{x_j}} = \frac{e^{-m}}{e^{-m}}\frac{e^{x_i}}{\sum_j e^{x_j}}= \frac{e^{x_i-m}}{\sum_j e^{x_j-m}} $$ for any constant $m$. Obviously it is not true that $e^{...

View answer
From a statistical perspective: Fourier transform vs regression with Fourier basis
21 votes

They're the same. Here's how... Doing a Regression Say you fit the model $$ y_t = \sum_{j=1}^n A_j \cos(2 \pi t [j/N] + \phi_j) $$ where $t=1,\ldots,N$ and $n = \text{floor}(N/2)$. This isn't ...

View answer
Why the sum of two absolutely-continuous random variables isn't necessarily absolutely continuous?
Accepted answer
18 votes

Let $X$ be a standard normal random variable, and let $Y = -X$ (pointwise). Then both are a.c., but $X+Y$ is $0$ everywhere.

View answer
Why is P(A,B|C)/P(B|C) = P(A|B,C)?
18 votes

\begin{align*} \frac{P(A,B|C)}{P(B|C)} &= \frac{P(A,B,C)}{P(C)}\frac{P(C)}{P(B,C)} \\ &= \frac{P(A,B,C)}{P(B,C)} \\ &= P(A|B,C) \end{align*}

View answer
Cdf of minimum of two iid random variables
Accepted answer
14 votes

It's a common trick. If $X = \min(Y_1,Y_2)$ and $F$, $F_X$ are the CDFs of the $Y_i$s and $X$, respectively, then \begin{align*} F_X(x) &= 1 - P(X > x) \\ &= 1- P(Y_1 > x, Y_2 > x) \\...

View answer
Estimation of AR($p$) model by `lm` versus `arima` in R: different results
Accepted answer
14 votes

There are a few reasons. For one, your ARMA model doesn't include a mean/intercept. For another, the ARMA by default uses sum of squares only to find starting points for an iterative maximum ...

View answer
sampling/importance resampling - why resample?
11 votes

SIR uses two ideas. The first idea is importance sampling. The main idea is that you draw from one probability distribution (in your case, it's the uniform), in order to get information about another. ...

View answer
Is the MAP the maximum value of the posterior or its mode?
Accepted answer
11 votes

In Bayesian statistics, a maximum a posteriori probability (MAP) estimate is an estimate of an unknown quantity, that equals the mode of the posterior distribution. This is correct. As far as ...

View answer
Why do some formulas have the coefficient in the front in logistic regression likelihood, and some don't?
Accepted answer
11 votes

The second is a special case of the first. Your first reference discusses the case where each $y_i$ is distributed as a Binomial distribution with sample size $n_i$, while the second reference assumes ...

View answer
Why are the cut-offs used for Bayes factors and p-values so different?
Accepted answer
10 votes

A few things: The BF gives you evidence in favor of a hypothesis, while a frequentist hypothesis test gives you evidence against a (null) hypothesis. So it's kind of "apples to oranges." These two ...

View answer
Estimate the Kullback–Leibler (KL) divergence with Monte Carlo
10 votes

I assume you can evaluate $f$ and $g$ up to a normalizing constant. Denote $f(x) = f_u(x)/c_f$ and $g(x) = g_u(x)/c_g$. A consistent estimator that may be used is $$ \widehat{D_{KL}}(f || g) = \left[...

View answer
State space representation of ARMA(p,q) from Hamilton
10 votes

This is the same as above, but I thought I would provide a shorter, more concise answer. Again, this is Hamilton's representation for a causal ARMA($p$,$q$) process, where $r=\max(p,q+1)$. This $r$ ...

View answer
Why are contours of a multivariate Gaussian distribution elliptical?
Accepted answer
9 votes

You can understand the shape of the ellipsoid better if you look at the spectral/eigen decomposition of the precision matrix (inverse of the covariance matrix). You want to look at the eigenvalues of ...

View answer
What does it mean by independently and identically distributed random variables?
9 votes

"IID" stands for independent and identically distributed. An IID sample is also known as a "random sample." It is referring to the distribution of all of your random variables: $X_1, \ldots, X_n$. In ...

View answer
Simulate AR(1) process in R with specified nonzero mean and AR coefficient
8 votes

Notice that this is the same model as $$ X_t - 10 = .5(X_{t-1} - 10) + Z_t . $$ $10$ is the mean, while $5$ was the intercept. This means we can add ten to the mean zero series. In other words, if ...

View answer
How to calculate the probability of a data point belonging to a multivariate normal distribution?
Accepted answer
8 votes

Yeah, that sounds right. If you have parameters $\mu$ and $\Sigma$ and data point $x$, then the set of all data points that are less likely than $x$ are the ones that have less density, or in other ...

View answer
Why do independent priors for two random variables not result in an independent joint posterior distribution?
Accepted answer
8 votes

When you have an independent prior on $X$ and $Y$, then the posterior might not factor into $X$ and $Y$ pieces just because the likelihood doesn't factor into $X$ and $Y$ pieces. It's easy to see ...

View answer
Yule Walker equations of an ARMA(1,1)-process
Accepted answer
8 votes

Yule Walker (for parameter estimation) is usually only used for AR models, but this method you're using is still a valid technique for finding the autocovariance function. I'm assuming that's what you'...

View answer
What is the difference between Markov chains and Markov processes?
Accepted answer
8 votes

From the preface to the first edition of "Markov Chains and Stochastic Stability" by Meyn and Tweedie: We deal here with Markov Chains. Despite the initial attempts by Doob and Chung [99,71] to ...

View answer
Show that $E\{ZΦ(Z)\} = 1 / \left( 2\sqrt{\pi} \right)$
Accepted answer
8 votes

This is S. Catterall's hint: \begin{align*} E\{ZΦ(Z)\} &= \int_{\mathbb{R}} z \Phi(z) \phi(z) dz. \end{align*} And an extra hint: let $u=\Phi(z)$ and $v' = z\phi(z)$. One more hint: $v = \int_{-\...

View answer
Does uncorrelation imply independence for marginally Gaussian random variables?
Accepted answer
7 votes

If $X = (X_1,\ldots, X_n)$ are jointly normal, too, then yes. Otherwise, no. In this case $\Sigma = \text{diag}(\sigma_1^2,\ldots, \sigma_2^2)$ and $\mu = (\mu_1,\ldots,\mu_n)'$ \begin{align*}f_X(x) ...

View answer
How to interpret Bayesian (posterior predictive) p-value of 0.5?
Accepted answer
7 votes

The model is true if the data are generated according to the model you are doing inference with. In other words, the unobserved parameter is generated by the prior, and then, using that parameter draw,...

View answer
How to derive the form of the posterior for regression?
Accepted answer
7 votes

\begin{align*} P(\theta \mid y,x) &= \frac{P(\theta, y,x)}{P(y,x)} \tag{defn. condtl. prob} \\ &= \frac{P(y \mid \theta, x)P(\theta \mid x)P(x)}{P(y \mid x)P(x)} \tag{defn. condtl. prob}\\ &...

View answer
Acceptance-Rejection Method Acceptance Probability Proof
Accepted answer
7 votes

If you target $f$ with $g$, and you know $f(x) \le g(x)c$, then \begin{align*} P(\text{accept proposal}) &= P\left( U \le \frac{f(X)}{g(X)c} \right) \\ &= E\left( \mathbf{1}\left[U \le \frac{f(...

View answer
how can i simulate with arima.sim drift, intercept and trend
Accepted answer
7 votes

Deterministic Trend If your drift intercept is $c$, you can just add the function $c t$ to the zero mean process. Code: xt <- arima.sim(n=50, list(order=c(1,0,1), ar = c(.9), ma = -.2)) ...

View answer
sequential/recursive/online calculation of sample covariance matrix
Accepted answer
7 votes

It's easy if you write $$ \hat\Sigma_n= \frac{1}{n-1}\sum_{i=1}^nX_i X_i^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T. $$ Split up the sum over $n$ elements into two parts. One will involve the first $n-...

View answer
Importance Sampling
Accepted answer
7 votes

Pretty close. Remember that exponentials have support on the positive reals. $$ \frac{1}{N}\sum_i f(x_i)/g(x_i) \to E_g[f(x)/g(x)] = \int_0^{\infty}f(x)/g(x) g(x)dx = \int_0^{\infty}f(x) dx \neq \...

View answer
Where is the dominated convergence theorem being used?
Accepted answer
7 votes

Assuming absolute summability of the autocovariance function (i.e. $\sum_{h=-\infty}^{\infty}|\gamma(h)| < \infty$) \begin{align*} \lim_n n \text{Var}(\bar{X}_n) &= \lim_n n^{-1} \sum_i \sum_j \...

View answer
KL divergence between gaussian and uniform distribution
6 votes

The KL divergence $$ KL\left(P \middle\| Q\right) = \int \log \frac{d P}{d Q }dP $$ is only defined if the Radon-Nikodym derivative exists, which is when $P$ is dominated by $Q$ (written $P \ll Q$). ...

View answer
1
2 3 4 5
17