New answers tagged

3

Let us consider as an example a coin with $p=0.1$ provability of heads. If we learn that the coin turned heads, our "surprise" is given by $\log_2{\frac{1}{0.1}}\approx3.3$, which is indeed greater than the 1 bit yielded by the fair coin. On the other hand, if we learn the coin turned tails, our surprise is only $\log_2{\frac{1}{0.9}}\approx0.15$, which is ...


1

You start with a population of unmarked individuals and take a sample of which some are marked. It is assumed that the marked individuals randomly distribute throughout the population. This is key. If they are truly randomly distributed throughout the population, then when you take the second sample, the proportion marked in the sample should be the same as ...


1

"A statistic t=T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic t = T(X), does not depend on the parameter θ." If the sampling distribution for some data $X$ does not depend on $\theta$ then how can that data say anything about $\theta$? It would be like estimating some ...


4

I think a common way to motivate the mathematical definition is the following. Say you have the sufficient statistic $T(X)$, and I just have the data/random sample $X$. By the mathematical definition of sufficiency, $$ p(X|T(X), \theta) = P(X|T(X)).$$ The r.h.s. is a probability distribution you, again per definition and in theory, know, and can use to do ...


2

Let me try and help with the question. I think the best way to understand a Multivariate distribution is less as random variables, but instead as random vectors in $\mathbb{R}^n$. I may be wrong but I get the sense that you are thinking that $X_1, X_2, ..$ are drawn separately. Instead think of the vector as produced from the sample space ($\Omega \to \...


0

The Wold decomposition itself is a trivial fact. It is just the Gram-Schmidt orthogonalization procedure. In the time series context, the Hilbert space in question is the space of random variables with finite second moments. Just to state the Wold decomposition: For any covariance stationary time series $\{X_t\}$, there exists innovations $\{\epsilon_t\}$ ...


2

My reasoning is somewhat similar to Cagdas', but I'd like to look at how the things develop as $\lambda$ goes into extremes. If I did my algebra right, the derivative of the coefficients is given by: $$\frac{\partial \hat{\beta}}{\partial\lambda} = -(X'X + \lambda I)^{-1}\hat{\beta}$$ Now, for $\lambda \rightarrow 0$, ridge regression approaches ordinary ...


1

Ridge solution: $$\hat{\beta_\lambda} = (X'X + \lambda I)^{-1}X'y$$ If my matrix algebra is right, derivative of Ridge with respect to $\lambda$: $$\frac{\partial \hat{\beta_\lambda}}{\partial\lambda} = -(X'X + \lambda I)^{-2}X'y$$ which is: $$\frac{\partial \hat{\beta_\lambda}}{\partial\lambda} = -(X'X + \lambda I)^{-1}\hat{\beta_\lambda} = -A_\lambda \...


2

I think the key intuition to think of here is a situation where you are interested in modelling a single outcome of interest, $Y$, with three possible independent variables, $W$, $X$, $Z$. Imagine in a standard regression, $W$ and $X$ are found significant whereas the coefficient on $Z$ is found insignificant and close to zero, although $Z$ and $Y$ are ...


1

Lagrange multipliers are fine but you don't actually need that to get a decent intuitive picture of why eigenvectors maximize the variance (the projected lengths). So we want to find the unit length $w$ such that $\|Aw\|$ is maximal, where $A$ is the centered data matrix and $\frac{A^TA}{n} = C$ is our covariance matrix. Since squaring is monotonically ...


1

Using basis expansion one can easily extend simple linear regression into non-linear models. Here is an example of how basis expansion works (with Fourier and polynomial basis). Depending on the data, we can chose the right model to fit. In the link, we are trying to fit a periodic data, so it is better to use Fourier basis. Note that it is one ...


4

Well, depending on the nature of the response variable $Y$ there are many possible regression models using only one predictor $x$ (and an intercept, mostly.) Logistic regression when $Y$ is binary Other glm's (generalized linear models) If $Y$ is ordinal, ordinal regression, How to handle ordinal categorical variable as independent variable If $Y$ is ...


Top 50 recent answers are included