25

Hypothesis tests are still used because they are motivated by a different need in statistical inference than interval estimators are motivated by. The purpose of a hypothesis test is to make a decision as to whether there is evidence for the alternative hypothesis' expression of the population parameter. Confidence intervals serve a different purpose: they ...


18

$\exp(\mathbb E[\log(X)])$ is the geometric mean of a positive random variable $X$ not the mean of a geometric random variable. So either the homework directions put the words in the wrong order, or you transcribed them incorrectly


16

I am going to assume that you are referring to the conditional distribution of $Y$ in the regression (i.e., given the explanatory variables), which follows directly from the underlying error distribution. So you are really asking what happens when the underlying error terms are not normally distributed. The distribution of the OLS estimator is quite robust ...


14

One reason to use traditional hypothesis testing methods (when they can be used) is that it is computationally efficient to do so compared to bootstrap sampling. Depending upon the number of dimensions in your data, the number of bootstrap samples required to estimate p values (or confidence intervals) can be very large. Central limit theorem is not always ...


11

Why do we multiply log likelihood times -2 when conducting MLE? We really don't. The -2 was not about parameter estimation; for that, we'd just use the (negative log-)likelihood. It was about hypothesis testing. Your intuition about negation is correct. Traditionally, in the optimization literature, we minimize functions. It's easy enough to convert a ...


11

Let $R = (\rho_{ij})$ be the correlation matrix so that the covariance matrix is $\Sigma = \sigma^2 R.$ Consider $\mathbf x = (x_1,\ldots, x_n)^\prime$ to be a single observation of the $n$-variate Normal distribution with zero mean and $\Sigma$ covariance. Because the log likelihood of $N\ge 1$ independent such observations is, up to an additive constant ...


11

Let's see what actually happened with your example You started with a mixture of three normal distributions, where the probability of exceeding $\$80k$ was about $0.32576$ You used this to construct a population of $300000$ with $97751$ cases exceeding $\$80k$, a proportion of about $0.32587$ You sampled $15000$ without replacement from the $300000$ ...


10

Consider a population $Y|X$ that follows some distribution according to a true model, and you have a set of trained models $f(X,\theta)$ that make predictions of $Y$ given $X$ and are parameterized by $\theta$. The goal is to find out what the error of the models is, in making predictions about samples from the population, as function of the parameter $\...


9

The score for a multiple parameter problem (a vector parameter) is itself a vector. We need to take partial derivatives of the log likelihood with respect to each model parameter. Let's consider an example. Find the score vector for $X_1, X_2, \ldots, X_n \sim N(\mu, \sigma^2)$ where the $X_i$ are iid $N(\mu, \sigma^2)$ samples. Let $\mathbf{x} = (x_1, x_2, \...


9

Your least squares loss function is $$ \ell(\beta_1,\beta_2) = (Y_1 - \beta_1)^2 + (Y_2 - \beta_2)^2 + (Y_3 - \beta_1 - \beta_2)^2 $$ Now, it's not too difficult to take derivative wrt to $\beta_1$ and $\beta_2$, set them both to zero, and solve the system of 2 equations and 2 unknowns. After doing this you will obtain $$ \hat\beta_1 = \dfrac{2Y_1 - Y_2 + ...


8

There are important philosophical differences between a frequentist and a Bayesian approach. For this reason it is not appropriate to say that "if you use uniform priors in Bayesian Analysis, it is the same as doing Frequentist Analysis". The frequentist idea of probability is that we assume there is an underlying true process that generates the ...


8

Furthermore, the Non-Parametric Bootstrap also allows you to evaluate population inference and confidence intervals - regardless of the population's true distribution. Thus, why do we still use classical hypothesis testing methods? The only reason I can think of, is when there are smaller sample sizes. But are there any other reasons? If you have a large ...


7

There are many reasons for choosing a model that actually reflects the data generating process. A model tailored to binary Y will not give predictions outside of [0,1] The model will not require silly looking interactions to be added to it just to keep the [0,1] constraint When normality of residuals is not satisfied (as needed by OLS), statistical ...


7

Actually, quadratic loss function $\mathcal L (y,\hat y)=(y-\hat y)^2$ and OLS can be applied to binary outputs. Some people do it. However, when the dependent variable (DV) is binary, usually, cross entropy loss $y \ln \hat y$ is used. So where does this entropy loss come from? In fact, the real question is: how do you pick a loss function? Why even ...


6

The M in MAP stands for "Maximum", so it should be or no surprise that optimization is involved . In order to obtain estimates for the parameters, you compute $$ \hat{\theta} = \underset{\beta \in \mathbb{R}^2}{\text{argmax}} \left\{ \text{Log Prior} + \text{Log Likelihood} \right\}$$ You can't do MAP with samples from the posterior (and I'm not ...


6

One obvious benefit is that given a probability distribution and a loss function, you can derive the optimal point estimate, i.e. one that minimizes the expected loss. This is not strictly specific to Bayesian methods but to any method that delivers a probability distribution instead of a point or an interval estimate. However, Bayesian methods are perhaps ...


6

Let $X_{1},X_{2},\dots ,X_{n}$ are $n$ random samples drawn from a population with overall mean $\mu$ and finite variance $\sigma ^{2}$ and if $\bar {X}_{n}$ is the sample mean, then We know sample mean (statistic) is an unbiased estimator of the population mean (parameter) i.e., $E[\bar{X_n}]=\mu$ By SLLN we have $\bar{X_n}\overset{a.s.}{\rightarrow}\mu$ ...


5

I don't think an entire data presentation is needed to give some intuition behind this phenomenon. While we would expect that a logistic regression and OLS model will, on average, produce parameter estimates (slopes or log-odds ratios) that are similar sign, it's entirely possible they will disagree for a given dataset and analysis. The most likely issue is ...


5

I find the literature in MLE a bit fuzzy with nomenclature here, so I might have some stuff off, and I will try to stick to the nomenclature you introduced. We have the observed Fisher information: $$\left[\mathcal {J}(\theta)\right]_{ij} = -\left(\frac{\partial^2 \log f}{\partial \theta_i \partial \theta_j}\right)$$ And the empirical Fisher information: $$\...


5

Maximum-likelihood is applied mathematical optimisation --- learn the latter well This is too large a field for us to give you a comprehensive answer, but perhaps we can point you in the right direction to find the resources you need. The first thing to stress here is that, mathematically speaking, all forms of maximum likelihood involve maximising a ...


4

Yes, you can absolutely do this. One of the great things about resampling methods (including the bootstrap and jackknife) is that they really don't care how you came up with your point estimator. One thing to realize, though, is that by using these tools you're estimating the sampling variance of your estimator, which is not generally the same thing as the ...


4

This is a more general result without assuming of Normal distribution. The proof goes along the lines of this paper by David E. Giles. First, we consider Taylor's expanding $g(x) = \sqrt{x}$ about $x=\sigma^2$, we have $$ g(x) = \sigma + \frac{1}{2 \sigma}(x-\sigma^2) - \frac{1}{8 \sigma^3}(x-\sigma^2)^2 + R(x), $$ where $R(x) =- \left(\frac{1}{8 \tilde \...


4

The posterior expectation writes as \begin{align*}\mathbb E^\pi[\theta|x]&=\dfrac{\mathbb E^\pi[\theta^2x+\theta(1-\theta)(1-x)]}{\mathbb E^\pi[\theta x+(1-\theta)(1-x)]}\\ &=\dfrac{\mathbb E^\pi[2\theta^2 x-\theta x +\theta-\theta^2]}{\mathbb E^\pi[2\theta x+1-\theta-x]}\\ &=\dfrac{\mathbb 2\frac{3}{8} x-\frac{1}{2} x +\frac{1}{2}-\frac{3}{8}}{2\...


4

C1: Yes, by the argument for C2. C2: Yes, this is possible. The key result here is that any continuous function $f$ on $[0,1]$ is also uniformly continuous: for any $\epsilon$, there is some $\gamma$ such that if $|x_1-x_2|<\gamma$ then $|f(x_1)-f(x_2)|<\epsilon$. So it is enough to have a probability of $1-\delta$ of approximating $E[X]$ to within $\...


4

You can apply GLS equations, e.g. top of p.5 in these lectures: http://halweb.uc3m.es/esp/Personal/personas/durban/esp/web/notes/gls.pdf The equation for MSE is $\sigma^2=1/n \sum_{ij} x_i r^{-1}_{ij} x_j$ if you set X to zero and correspond your r to their V This equation doesn’t work for extreme cases such as correlation 1 between all variables. This case ...


4

I think you want isotonic regression, which would consistently estimate $g(0)$ even if $Z$ was sampled from a continuous distribution (though only at $n^{-1/3}$, not $n^{-1/2}$ rate). There's a fast algorithm ('pool adjacent violators') and quite a lot of theory.


4

Can you use this model for time series analysis? Theoretically, yes. But you would have to test your assumptions here before using this model for practical application. Here, you are assuming a deterministic component $Ct^3+Dt^4$ and a stochastic component which is modeled using an AR(2) process $$X_t=a_1X_{t-1}+a_2X_{t-2} + \epsilon_t.$$ You are also ...


4

The bootstrap cannot resample events that didn't occur in the dataset. If the probability of some event was very low and ends up not occuring -- e.g., the expected number of events in some range or bin is less than 1, and indeed zero are obtained -- the bootstrap procedure will of course never be able to produce (re)samples with a non-zero number of such ...


4

No, it's not related to Stein's Paradox; instead, it's related to Generalized Least Squares (GLS.) In SUR, there's an assumption that the regression errors of the individual equations are correlated across equations. This leads to a situation where you can, conceptually, construct a single, large, regression out of the component regressions, one which will ...


4

Because $Y$ has bounded support $\mathcal S,$ the set $\mathcal S \Sigma^{-1}X = \{y\Sigma^{-1}X\mid y\in\mathcal S\}$ is bounded, too. Let $c\gt 0$ be any upper bound of the norms of its elements. For $\beta\in\mathbb{R}^m$ and $y\in\mathcal S,$ the Cauchy-Schwarz Inequality implies $$|y \Sigma^{-1}X\beta| \le ||y\Sigma^{-1}X||\,||\beta|| \le c||\beta||.$$...


Only top voted, non community-wiki answers of a minimum length are eligible