0
$\begingroup$

I'm taking this course on statistical inference through datacamp.

The instructor went over the idea of a confidence interval. The instructor said that we take a large number of bootstrapped samples from our observed data, calculate a sample proportion from each bootstrap and then plot the distribution of the sample proportions.

The instructor said that, since it is normally distributed, we can capture 95% of the data in +/- 2 standard errors from the mean and, since it is 95% of the possible sample statistics, we can say we are 95% sure that interval captures the parameter.

The instructor called that the t-interval, but I'm googling the phrase and can't find anything.

Is that idea the same thing as the student's t-distribution?

$\endgroup$
  • 1
    $\begingroup$ Could you clarify what parameter you are computing a confidence interval for and what random variable must be normally distributed? $\endgroup$ – whuber Apr 23 at 6:43
  • $\begingroup$ In this case I am running an A/B test to see the difference in conversion rate based on a change I've made to the landing page. There is a statistically significant difference between the control and experimental group and I'd like to predict the real difference in conversion when we release the change to everyone. $\endgroup$ – Cauder Apr 23 at 14:36
  • $\begingroup$ That's a rather different question than the one you have asked! Which one do you really want answered? $\endgroup$ – whuber Apr 23 at 14:37
  • $\begingroup$ Haha! I guess I'm just trying to wrap my head around these concepts. That's the scope of the question, but I am trying to learn more about what the t-distribution is that we use for estimating that confidence interval $\endgroup$ – Cauder Apr 23 at 14:41
1
$\begingroup$

t-interval vs. t-distribuion

The answer to your explicit question is no. A t-interval is not the same thing as a t-distribution (AKA Student's t-Distribution). Instead, a t-interval is a confidence interval arising from a t-distributed test statistic. In my experience, it's not a common term. Most people just say "confidence interval" without specifying the distribution of the test statistic (cf. a z-interval, which is when the test statistic is normally distributed).


Computing Confidence Intervals

While the sampling distribution of means is normally distributed1, there are some inaccuracies in the remaining illustration. Some are a little nit-picky, for instance, in a normal distribution, 95% of the probability mass is within 1.95996... standard deviations from the mean, not 2 (in the case of a sampling distribution, your estimate of the standard deviation is the standard error of your sample). For the bigger problem, it's difficult to pinpoint exactly where it arises in your description, so I'll approach it more broadly.

You know that a confidence interval (CI) is bounded by points that are some number of standard errors away from the mean. Since all the points on the line are sample means, I'll refer to a potential CI boundary point as $\bar x$. The following formula computes how many standard errors a point is from the mean. $$t=\frac{\bar x-\mu}{s/\sqrt n}$$ Formally, $\mu$ is the population mean. Using your instructor's illustration, it is equivalently the mean of the sampling distribution. $s$ is the sample SD and $n$ is the sample size.

You can compute $t$ for every bootstrap sample as well and plot a histogram of those values. It will look a lot like the shape of the sampling distribution of means but with a key difference. While the distribution of $\bar x$ is normally distributed, $t$ is not. It is t-distributed. While similar, the t-distribution is usually described as having "fatter tails" than the normal distribution. The parameter that dictates the shape of the t distribution is the degrees of freedom (which equals the number of observations minus one). If you somehow had infinite degrees of freedom (i.e., having infinite observations in your sample), the t distribution would be identical to a normal distribution.

So now we need to know how many standard errors from the mean contains 95% of the probability mass of this t-distribution (the critical value, or $t^*$). It's not a constant value like with the normal distribution (~1.96). Instead, the critical value depends on the degrees of freedom parameter. The table here shows the value for different degrees of freedom. Look at the two-sided, 95% column; it is only exactly 2 for 60 degrees of freedom.

Once you know $t^*$, to obtain the 95% CI, you simply multiply that by the standard error and add to/subtract from the mean.$$95\% CI=\bar x \pm t^* \frac{s}{\sqrt n}$$


Definition/Interpretation of Confidence Intervals

Last point, words like "sure" are often unhelpful in defining a confidence interval. In fact, I don't really like the word "confidence" either. These words deal with belief, in common usage. The statistics you are learning are rooted in an interpretation of probability that does not have a rigorous definition of belief. Conflating probability and common-usage definitions of belief leads to extremely common misinterpretations of confidence intervals.

One way to conceptually define a confidence interval is to think about it in terms of a long series of identical experiments. If you were to run an experiment a large number of times, and each time you computed a 95% confidence interval using just the data from that run of the experiment, the intervals would be different every time, and 95% of those intervals will contain the population parameter.

Another interpretation that makes more sense for why confidence intervals are useful, since people rarely conduct the same experiment an arbitrarily large number of times, requires you to take the perspective of having not conducted the experiment yet. There is a 95% probability that when you run the experiment, the 95% confidence interval computed from the data will contain the population parameter. Note that this a statement about the probability of data, not the probability of the population parameter. This is not the same thing as saying that there is a 95% probability the population parameter is within your computed interval. There is nothing random or probabilistic about the population parameter. It is what it is. It is either in your interval or it is not. It's the data you collect, and therefore the interval values, that are random/probabilistic.


1 I use the phrase "sampling distribution of means" since it is more general, will lead to better subsequent search results for the OP, and a proportion is just the mean of a vector or 0-1, dichotomous data. Additionally, it is only approximately normally distributed. For example, a true normal distribution can't be bounded between 0 and 1. However, the core concepts in what is being illustrated by the instructor are often robust to these problems. But not always. Finally, I'm assuming that the bootstrapping procedure effectively created a sampling distribution as if all the samples were taken from the overall population.

$\endgroup$
  • 1
    $\begingroup$ It is difficult to read very far into this answer, because a (nonparametric) bootstrap distribution of a proportion is never t-distributed. Could you make your meaning more precise? $\endgroup$ – whuber Apr 23 at 14:39
  • $\begingroup$ Is a t-distribution used exclusively in the context of bootstrapped distributions of sample statistics? Also, is a t-distribution the same as a student's t? $\endgroup$ – Cauder Apr 23 at 14:39
  • $\begingroup$ @whuber In trying to infer what the instructor meant through shortcuts they took, I made some of my own that hopefully are corrected now. Cauder t-dist. and Student's t-dist. are the same thing. To your first question, the short answer is no. $\endgroup$ – le_andrew Apr 25 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.