14
$\begingroup$

How significant is a value compared to a list of values? In most cases statistical testing involves comparing a sample set to a population. In my case the sample is made by one value and we compare it to the population.

I am a dilettante in statistical hypothesis testing confronted with perhaps the most basic problem. It is not just one test but hundreds of them. I have a parameter space, and must do a significance test for every point. Both value and background list (population) are generated for each parameter combination. Then I am ordering this by p-value and find interesting parameter combinations. In fact, the finding of parameter combinations where this p-val is high (nonsignificance) is also important.

So let's take one single test: I have a computed value generated from a selected set and a background set of values computed by choosing a random training set. The computed value is 0.35 and the background set is (probably?) normally distributed with a mean of 0.25 and a very narrow std (e-7). I actually don't have knowledge on the distribution, because the samples are computed from something else, they are not random numbers samples from some distribution, so background is the correct word for it.

The null hypothesis would be that "the mean of the sample test equals my computed value, of 0.35". When should I consider this to be a Z-test or a T-test? I want the value to be significantly higher than the population mean, therefore it is a single-tailed test.

I am a bit confused as to what to consider as a sample: I either have a sample of one (the observation) and the background list as the population OR my sample is the background list and I am comparing that to the whole (unsampled) population which according to the null hypothesis should have the same mean. Once this is decided, the test goes to different directions I guess.

If it is a T-test, how do I compute its p-value? I would like to compute it myself rather than using an R/Python/Excel function (I already know how to do that) therefore I must establish the correct formula first.

To begin with, I suspect a T-test is a bit too general, since in my case the T-test would be linked to the sample size and would have the form: $$T=Z/s,$$ where $$Z=\frac{\bar{X}}{\frac{\sigma}{\sqrt{n}}}$$ and s is $$s=\hat{\sigma}/\sigma$$, the sample std versus the population std. So I have two cases: either my sample size is the size of the population, which I "guess" would mean I am dealing with a Z-test, or the population statistics (n and std) are unknown but the distribution can be in some way approximated and I am really dealing with a T-test. In any case my following questions are:

  1. How do I compute a p-value? (i.e. not using an R/Python/Excel function or p-value table look-up but actually compute it based on a formula, because I want to know what I am doing)
  2. How do I decide a significance threshold based on my sample size? (a formula would be nice)
Improve this question
$\endgroup$
3
  • 5
    $\begingroup$ Why test? Chebyshev's Inequality implies that in any real population it is mathematically impossible for the largest value to be $10^6$ SDs above the mean, but that is what you observed ($0.35 = 10^6 \times 10^{-7} + 0.25$). Therefore the $0.35$ did not come from your population, period. $\endgroup$
    – whuber ♦
    Commented Feb 28, 2012 at 19:26
  • 1
    $\begingroup$ @grokkaine - This question raises interesting issues and seems valuable, but I'd find it even more valuable if you edited it a bit, taking care to be very precise with your terms. $\endgroup$
    – rolando2
    Commented Feb 29, 2012 at 0:48
  • $\begingroup$ It is not just one test but hundreds of them. I have a parameter space, and must do a significance test for every point. Both value and background list (population) are generated for each parameter combination. Then I am ordering this by p-value and find interesting parameter combinations. In fact, the finding of parameter combinations where this p-val is high (nonsignificance) is also important. I'll try to edit my post a bit later. $\endgroup$
    – grokkaine
    Commented Feb 29, 2012 at 10:57

2 Answers 2

Reset to default
12
$\begingroup$

You raise an interesting question. First thing first, if you have an observation of 0.35, a mean of 0.25, and a standard deviation of 1/10^7 (that's how I interpret your e^-7 bit) you really don't need to go into any hypothesis testing exercise. Your 0.35 observation is very different than the mean of 0.25 given that it will be several thousands standard deviation away from the mean and it will probably be several millions standard errors from the mean.

The difference between the Z-test and the t-test refers mainly to sample size. With samples smaller than 120, you should use the t-test to calculate p values. When sample sizes are greater than that, it does not make much difference if at all which one you use. It is fun to calculate it both ways regardless of sample size and observe how little difference there is between the two tests.

As far as calculating things yourself, you can calculate the t stat by dividing the difference between your observation and the mean and divide that by the standard error. The standard error is the standard deviation divided by the square root of the sample size. Now, you have your t stat. To calculate a p value I think there is no alternative than to look up your t value within a t test table. If you accept a simple Excel alternative TDIST(t stat value, DF, 1 or 2 for 1 or 2 tail p value) does the trick. To calculate a p value using Z, the Excel formula for a 1 tail test is: (1 - NORMSDIST (Z value). The Z value being the same as the t stat (or the number of standard error away from the mean).

Just as a caveat, those methods of hypothesis testing can get distorted by sample size. In other words, the larger your sample size the smaller your standard error, the higher your resulting Z value or t stat, the lower the p value, and the higher your statistical significance. As a short cut in this logic, large sample sizes will result in high statistical significance. But, high statistical significance in association with large sample size can be completely immaterial. In other words, statistically significant is a mathematical phrase. It does not necessarily mean significant (per Webster dictionary).

To get away from this large sample size trap, statisticians have moved on to Effect Size methods. The latter use as a unit of statistical distance between two observations the Standard Deviation instead of the Standard Error. With such a framework sample size will have no impact on your statistical significance. Using Effect Size will also tend to move you away from p values and towards Confidence Intervals which can be more meaningful in plain English.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer, I am a bit confused as to what to consider as a sample: I either have a sample of one (the observation) and the background list as the population OR my sample is the background list and I am comparing that to the whole (unsampled) population which according to the null hypothesis should have the same mean. Once this is decided, the test goes to different directions I guess. $\endgroup$
    – grokkaine
    Commented Feb 29, 2012 at 11:07
  • $\begingroup$ Use all the observations you have as the sample (whatever you call it). And, compute the statistical distance between your one observation and the mean of the sample as defined. Calculate the standard deviation and standard error of your sample. And, the statistical distance of your observation from the mean is: (Observation - Mean)/Standard error = t stat. Use the Excel TDIST function (DF, t stat, 1 (for one tail)) and you get your p value. $\endgroup$
    – Sympa
    Commented Feb 29, 2012 at 16:58
5
$\begingroup$

Hypothesis testing always refers to the population. If you want to make a statement about the sample, you do not need to test (just compare what you see). Frequentists believe in asymptotics, so as long as your sample size is big, do not worry about the distribution of your data. Z-test and T-test do basically the same in terms of calculating the test statistic, just the critical values are obtained from different distributions (Normal vs Student-T). If your sample size is large, the difference is marginal.

Regarding Q1: Just look it up from the T-distribution with n-1 degrees of freedom, where n is the sample size.

Regarding Q2: You compute the threshold based on your desired significance level for a Z-test, and based on significance level on sample size in case of the T-Test.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for the answer. It was in fact the t-dist that I was using, but I wanted to also understand "why" do I use it. How do you define a "large" sample and how is the p-value different. More importantly, how do we know when a distribution is normal or student-t? Is there a statistical test for it? Maybe use the kolmogorov-smirnov test for the second and hmm.. what to use for the first? $\endgroup$
    – grokkaine
    Commented Feb 28, 2012 at 17:51
  • 2
    $\begingroup$ large ... well Z and t converge starting with n=60. Just compare the p-values you get from both tests. The t/Normal distribution assumption does not depend on the distribution of the underlying data. It is based on the assumption, that the sampling distribution of the mean is normal. Even if the variable you are testing is distributed Gamma, that still holds. With n=200 or so it should work fine. Again, all this stuff is based on frequentist statistics. $\endgroup$
    – joint_p
    Commented Feb 28, 2012 at 18:12
  • $\begingroup$ +1 for the comment on hypothesis testing always refering to the population but -1 for seeming to miss the point that the questioner has a sample of 1. $\endgroup$
    – Peter Ellis
    Commented Feb 28, 2012 at 18:15
  • 1
    $\begingroup$ I wasn´t really sure what "I have a computed value and a background set of randomly generated values. The computed value is 0.35" was supposed to mean ... I thought this somehow implied that there is more than 1 observation. $\endgroup$
    – joint_p
    Commented Feb 28, 2012 at 19:23
  • $\begingroup$ re-posting my comments from the other paragraphs: I am a bit confused as to what to consider as a sample: I either have a sample of one (the observation) and the background list as the population OR my sample is the background list and I am comparing that to the whole (unsampled) population which according to the null hypothesis should have the same mean. Once this is decided, the test goes to different directions I guess. $\endgroup$
    – grokkaine
    Commented Feb 29, 2012 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.