1
$\begingroup$

I have a sample with a sample size of > 50,000 (With a mean of 0 and a known standard deviation).

Now, I'm picking a subsample of the bigger sample with a sample size of 5,000 (With a mean of 2 and a known standard deviation).

The goal is to find out, if the mean of the subsample is significantly differnt to zero (not different to the mean of the bigger sample, but to zero!). Therefore, I would do a one-sample t-test (wikipedia one-sample t-test).

enter image description here

Which standard deviation s is the right one to use? Following wikipedia, the standard deviation of the subsample should be taken. However isn't this usually done, because the standard deviation of the subsample is usually the best approximation of the standard deviation of the whole population? Wouldn't it be more accurate to take the standard deviation of the bigger sample in my case?

Edit for clarification (Hope this helps):

I have the following:

  • Whole Population: Mean is known (=0), Standard Deviation (SD) is unknown
  • Subpopulation (Sample): Mean is known (=0), SD is known (representative for whole population)
  • 10 Subsamples of subpopulation: Mean is known ( !=0), SD is known and can be slightly different to subpopulation SD.

The subsamples are created by a classification, which was done before. But, the subamples are not classified by the variable, which is now used for calculating the mean or SD! In my understanding, the subsamples are therefore random.

Imagine the following simplified example:

  • Whole Population: Stock universe
  • Subpopulation: Representative Index
  • 10 subsamples: Stocks ranked by their market capitalization (e.g. subsample one consists of all the smallest stocks, subsample 10 of the biggest stocks)
  • Target Variable: Relative returns of the stocks to the index. Mean and SD are calculated over the returns of all stocks in one subsample.

Information which I want from the t-test:

  • Is the mean of a subsample significantly different to zero

My Question:

  • Is the t-test the right way to do this?
  • If yes, do I use the subsample SD or the subpopulation SD for calculating the t-values?
$\endgroup$
1
  • 2
    $\begingroup$ It is not legitimate to apply that test because it assumes the subsample was obtained randomly. Yours was not: it was obtained to have a mean of 2, period. Therefore its mean is 2 and no test is needed or appropriate. $\endgroup$
    – whuber
    Jul 16, 2017 at 17:23

1 Answer 1

3
$\begingroup$

I understand that your subsample belongs to a subpopulation of your whole population - if it didn't, your test wouldn't be very interesting. The first question is whether you have any reason to think that the standard deviation of the subpopulation is the same of the whole population. If there is a strong reason to suppose both are the same, use the known standard deviation.

If you don't know, you could test if both variances are the same of if variance in subpopulation is equal to the known variance and act accordingly.

However, since your subsample is very large, its variance will be a very good estimate of subpopulation variance. Therefore you can use a t-test on the subsample without worrying about variance of the population or the bigger sample.

Expanding after comment

From the comment, it seems that the subsample is a rather arbitrary subset of the sample. That could result in a few consequences:

  • If the subsample is not a representative sample of some population with known variance, this is a problem about a test with unknown population variance. Fortunately, with such a large sample this is not a problem and t-test can give good results.
  • One assumption of t-test is independence of observations in the sample. I think this assumption is unlikely to be violated in such a way that harmed the t-test, but it should be a concern.

Furthermore, the coment mentioned about 10 subsamples. Beware of the multiple comparisons problem: you can't just make 10 t-tests and claim significant findings according to the p-value of the most significant ones.

$\endgroup$
7
  • $\begingroup$ Yeah, you assumed everything correctly and there are good reasons, that the standard deviation of the subpopulation is representative for the whole population. But, for some subsamples it could be the case, that the subsample is not representative for the subpopulation or the population, because the subsamples are created in a certain way (machine learning classification which e.g. results in 10 subsamples/classes). Does that change your answer? $\endgroup$
    – SebastianB
    Jul 16, 2017 at 16:58
  • 1
    $\begingroup$ I expanded the answer. However you are not being clear about what you want to do, and that doesn't help to produce a useful answer. $\endgroup$
    – Pere
    Jul 16, 2017 at 17:21
  • $\begingroup$ In a comment to the question I argue that an answer like this cannot be correct: hypothesis tests of the mean are not appropriate, due to how the subsample was constructed. Abstractly, the question comes down to "I have a bunch of numbers constructed to have a mean of 2. Which standard deviation do I use to test whether 2 = 0?" The only conceivable correct answer has to be that $2\ne 0$ and the simplest "test" of that is to assert the result as a mathematical theorem. $\endgroup$
    – whuber
    Jul 16, 2017 at 17:24
  • 2
    $\begingroup$ I think that the question now clarifies that the subsample has not been cherry-picked to get a given mean, which caused the concerns that @whuber explained. $\endgroup$
    – Pere
    Jul 16, 2017 at 22:48
  • 1
    $\begingroup$ Both: If you actually know the subpopulation SD, use it (although it seems surprising to me that you know it), but with a sample size in the thousands range it doesn't matter if you take population SD or sample SD. $\endgroup$
    – Pere
    Jul 17, 2017 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.