I have already spoken to a statistician and he did not seem certain about the answer, so perhaps it may be of use to others. The study is not typical, so please bear with me while I try to explain it.

I have about 75,000 sentences of English conversations for a sample. I'm interested in whether sentences with a certain grammatical structure (specifically, missing verb phrases) tend to clump together. First, I calculated the relative frequency (empirical probability) of these sentences and got 910 hits/75,000 total sentences ≈ 0.012. To get confidence intervals for this statistic, I used bootstrapping with 10,000 resamples of the entire data set with replacement, and found that the 99% confidence interval is about 0.0109 - 0.0134.

To see whether and to what extent these sentences tend to be near each other in the data, I made a new sub-sample. For each sentence with a missing verb phrase, I counted how many sentences also had missing verb phrases in the five previous sentences. So, I have five "slots", each with a relative frequency of how often sentences in that slot lack verb phrases (i.e., the immediately preceding sentence is a slot, then the next preceding sentence, and so on). This gives me results like this:

Slot 1: 56 hits/910 total sentences = 0.061

Slot 2: 32 hits/902 total sentences = 0.035


My question is this. Can I compare these relative frequencies for each slot to the confidence interval I calculated from the whole data set? In other words, if slot 1 has a relative frequency of .061, that is beyond the 99% confidence interval from my bootstrap of the whole data set, so is it therefore valid to claim that this is higher than could be expected in that slot?

And if it isn't, is there a relatively straightforward way to make it work?

  • $\begingroup$ If you are looking at proportions why do you have to bootstrap to get confidence intervals? Even if you take random samples when you have subgroups with specific characteristics the distribution for the subgroup may have no relationship to the population distribution. So I don't see why you would think the confidence interval for the population proportion would tell you much about the proportion in the subgroup. $\endgroup$ Commented Sep 29, 2012 at 1:46
  • $\begingroup$ A polling analogy. Say Obama trails Romney 52% to 48% in Illinois based on a random sample of voter but it is known that Chicago is traditionally Democratic. Obama probably has a higher proportion of votes than Romney in Chicago but do you think knowing the result for the entire state would help you predict what the vote proportions would be restricted to Chicago? $\endgroup$ Commented Sep 29, 2012 at 2:01
  • $\begingroup$ @MichaelChernick: Thank you for your comments. Let's say we don't know that Chicago is more democratic, and that's what we want to know. You find that 2/5 people in the state are democrats. You get a confidence interval, and then find that in Chicago the proportion is higher than that confidence interval, would it be valid to say that Chicago is in fact more democratic than the state at large? Keeping in mind that this is all from the same poll. I'm not trying to predict anything as such, but merely trying to tell if the subgroup is in fact significantly different from the group overall. $\endgroup$
    – Alan H.
    Commented Sep 30, 2012 at 2:25
  • $\begingroup$ In other words, what I am expecting to find is that the subgroup is not predictable and does not have the same distribution compared to the group at large, precisely because the subgroup is not random in a certain theoretically interesting way. $\endgroup$
    – Alan H.
    Commented Sep 30, 2012 at 2:32


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.