I am new here, so I might have missed a similar question. I am trying to do things good in my research and using proper statistical approaches, but in computer science, we had quite a weak training on statistical methods applied to our research.

So, what I am asking is probably trivial. Here is the problem: We have a dataset of del.icio.us (the social bookmarking website) and want to work on a subset of this dataset for a manual study of the usage of tags. In this dataset, we have urls, users and tags that are linked by triplets defining an association of a tag by a user to a url. We have 401 970 328 such triplets in our dataset, so we can't look at all of them and want to choose a sample of it.

Naively, at the beginning, we started with a purely random sampling to choose 500 user-bookmark pairs and all their associated tags. Because there are some more popular tags on del.icio.us, our sample has a long-tail distribution of tag usage, similar to the one from the whole dataset.

What I would like to check is that the actual distribution of each tag is similar in the sample as in the original dataset. That is, if "java" appears 10 times more often than "indonesia" in the dataset, it should be a similar distribution in the sample.

How would I go at doing this? one of the issue is that in the sample, some classes (tag) present in the full dataset will never appear.

The second question is also: how to decide of a good size for the sample?

  • $\begingroup$ @Mortimer, good question, can you tell us what you want to do with this sample ? I guess this info will help to get better answer about the size. From a cost loss perspective, the loss of small size is the loss of information which may be different depending on the use of the sample, and the cost of a large size is the computation cost that higly depends on what you do with the sample. $\endgroup$ Feb 17 '11 at 12:25
  • $\begingroup$ I am not an expert on this stuff, so someone please correct me if i am wrong. I agree with robin that it really depends on what you need to do with the data. In your case a small sample will lead to a loss of information from the original sample (missing tags), which means your sample will not be truly representative. For the more popular tags you can use a simple significance test (like a chi-square test), but for the rarer tags that are missing from your sample, their absence will be significant. It is up to you if this is acceptable. $\endgroup$
    – rm999
    Feb 17 '11 at 13:19
  • $\begingroup$ thanks. What we are trying to do is to show how search works on such tagging system and we need to manually annotate the tags. In particular, someone might use "javaisland", someone else "java_island" etc.. That's why we can't use the whole dataset. We assumed that randomly selecting tag won't make a difference, but it seems that this leads to a very "computer/web oriented" vocabulary. This seems to be the case in the whole delicious dataset apriori, but I would like to have a hard number to check that this distribution of vocabulary isn't just a feature of the sampling. $\endgroup$
    – Mortimer
    Feb 17 '11 at 14:39
  • $\begingroup$ so, to make it more concrete, the tag "design" appears a lot in the sample, it also appears a lot in the dataset and the tag "crazywebsite" rarelly appears in the dataset and rarelly appears in the sample. Now, I can looks at them all manually, but they don't all have exactly the same distribution and would expect that there is some test to tell me that the general distribution of each class is within some interval of the original distribution. $\endgroup$
    – Mortimer
    Feb 17 '11 at 14:46
  • $\begingroup$ @Mortimer and @rm999 if you don't use @name to answer then I can know that you are talking to me (the sofware detect @robin and send me a notification somehow when it comes). $\endgroup$ Feb 18 '11 at 13:34

I believe a chi-squared test is what you are looking for. Because your dataset has a long tail, many tags will not be sampled well or will not end up in your sample at all. You may want to look into Yates' chi-square test, which attempts to correct for this by loosening the standards of what is significant for rare tags.


What you might want to try is to take many bootstrap samples of 500 pairs and then build a confidence interval of this distribution to see if the population mean is included.

However you really first need to go through your tags and spell correct, replace synonyms etc.


-Ralph Winters

  • $\begingroup$ Ralph, I do not really understand what you mean by: "build a confidence interval of this distribution to see if the population mean is included.". What we want is to show properties such as the amount of spell variants and synonyms, so we are doing this, but if our sample is not representative of the general population, it's not clear that we can generalize. $\endgroup$
    – Mortimer
    Feb 17 '11 at 20:29
  • $\begingroup$ Mortimer, I am not sure exactly what you are trying to do. But I am suggesting that if you know that "javaisland" + "java_island" represent 2% of your population, you bootstrap sample from your 500 sample, say 10000 times. You obtain a confidence interval for your "javaisland" + "java_island" term frequency of 1.5% plus/minus 1%. Since the confidence interval contains the population mean of 2%, you have no reason to suspect that your sample is not representative for the term "java_island". $\endgroup$ Feb 18 '11 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.