2
$\begingroup$

In my research, I aim to compare websites and need to identify if the websites differ significantly from each other. I have divided the website into two categories. I have sampled 45 websites in the first category and 30 websites in the the other category. I need to identify if the first set of websites differ significantly from the second set. My data is largely nominal, as I just count the presence or absence of certain elements on the webpages.

So, during my data collection stage, I counted the Presence or Absence of certain elements on these website, and calculated the the percentage of occurrence of these elements, for the two sets of websites.

When I had the percentage of each elements for both the sets, I applied the Two Sample Test of Proportions, and tested my hypotheses (Is one set different from the other in terms of element A, B, C...etc). I had 60 elements, and I applied the test separately 60 times for each element.

My results do make sense, but I am just not sure if the test is a valid measure for my research. Can someone please help.

$\endgroup$
2
  • $\begingroup$ Why are you not sure if the test is a valid measure for my research? $\endgroup$
    – ttnphns
    Commented Jul 18, 2012 at 13:44
  • $\begingroup$ well, mainly because its the first time i am applying it and when i read about it, it says that the test assumes Normal Distribution. I am not sure if I can assume this. $\endgroup$
    – MaO
    Commented Jul 18, 2012 at 16:59

1 Answer 1

2
$\begingroup$

The testing is fine. The multiplicity (60 tests or more). The p-value for each test does not address familywise error rate from multiple tests. This can be handled by p-value adjustment. Westfall's resampling methods or other well-established approaches can be used to adjust or bound the p-value. The other possible approach to multiplicity is to control the false discovery rate. False discovery is often used when a large number of tests are employed.

$\endgroup$
6
  • 1
    $\begingroup$ To elaborate on @Michael Chernick's point, suppose you are testing at 5% level of significance, then, even if all of the 60 null hypotheses are true, the expected number of incorrect rejection is 3, and this number increases with increasing number of tests. To control this issue that arises with multiplicity, we can apply Bonferroni correction (reducing $\alpha$ to $\alpha/n$), Sidak correction (reducing $\alpha$ to $1-(1-\alpha)^{\frac{1}{n}}$) or control quantities like FWER or FDR. A brilliant reference is Bradley Efron's IMS monograph titled 'Large scale Inference'. $\endgroup$ Commented Jul 18, 2012 at 15:50
  • $\begingroup$ Thank you very much for the prompt response. I still have one confusion; I have read that the test assumes normal distribution. Is it fine if I assume so for my study? Or can I apply some other statistical techniques? $\endgroup$
    – MaO
    Commented Jul 18, 2012 at 17:08
  • $\begingroup$ @user12694 The test assumes the binomial which really falls out by assuming that the presence of the elements are IID Bernoulli with probability of occurrence 0<p<1. Sometimes the normal approximation to the binomial is used but that is a valid approximation based on the central limit theorem and not an assumption of normality. However the exact binomial test can be used (or alternatively Fisher's exact test). $\endgroup$ Commented Jul 18, 2012 at 17:23
  • $\begingroup$ I really don't understand the IID Bernoulli part :-( Will the Fisher's test be better than the test sample test of proportions, for testing my hypothesis? $\endgroup$
    – MaO
    Commented Jul 18, 2012 at 18:50
  • $\begingroup$ They are really pretty much the same. Fisher puts the proportions into a 2x2 contingency table as counts and then requires the marginal totals to be fixed when looking at the distribution under the null hypothesis. IID Bernoulli means each time there is a probablity p of success and 1-p for failure and the cases are independent of each other. For you success equates to presence of the element. $\endgroup$ Commented Jul 18, 2012 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.