-2
$\begingroup$

In the table below, how do I prove statistically that more people are in the group: "Sum of Agree", than is possible by chance?

enter image description here

I do not know which test to use here, (although I am using the Extended Cochran Armitage test in R to get similar groups by using Compact letter displays as a separate task). Maybe goodness of fit? Thanks, Rayyan

| cite | improve this question | |
$\endgroup$
  • 3
    $\begingroup$ It may be worth considering whether this is a useful question to ask. Wouldn't you find it more helpful to summarize the rates of agreement and disagreement among these respondents rather than merely establishing that in some hypothetical larger population there likely is greater agreement than disagreement? $\endgroup$ – whuber♦ Oct 29 '19 at 16:11
  • $\begingroup$ I just wanted to see if I can say with statistical confidence that More people agree than disagree. i.e. this was not a fluke. $\endgroup$ – Ray92 Oct 29 '19 at 16:35
  • $\begingroup$ Are you planning to ignore the neutrals? $\endgroup$ – Peter Flom Oct 30 '19 at 10:52
  • $\begingroup$ One note on language. In this kind of statistical analysis, you can't "prove" anything. You simply have better or worse evidence for or against a hypothesis. Also, in this kind of statistical analysis, nothing is not "possible by chance". If you flip a fair coin 1000 times, it may come up heads every time by chance. This isn't likely, but it's "possible" with this kind of model. $\endgroup$ – Sal Mangiafico Oct 30 '19 at 10:56
1
$\begingroup$

As the discussion on the original post and answers so far suggest, it's really critical to determine precisely the hypothesis you wish to test. Looking at the proportion of "agree" answers relative to "disagree"/"neutral" is a different question than treating the response as an ordinal variable with three levels (which may make more sense in most cases). Deciding what you really want to know is the first step.

Perhaps one flaw in the proposed analysis is that presumably your questions were answered by the same participants. If that is the case, you should take that dependence in observations into account. If the goal is to distinguish among questions, it seems that Friedman test may be well suited here. In that case you would translate the responses into an ordinal variable (e.g. -1 for disagree, 0 for neutral, 1 for agree). Then, an appropriate test to distinguish among questions can be found by looking at the list of tests in the PMCMRplus package documentation that start with "fdrAllPairs". (e.g. Conover test for Friedman, Nemenyi test for Friedman). Considering that Friedman is essentially an extension of the Sign test, I suspect this would work out well even though the response is one of only three ordered categories.

If you don't have the information on which participant answered which response for which question, probably one thing you could do is make note of this fact, and then use a test for ordinal data across independent groups, like Kruskal-Wallis, or potentially, extended Cochran-Armitage test. I'm not sure that these tests are ideal in cases where the response is just three ordinal categories. Appropriate tests among questions might then include Dunn test (1964) or Conover test for Kruskal-Wallis.

Another approach is to treat the responses as essentially binomial: Agree / Disagree, where Neutral is either ignored or grouped with one of those responses. Essentially this would be the Sign test in the paired/dependent case, or Mood's median test (or other appropriate median test) in the independent case.

For a better understanding of options, I would recommend looking up a) One-sample Sign test, b) Two-sample Sign test, c) Mood's median test, d) the other tests mentioned in this response.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, I have another question. If I am attempting to find groups of similar question s within a survey that measures a 1-7 scale, is the Extended Cochrane Armitage test the best way, or should I do a PCA analysis? $\endgroup$ – Ray92 Oct 31 '19 at 15:31
1
$\begingroup$

You could do this with a binomial distribution:

Step 1: You have to define what you mean with "by chance"! For example you define your background probability for someone to agree with any statement to be 50% or to be 33.3%, because you have 3 choices that you could see equally possible. (Maybe you could make another survey assessing this random probability to agree to a question by asking a lot of "random" questions..)

Step 2: Based on the "random probability" that you defined, the probability by chance to observe an equal or higher "Sum of Agree" can be calculated with a binomial distribution.

For example question 1:

p... Your probability.

n... 8

k... 6,7 and 8

EDIT: What I want to say is that you can't know the significance without defining the baseline probability!

For example if you assume random people tend to agree to random questions 70% of the times then your p-value for 6 of 8 people agreeing is p=0.55!

You can calculate this yourself using: https://stattrek.com/online-calculator/binomial.aspx

But how could you ever justify such an assumption? The problem is not just the low N, but the design of the questionnaire.

I would redesign the questionnaire:

Your questionnaire currently has this psychological aspect of "agreeing to statements" that you want to avoid.

Maybe it would be good to redesign the questionnaire by clearly separating between your observable (your website) and the background (other websites). For example ask them to rate certain aspects of your websites and ask the same question for "other websites" of the field. Then you have 2 nice groups and can do proper statistics!

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Something's fishy about step 2. This approach appears capable of simultaneously concluding there are "significantly large" numbers of people who disagree and significantly large numbers of people who agree! That would occur in any sufficiently large dataset with relatively few neutral responses. $\endgroup$ – whuber♦ Oct 29 '19 at 16:41
  • 1
    $\begingroup$ Re the "infinitesimal p-value:" aren't you assuming all responses are independent? That seems highly implausible. $\endgroup$ – whuber♦ Oct 29 '19 at 16:45
  • $\begingroup$ You are right! But what I meant is that already the probability of a single question is very low. For example is you assume 33.333% then question 10 has a p-value of p=0.00015! I googled to find: stattrek.com/online-calculator/binomial.aspx $\endgroup$ – KaPy3141 Oct 29 '19 at 16:52
  • $\begingroup$ Exactly what null and alternative hypotheses would that be a p-value for? In light of my previous comments, I would suggest a good case could be made that it's grossly exaggerated. We are looking at what looks to be a survey of as few as eight people. We don't even know if they're giving consistent answers, but even if they are, around six feel positive and two feel negative about something. In a random sample of a population that is evenly divided, the chance of observing a split of that magnitude is $0.29,$ not significant at all. $\endgroup$ – whuber♦ Oct 29 '19 at 17:02
  • $\begingroup$ I just wanted to know if there is a test that could prove that these results indicate with certainty that more people chose to agree. IF these numbers are insufficient, what is the minimum sample size I should look for (but surely that depends on which test I am using)? $\endgroup$ – Ray92 Oct 29 '19 at 17:14

Not the answer you're looking for? Browse other questions tagged or ask your own question.