1
$\begingroup$

I have conducted a survey where 410 people were given 14 brands. Of those 14, they were asked, "Which brand do you consider to be the "#1 Trust Brand for Protection". Respondents could only choose one brand for their answer.

I have the frequency and proportion results for the brands selected (using Stata)

      Choice           Freq.    Perc.       Cum.
      Brand 1|         70       17.07       17.07
      Brand 2|         64       15.61       32.68
      Brand 3|         62       15.12       47.80
      Brand 4|         46       11.22       59.02
      Brand 5|         27        6.59       65.61
      Brand 6|         25        6.10       71.71
      Brand 7|         24        5.85       77.56
      Brand 8|         22        5.37       82.93
      Brand 9|         18        4.39       87.32
    Brand 10 |         15        3.66       90.98
    Brand 11 |         14        3.41       94.39
    Brand 12 |          9        2.20       96.59
    Brand 13 |          9        2.20       98.78
    Brand 14 |          5        1.22      100.00

I am tasked with determining if Brand 1 is statistically significantly higher than Brand 2 (and then eventually all other individual brands).

I have made each Brand its own binomial variable (i.e. 0/1 for each brand, if selected)

Here are my constraints for identifying the appropriate statistical test:

  • There is only one "group" (the n=410 sample). I'm not testing between any different groups
  • I know that this is a one-sample test, because it's the same respondent across the variables.
  • The test has to be appropriate for binomial variables (i.e. Brand 1 select 0/1).

I feel like I can't do a one-sample proportion test (such as prtest Brand1 = 0.0714 ) because I know the brands are NOT equally likely to be selected, since more popular brands are more likely to be chosen (I have a mix of popular and unknown brands).

I have followed this website (https://www.ssc.wisc.edu/sscc/pubs/sfs/sfs-prtest.htm) and essentially did prtest Brand1 == Brand2), knowing that this really isn't 2 sample. Is that proper?

I don't think a t-test is right because the brand variables (0/1) aren't independent (since they come from the same respondent). I don't think a paired t-test or McNemar is right (no before/after manipulation) or a chi-square is correct (such as tabulate Brand1 Brand2, chi2) because I don't have an expected distribution or outcome.

Can anyone help me understand what indeed is the right statistical test? I'm stuck because I don't have an expected frequency or proportion.

Improve this question
$\endgroup$
3
  • $\begingroup$ This problem is very amenable to a brute-force Bayesian approach, i.e. treat it as 14 separate binomial estimation problems where you calculate a credible interval for the binomial parameter for the incidence for each brand, and results from brands with intervals that don't overlap are statistically significant from one another. But they're not making you use frequentist methods, are they? $\endgroup$
    – NaiveBayesian
    Jun 14, 2021 at 1:28
  • $\begingroup$ Hi @NaiveBayesian - no one has dictated any specific test to me. To your point, I've looked at CIs. 100% right if the CIs don't overlap, then there's a statistically significant difference. However, in some cases, where the brands' CIs barely overlap, it's my understanding that this overlap does not necessarily indicate that the means are not statistically different. Is that a proper understanding? (See "Mistake #1: link ). I will admit that I'm not versed in Bayesian methods. $\endgroup$
    – Kari P
    Jun 14, 2021 at 11:48
  • $\begingroup$ You are correct, just looking at the overlap is definitely a conservative test. On the other hand, although it's still taught in the textbooks, problems with the use of p-values as the final word on statistical significance are being widely acknowledged. Note especially point 6 here, though for a simple binomial model like this, a p-value based test might be reasonable. I will post a more robust method, as a full answer below. $\endgroup$
    – NaiveBayesian
    Jun 15, 2021 at 2:46

1 Answer 1

Reset to default
0
$\begingroup$

Here is a Bayesian approach that uses a posterior-predictive simulation:

  1. Estimate a binomial parameter for all 14 brands, i.e. for each brand, 410 is the number of trials and column "Freq" is the number of successes. Using the conventional Bayesian conjugate-prior approach, binomial parameters will be Beta distributed.

  2. In cases where credible intervals do not overlap, you are done: The results are statistically significant.

  3. For each pair of brands where the credible intervals do overlap, run the following posterior-predictive simulation, i.e...

    1. Combine the observations for the two brands into a single aggregate category, and calculate its binomial parameter.
    2. Use the results to specify a posterior-predictive distribution of beta-binomial form (note this is one of the few cases where the posterior predictive distribution can be calculated in closed form).
    3. Now, generate random pairs of simulated observations from your beta-binomial posterior-predictive distribution. Compare the difference (or ratio) of the simulated results to the difference (or ratio) of the observed result, and determine in what percentage of cases the simulated result is larger. That is your measure of statistical significance.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.