0
$\begingroup$

Thanks for reading. I wanted to ask if my logic I used for my experiment makes any sense. My hypothesis is that watching Pepsi commercial increases Pepsi preference.

My experiment has 2 steps.

(1) Baseline measurement

I have gathered 80 participants and asked them to choose one of 2 drinks (Coke or Pepsi) for 20 days. If they chose Coke we gave them a can of coke, and vice versa. (They could also choose neither of them and get nothing.)

We tracked the preference and found that (1) they preferred Coke over Pepsi and (2) coke-preference was getting even more evident.

To confirm the 2nd finding, I performed the linear regression and the result is shown below(in red line).

enter image description here

(2) Preference Shift Test

In 21st day, before making a choice, I divided 80 people in to 4 groups (80 into 20, 20, 20, 20).

To 1st group I showed them Pepsi advertisement. To each of other group, I showed them different advertisements (group2: Fanta, group3: Sprite, group4: Dr. Pepper).

And I found that Pepsi-choice was increased to 60% in group 1, while in group 2~4, it slight decreased. And my goal was to show the increase in group 1 is statistically significant.

To do this, My plan was as below:

(a) Estimate the baseline level for choosing Pepsi using linear regression model

(b) Compare the baseline level to measured proportion (60%) by one-sample binomial test.

For (a) estimating the baseline level, I used the result from the linear regression above (Y=-0.01*X + 0.43). So I changed "X" to 21 and got Y = 22%.

Here's my question.
Would it be correct to use such 22% as chance level for one-sample binomial test? I am unsure because although it is based on the model prediction, I haven't seen anyone use binomial test like this before.

I'm having trouble because I cannot use two-sample proportion test (z-test) as my sample is not big enough to meet the assumptions (np>=10, n(p-1)>=10). I also can't use Fisher's Exact test nor Mcnemar Test.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

  1. The estimate of 22% is still an uncertain quantity. It's based on a sample. It has its own standard error. If you repeated these experiments you would not get the same fitted line and in turn not quite 22% the next time. You should not ignore this uncertainty.

  2. By looking at the data to choose the model for the same data, you introduce bias into the estimate calculated on that same data.

This question appears to be posing an XY problem. You're coming up with what seem to you to be good solutions to some problem and then asking about whether your solutions work. This could be a very long process as you may be able to come up with hundreds of things that don't work properly.

It would be much better to post a question asking about a good way to deal with the problem your suggested approaches were trying to solve.

https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem

http://mywiki.wooledge.org/XyProblem

https://en.wikipedia.org/wiki/XY_problem

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for the answer and advice. Never looked in that way. I'll improve on my next question. For your answers, if I tested all the other groups (group2~4) on the same test and did not get a significant difference, could the result (which is significant difference from the binomial test of group 1) be used to support my hypothesis even with the uncertainty? $\endgroup$
    – Roas Clack
    Commented Apr 27, 2022 at 3:19
  • $\begingroup$ Because the uncertainty and bias will also effect the other groups too, so it feels fair to use the uncertain quantity. But it also feels like this could be very wrong. $\endgroup$
    – Roas Clack
    Commented Apr 27, 2022 at 3:28
  • $\begingroup$ It is wrong. More things being uncertain doesn't result in less uncertainty about their differences, but rather, more uncertainty. Several things being biased only results in smaller bias on their difference (than the largest of the individual biases) if the biases are all in the same direction. This is not a safe bet in general. Your intuition is not serving you well on these issues. $\endgroup$
    – Glen_b
    Commented Apr 27, 2022 at 3:44
  • $\begingroup$ What do you think would be the correct way to show that only group 1 was changed in statistical significance? I couldn't 2-sample binomial test (day 20 vs day 21)because it did not meet the assumption of np>=10 (or even 5). Nor could I use Mcnemar Test . So I considered using 2x4 Fisher exact test (Freeman-Halton extension) to compare proportion of group1~4 on 21st day, but realized that I had 1 test group vs 3 control groups so it doesn't really fit for my situation. Any advice will be so thankful. $\endgroup$
    – Roas Clack
    Commented Apr 27, 2022 at 4:06
  • $\begingroup$ This would be a good subject of a new post. Among the things you say you couldn't do, you seem to have errors (e.g. either mistaken understanding of correct terminology re what a binomial test is or a mistaken understanding of the conditions for it) $\endgroup$
    – Glen_b
    Commented Apr 27, 2022 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.