# Compare and find p-value between two t-tests

I am running a statistical test to determine if females are more influenced by the framing effect. I designed a survey with three overall questions, each with a "positive frame" and a "negative frame". Each participant would be randomly chosen to answer either the positive or negative frame, and each frame would have a 1-6 Likert scale where participants would rate whether or not they would choose one of two options presented in the questions' scenarios (i.e. rate 1 for definitely choosing drug A and rate 6 for definitely choosing drug B). I already compared the means of the positive frame and negative frame for each gender using a two-sample t-test (see table below), and I was wondering how I would go about analyzing whether or not females are more influenced by the framing effect.

Essentially, I'm asking for a test/procedure that could provide a p-value or other measure that can identify if the difference between the frames for each question, for each gender, is statistically significant. (For example, how would I determine if 0.592 and 0.724 in the table are statistically different?)

• Welcome to Cross Validated! This sounds like a difference in differences. You might have luck doing some research about such an approach.
– Dave
Commented Jun 1, 2023 at 1:49
• I did some research on DiD before, and from what I found, it is meant to be used to interpret differences in trends before and after treatment on a time scale. Would that still be applicable in my case, where there is no time involved but rather its a "difference in differences" literally? Commented Jun 1, 2023 at 2:00

You could study this as a model with main effects for 'female/male' and for 'framing' along with different intercepts per question. Then, test whether adding an interaction for the two main effects makes the estimates significantly better.

A problem is however how to make this model. Is it gonna be a typical linear model or some more fancy function? And how are you gonna incorporate the interaction? In addition, are you gonna model the mean (is that a good parameter that describes the results) and how do you deal with the randomness and distribution of different participants?

• The interaction might be difficult to interpret.

The result might be more difficult depending on whether you regard the scale as linear or not. Is a change from 6.5 to 7 the same as a change from 6 to 6.5? You will need to investigate the scale before and use studies that designed the scale and questions to get an idea about the range and distribution of the response based on which you can create a sensible scale to compare results.

Example: say you test the performance of females and males on a running test and the effect of some training program. The men have on average times of 60 seconds and 50.4 seconds for with and without the training. The women have on average times of 70 seconds and 69.3 seconds for with and without training. In this case do you consider there to be a different effect with the men having 0.6 seconds improvement and the women having 0.7 seconds improvement, or do you consider there to be the same effect with both cases having 1% improvement?

• The interaction might be incorporated in different ways. Is the difference about a change in magnitude or about an absolute difference.

For example in your data, the men had effects on the three questions of M = 0.592, 0.393 and -0.647 while the women had effects of F = 0.724, 0.525 and -0.708. Do you consider a difference in the magnitude F/M ≈ 1.223, 1.336 and 1.094 where the difference between men and women is in the same >1 direction on all three questions? Or do you consider a difference in the absolute level F-M ≈ 0.132 0.132 -0.061 where the difference between men and women is not the same (positive/negative) direction on all three questions?

• The distribution of the likert scale is not a normal distribution. You could analyse it as a normal distribution, and for the mean this might be potentially without much problems (the estimate of the mean could be approximately normal distributed) . But possibly the change of the distribution in other aspects than the mean might be interesting. Possibly the median is more interesting or the percentage of people that differ with a specific amount. A similar change of the mean on the likert scale can be made if 50% of the people change by 2 points or if all change by 1 points, the two can have different interpretations.

A useful model to learn about is two-way ANOVA. But, as mentioned above, the situation might be more complicated and require a more complicated model (E.g a Rasch model).

• Thank you for your input - I looked into both models suggested and I found that they were not applicable to my data due to the fact that I must first compare the difference between the positive and negative frame. If I converted my likert scale to a binary scale (i.e. 1-3 turns into 0, and 4-6 turns into 1), and I analyze the data in terms of proportions and z tests, could that avoid the pitfalls as stated here and also be a viable method to analyze my data? Commented Jun 2, 2023 at 12:23
• In addition, if I would want to avoid losing data by converting to binary, could I still use the differences in differences approach as mentioned above, even though I am not analyzing on a time scale (use gender male/female instead on the time scale)? Commented Jun 2, 2023 at 12:27