0
$\begingroup$

Good evening everyone.

Ill get to the point straight away. For my master thesis I've set out three different questionnaires to check which one provided the most valuable explanation of a dependent variable. Each respondent only received one type of questionnaire.

I now got three datasets with an Y-variable and dummies for subjects they mentioned (or not). My supervisor, allowed me to use multiple linear regression (even though in my limitations I should state this study should be replicated using another regression method).

I want to somehow conclude which model explains most variation in the Y-variable, but was also told I can not rely on adusted-r2 (there's a lot of other reasons you cannot rely on adjusted-r2, I know) because the differences between the models can be due to differences in the different samples.

Is there some kind of metric you guys advise me to read about to get more knowledge on how to compare the different models?

My goal was never to come up with statistical evidence, as I was just planning to compare in a qualitative matter how the responses on the different questionnaires differed, but the dean of my university wanted this to be added, because I now have the datasets anyway. It seems like a logical advise, but I simply do not know how to compare them.

Thanks in advance!

Improve this question
$\endgroup$
3
  • $\begingroup$ Why not simply stack the data up forming an ANOVA using a 3 level factor or variable specifying the partitions? Then test for between group differences in Y conditional on the subjects (or topics) mentioned as well as any other relevant control factors. Of course, this assumes that the design of topic exposure is appropriately balanced. $\endgroup$
    – user78229
    Commented Apr 3, 2017 at 20:20
  • $\begingroup$ Thanks for your reply. What do you mean with "topic exposure is appropriately balanced"? One format included a predefined set of question options, another format included an open question and the third format included both. After coding all open responses in the open and mixed format, dummy columns where made in each dataset for topics that were mentioned. $\endgroup$
    – Luti
    Commented Apr 5, 2017 at 7:45
  • $\begingroup$ Unbalanced ANOVAs require slightly different statistical assumptions and software that accomodates it. $\endgroup$
    – user78229
    Commented Apr 5, 2017 at 10:35

1 Answer 1

Reset to default
1
$\begingroup$

While this does depend on the theoretical nature of the model, you might want to try using an F-test, where the F statistic is the variation between sample means/variation between the samples.

This test is used to compare models in order to determine which one can best explain the variation in the dependent variable. You might consider incorporating this test into a one-way ANOVA: Understanding Analysis of Variance (ANOVA) and the F-test

That being said, you mention that you have used three different questionnaires. Be cautious if the number of observations for the three regression models are not equal, in which case the F-test could also be unreliable. e.g. if a model has 100 observations, the F-test could show a lower "fit" than one with 200, but if the number of observations for the first model had been increased, then it could in fact have the best fit.

You could also compute the power of a test for your three samples - i.e. identification of the minimum number of observations that would be needed for your results to be reliable. If your sample for the three models is shown to be large enough, then tests such as the F-test would be more reliable.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for your reply. I have 70 observations per format of the questionnaire. We did discuss ANOVA in class, but used it for different purposes. That might be why I haven't thought about ANOVA before. $\endgroup$
    – Luti
    Commented Apr 5, 2017 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.