# Comparing different linear regression models

Good evening everyone.

Ill get to the point straight away. For my master thesis I've set out three different questionnaires to check which one provided the most valuable explanation of a dependent variable. Each respondent only received one type of questionnaire.

I now got three datasets with an Y-variable and dummies for subjects they mentioned (or not). My supervisor, allowed me to use multiple linear regression (even though in my limitations I should state this study should be replicated using another regression method).

I want to somehow conclude which model explains most variation in the Y-variable, but was also told I can not rely on adusted-r2 (there's a lot of other reasons you cannot rely on adjusted-r2, I know) because the differences between the models can be due to differences in the different samples.

Is there some kind of metric you guys advise me to read about to get more knowledge on how to compare the different models?

My goal was never to come up with statistical evidence, as I was just planning to compare in a qualitative matter how the responses on the different questionnaires differed, but the dean of my university wanted this to be added, because I now have the datasets anyway. It seems like a logical advise, but I simply do not know how to compare them.

• Why not simply stack the data up forming an ANOVA using a 3 level factor or variable specifying the partitions? Then test for between group differences in Y conditional on the subjects (or topics) mentioned as well as any other relevant control factors. Of course, this assumes that the design of topic exposure is appropriately balanced. – Mike Hunter Apr 3 '17 at 20:20
• Thanks for your reply. What do you mean with "topic exposure is appropriately balanced"? One format included a predefined set of question options, another format included an open question and the third format included both. After coding all open responses in the open and mixed format, dummy columns where made in each dataset for topics that were mentioned. – Luti Apr 5 '17 at 7:45
• Unbalanced ANOVAs require slightly different statistical assumptions and software that accomodates it. – Mike Hunter Apr 5 '17 at 10:35

While this does depend on the theoretical nature of the model, you might want to try using an F-test, where the F statistic is the variation between sample means/variation between the samples.