0
$\begingroup$

I have trained two linear regressor models with the same response $Y$ and the same features $x_1$ and $x_2$, they are basically the same model, however only the training data differs: in model1 data was from country1 while the data from model2 was from country2

I run a F-Test Anova and I get a p-value of 0.07. I am not quite sure how to interpret this? isn't this a 7% significance level?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You can't properly interpret an F-test on two models in the way that you did it. An F-test comparing two models requires them to be nested, in the sense that the predictors in one model form a subset of those in the other, and that the underlying data are the same. See this answer.

The best way to see if there is a difference between the 2 countries (which seems to be your interest) is to combine all the data, include an indicator variable for the country, and run your regression model with the country variable added to your set of predictors. The coefficient for the country variable then represents the systematic difference between the 2 countries, with the other predictors taken into account.

If you perform the analysis that way you can do a proper F-test, comparing models with versus without the country variable but both models built on all of the data for both countries. That will document the "significance" of the difference between the countries.

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks for the feedback! If I do it the way you are suggesting, and get for example a p-value of 0.06, does the analysis stay the same? I mean, can I claim for example a 6% significance level? $\endgroup$
    – dm_reader
    Commented May 19, 2021 at 12:45
  • $\begingroup$ @dm_reader yes; if the appropriate assumptions hold, the p-value is the fraction of experiments in which you would get at least that big a difference in the observed result even if there wasn't a true difference. But you shouldn't be focused on p-values in that way. See this page for example. It's better to pre-specify a level of "significance" and stick with it. Or report the effect and confidence intervals for country. $\endgroup$
    – EdM
    Commented May 19, 2021 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.