2
$\begingroup$

I am struggling with one question. Is goodness of fit of a model necessary when your purpose is to test hypothesis regarding a coefficient? To be specific, I am regressing formal credit access on gender (variable of interest) and other predictor variables. Logistic regression with svy prefix in Stata is used. It is observed that while full model is a good fit as per Archer-Lemeshow test, logistic models on sub-population are mostly not. Is it necessary that all the sub-population models should be a good fit?

$\endgroup$
1
  • 2
    $\begingroup$ Please clarify what you mean by "sub-population models". $\endgroup$
    – utobi
    Commented Oct 25, 2023 at 6:46

2 Answers 2

2
$\begingroup$

Hypothesis Testing Assumes a Model & Conditions Need Met

Hypothesis testing inherently involves a model to reason about the likelihood of rejecting the null hypothesis.

In a way, hypothesis testing is almost always able to be seen as a goodness of fit test where a "do not reject null hypothesis" is a "good enough fit" to the hypothesis tests assumed model and focused statistic of interest.

However, your question hinges on whether the hypothesis test's assumed model is the correct model you care about for your data.

If sub populations are not resulting in a good fit then this demonstrates an issue in certainty of the model at a local level for your data.

It would be more desirable and a clearer cut answer if the sub populations were a good fit. Since they are not that lack of fit indicates higher uncertainty in localized regions more than the global perspective (full model as you put it).

I would suspect that if the conditions to apply the hypothesis test of choice are not satisfied (assuming the sub population goodness of fit is such a condition) then you cannot appropriately conduct that hypothesis test on the data. It'd be an improper application of the test.

An example: if a hypothesis test about significantly different mean of data to null hypothesis model's mean, then there has to exist a first moment (expectation is the mean) in the underlying model of the data, which requires proper data design or knowledge of the family of underlying distribution of the data.

Specifics in the Archer-Lemeshow test

With respect to the Archer-Lemeshow test, the paper refers to it as a "Goodness-of-fit test, called the F -adjusted mean residual test" [1, pg 101]. Though Archer was a grad student of Lemeshow so I am uncertain if there is another test you are referring to. The paper is linked and open access.

After fitting the logistic regression model, the test in this paper breaks the residuals into 10 bins (deciles). They discuss using the F-corrected Wald statsistic over those bins with g - 1 degrees of freedom, where here g=10.

From my understanding, the lack of fit within can be an issue if severe enough. However, further digging into the conditions necessary to apply this test on your data needs to be done.


References

  1. K. Archer, S. Lemeshow. "Goodness-of-fit test for a logistic regression model fitted using survey sample data". 2006. https://www.stata-journal.com/article.html?article=st0099
$\endgroup$
1
1
$\begingroup$

I will give a general answer that won't tell you what to do in your specific situation.

Generally, (a) if a statistical method comes with certain model assumptions, this means that the method has certain desirable performance features if in fact the model assumptions are fulfilled. It does not necessarily mean that the method is bad if model assumptions are not fulfilled. This may or may not be the case.

(b) Statistical models are idealisations. They are never perfectly fulfilled in reality, so it is wrong to say that model assumptions must be fulfilled for the method to be applied, because if this were so, we could never apply any model-based statistical method.

(c) What is really important is that the model is not violated in ways that mislead the analysis, i.e., that lead to results that are interpreted in ways deviating from what the situation actually is (as indicated by the data). Unfortunately, it is very subtle and hard to know whether this is the case or not. It will depend on the specific situation and on the use and intended interpretation of results.

(d) Goodness of fit as measured by specific goodness of fit tests and/or metrics is often somewhat related to this, but it isn't the same, meaning that you can have (i) issues that invalidate your interpretation but that goodness of fit tests can't find (such as nonrepresentative samples), but also (ii) results that are fine even if a goodness of fit test finds problems (which is for example the case with many deviations from the normal distribution and large enough samples so that the Central Limit Theorem applies). What is required is a more thorough understanding of the method and the situation, often helped by good data visualisation, because this can help to distinguish bad from not so bad problems. Given what is said in (c), statisticians often recommend non-statisticians to look at goodness of fit, because although this is far from perfect, it may at least prevent some things from going wrong (but also use all background knowledge on data collection etc. that you have).

(e) As mentioned in the answer by @prijatelj, in principle one could say that a test never requires model assumptions, as its aim is to find out whether data are compatible with the null hypothesis model (in the sense defined by the test statistic), but of course the null hypothesis doesn't have to be fulfilled when testing (otherwise the test would be pointless), and it can just be rejected in case data deviate from it. However this can still lead to misinterpretation if deviations from model assumptions would lead the test to a misleading conclusion, like rejecting a technically wrong null hypothesis even if reality acts in ways that we would think of as being "close" to the null hypothesis, or non-rejection if a large majority of data is clearly not in line with the null hypothesis in relevant ways and only an outlier brings the test statistic overall to non-rejection.

I can't comment on your specific problem; if I were in your position, I'd look at the data in order to find out what exactly leads to your "violation in sub-populations", and then think about what impact this can be expected to have on the results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.