5
$\begingroup$

Does anyone know of any $\chi^2$ tests to compare the fit of logistic models which factor out the sample size? I'm dealing with a very large sample and I fear the significant $\chi^2$ test I get when adding a single variable to the model is simply the result of the sample size (>200,000 cases). I'm doing what is known as differential item functioning analysis with logistic regression. Basically it's as if I'm checking whether giving the right answer to a question (dependent variable) depends on your ethnicity when controlling for the total exam score.

Model 1 Q1~TotalexamScore

Model 2 Q1~TotalexamScore+ Group

I'm basically using a chi-squared test to compare model1 to model2. The coefficient significance is not that important but $\chi^2$ and sometimes $R^2$ are generally recommended to check differential item functioning. My problem is that my sample is very large. In theory (for the question I'm considering) there should be no real difference across groups, so I suspect it's simply the sensitivity of the $\chi^2$ to sample size.

I'd rather use the whole dataset instead of taking (small) random samples as it is highly skewed. I've seen things like Phi and Cramer's V for crosstabs but I'm not sure whether they have been used before on logistic regression, if there are better ones and if there are any packages (I generally use Spss, Mplus, Stata, R).

Improve this question
$\endgroup$
1
  • $\begingroup$ I've added a link describing DIF so that people get a better idea of what you are really testing. In your case, as you do not consider any interaction between trait level and group, an LRT of M1 vs. M2 will assess uniform DIF. And +1, this is a good question. $\endgroup$
    – chl
    Commented Aug 1, 2012 at 17:45

3 Answers 3

Reset to default
3
$\begingroup$

One hueristic way you can take account of sample size is to make a random group variable which has the same marginal propensity as your "Group" variable. Then check the chi square statistic for this random group. If it's greater than the chi square for your variable then you have a fair case to dismiss the effect as noise. A more robust version would be to create many noise variables and see if any of their chi square statistics are greater than the chi square for your variable.

Another thing you should do is examine the beta coefficients or "effect sizes" for the "Group" variable. Do they make intuitive sense? For example can you explain why a coefficient should be positive or negative? Can you explain why the magnitude of the coefficient should be bigger or smaller than the other coefficients?

As far as more formal tests go I would recommend BIC as it tends to be conservative. If BIC favours the larger model, then just about any other test will. This usually means "low power" when the sample size is small, but your sample size is large. You can show that using BIC is approximately the same thing as setting the p-value for significance in a likelihood ratio chi square test equal to $Pr(\chi_q^2>q\log[N])$ where $q$ is the number of additional parameters in the larger model.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you. I particularly like the interpretation of the chi-square in terms of BIC. $\endgroup$
    – Marcelo
    Commented Aug 5, 2012 at 8:52
1
$\begingroup$

You should realize that the statistical power is not going to be determined by the number of cases, but rather by the number of events (or the smaller of the 0/1 categories). There are a variety of goodness of fit statistics that have been proposed. I generally avoid them, since I am more interested in looking at effect magnitudes rather than in global fit. I generally set my threshold for considering differences in the model X^2 much higher when working on large data. My data is on the order of 10 times as large as yours but with only 1% of cases being events. In that context I find that models with X^2 differences of 10 or less with one degree of freedom generally have very little meaningful difference when examined for effect differences. And I also require a X^2 difference of 30 when I know if have done extensive multiple testing.

Along the same lines, when doing model comparison with large datasets, you should be using penalization that takes into account the number of model comparisons. I sometimes figure my degrees of freedom are in the range of 50-100 considering the number of different models I have examined. You will have the opportunity to examine non-linearity and the potential for interactions, and you should definitely avail yourself of that possibility. And you should consider the search for non-linearity in your estimates of the number of degrees of freedom needed. There is general agreement in the R-help community that Frank Harrell's text "Regression Modeling Strategies" has excellent discussions on these points, and I am basically repeating things I learned from Frank. (I do hope I haven't misrepresented them.) The R 'rms' and 'Hmisc' packages implement the advice and methods Harrell recommends.

This is the list of potential "model metrics" that Harrell's lrm function will offer:

model likelihood ratio chi-square, d.f., 
P-value, 
c index (area under ROC curve), 
Somers' D_{xy}, 
Goodman-Kruskal gamma, 
Kendall's tau-a rank correlations between predicted probabilities and observed response, 
Nagelkerke R^2 index, 
Brier score computed with respect to Y > its lowest level, 
g-index, 
gr (the g-index on the odds ratio scale), 
gp (the g-index on the probability scale using the same cutoff used for the Brier score).

As I said I generally am considering comparing penalized LR statistics, but the Nagelkerke pseudo-R^2 should be somewhat free of sample-size inflation.

Improve this answer
$\endgroup$
1
  • $\begingroup$ What the OP didn't say, unfortunately, is that the kind of model he is interested in is generally of the form $f(\text{response}\mid\text{trait level},\text{group})=\beta_0+\beta_1\cdot\text{trait level}+\beta_2\cdot\text{group}$, meaning that response is conditioned on individual trait level (usually, assumed to be a latent variable). This amounts to consider a mixed-effect logistic regression, as our (psychometric) measurement model. An alternative approach is to use raw scores stratified by levels of total score, which amounts to rely on some variant of the Mantel-Haenszel test. $\endgroup$
    – chl
    Commented Aug 5, 2012 at 8:56
1
$\begingroup$

Any test of statistical significance will be sensitive to sample size, that's one of the problems with statistical significance measures.

There are alternatives: 1) Point out just what you did above: Present the chi-square statistic and note its significance but then discuss the size of the effect of adding group.

2) Use a measure that accounts for model complexity (e.g. AIC, AICC, BIC). These may show that the simpler model is better, despite the huge sample size. However, AFAIK, the difference in AIC doesn't yield a p-value.

and probably other things as well.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Because the penalty in AIC does not vary with the sample size I would have thought it would be subject to the same problem as ordinary significance testing - the saturated model will eventually be favoured if the sample is made big enough. $\endgroup$
    – probabilityislogic
    Commented Aug 5, 2012 at 4:05
  • $\begingroup$ I am not sure. Does loglikelihood change systematically with sample size? $\endgroup$
    – Peter Flom
    Commented Aug 5, 2012 at 12:17
  • $\begingroup$ my thinking was that AIC corresponds in regression to setting your significance level to 0.15 - regardless of the sample size. And Im pretty sure all fixed significance levels are eventually passed for a large enough sample. $\endgroup$
    – probabilityislogic
    Commented Aug 5, 2012 at 23:15
  • $\begingroup$ How could AIC correspond to a particular significance level? I don't understand $\endgroup$
    – Peter Flom
    Commented Aug 6, 2012 at 10:06
  • $\begingroup$ if you're comparing two nested hypothesis which are one parameter different, then you come to the same conclusion as someone who use a LR test with p value of $0.15$ and this does not depend on the sample size. They both accept the more complex model if the likelihood ratio is above the same threshold independent of the sample size. $\endgroup$
    – probabilityislogic
    Commented Aug 6, 2012 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.