2
$\begingroup$

I've built a logistic regression model to find predictors of having a performance issue in the workplace. Staff are either 1 - they have a performance issue, or 0 - they do not.

Approx 50 out of 3000 staff have a performance issue. I've excluded some groups from the dataset before i run the analysis as they have zero individuals in the performance issue group. The standard errors for their coefficients had come up as very large and seemed to indicate an issue. These groups were particular staff grade levels and divisions.

Should i also exclude groups which have only 1, 2 or 3 etc occurrences also? I wonder if they lead to less accurate estimates of the other coefficients. If i do then it affects what characteristics come up as statistically significant. E.g. in one model age is significant, in another disability, in another work location. The p-values move from being just above 0.05 to just below.

Is there a relatively easy way to justify which groups to include/exclude? Thanks Rob

$\endgroup$
4
  • 2
    $\begingroup$ I’m not at all convinced that creating this model is ethical or just. But estimating from small sample sizes is always dangerous. You need to either get more data or impose an inductive bias (some kind of regularization) to govern how much certain features are considered. $\endgroup$ Sep 13 at 14:24
  • 1
    $\begingroup$ Further, the just-above-or-below-0.05 threshold isn’t meaningful in a practical sense. Even if it used to be used for scientific gatekeeping, 0.05 is recognized as extremely arbitrary. Don’t stress too much about whether you are over that hurdle. $\endgroup$ Sep 13 at 14:26
  • $\begingroup$ Thanks @AryaMcCarthy is it a small sample though? I'm looking at all staff over a 2 and a half year period $\endgroup$
    – Rob Green
    Sep 14 at 11:39
  • $\begingroup$ @AryaMcCarthy regarding disregarding or putting less focus on the p-value. How else can one draw conclusions from the models? Just by looking at the odds ratios? These are sometimes large but not significant. That doesn't seem to happen often but does at least once in my models $\endgroup$
    – Rob Green
    Sep 14 at 11:42
4
$\begingroup$

Small group sizes combined with a small number of cases is a recipe for problems.

There isn't much you can do about the small proportion other than collect more data. As for the group sizes, yes it would be a good idea to combine them into another group, (or with another group if that makes sense). P-values will, of course, change when you change the data, but try not to be too concerned with p-values - they are little more than a function of sample size.

$\endgroup$
2
  • $\begingroup$ Thanks @Robert Long. Along the same lines as the discussion with Arya. I'm not sure if people think my sample is large or small but it is 2 and a half years of data so we've tried hard to get a large dataset so we can have some meaningful analysis. $\endgroup$
    – Rob Green
    Sep 14 at 11:45
  • $\begingroup$ And on the lower focus on p-values. So would you just look at the odds ratios? It could be helpful to do that, as e.g. when i include 23 dependent variables and exclude a couple of groups with 0 occurrences I get Disabled odds ratio=1.9, p-val=0.056. In other models with less dependent variables and with more, or less excluding groups, the ratio varies only from 1.9 to 2.2, whilst the p-val varies from 0.02 to 0.06 $\endgroup$
    – Rob Green
    Sep 14 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.