8
$\begingroup$

I'm doing an analysis to see if having children is associated with treatment outcome. The children variable is categorical and takes values 0-7. When I dichotomise it into 'children' and 'no children', there is no effect.

enter image description here

But if I keep all the levels, there is a significant association with having 3 children.

enter image description here

I'm not sure how to interpret it. Should I keep all the categories? Is this a real finding or a fluke? I know dichotomising is generally not a good idea but I thought it's warranted with such a variables where Yes and No are so clearcut.

For context, this is not my main research question, I'm looking at many potential risk factors and it's just one of them so I don't necessarily need to analyse it too deeply.

$\endgroup$
20
  • 2
    $\begingroup$ You are completely right that dichotomising is frowned upon (here). But categorising an inherently numerical variable like "number (!) of children" is pretty much the same. It amounts to treating the relationship between $n$ and $n+1$ children as precisely the same as the relationship between any $n$ and $m$ children, and that does not make a whole lot of sense: the outcomes between 1 and 2 children should be more strongly related to each other than the outcomes between 1 and 7 children. $\endgroup$ Feb 2, 2023 at 10:29
  • 2
    $\begingroup$ A flexible way to allow for non-linearity, if supported by the data, is to use splines. I would also make (and show) a plot of the raw data. $\endgroup$
    – dipetkov
    Feb 2, 2023 at 10:29
  • 2
    $\begingroup$ Also @dipetkov According to my answer there's no indication of anything meaningful going on here. I'd advise against nonlinear modelling here because doing even more based on looking at the data first will involve even more "data dredging" and p-values become even less useful after such actions. (I'd say different things if there were a clear indication that something nonlinear is in fact going on.) $\endgroup$ Feb 2, 2023 at 10:38
  • 1
    $\begingroup$ @ChristianHennig: we can certainly agree to disagree here. I would just like to state for the record (and the OP) that my position is not quite as strict as yours: if I see the results of an analysis that I can argue is problematic without relying on the results, I would definitely recommend running a correct analysis over interpreting the results of an inappropriate analysis "by the book". YMMV. $\endgroup$ Feb 2, 2023 at 10:51
  • 1
    $\begingroup$ I would not start interpreting insignificant effects - they might be just random noise, especially since you presumably have very few data points with 6 or 7 children. (And because I still believe an analysis with categorical data makes no sense.) $\endgroup$ Feb 2, 2023 at 10:56

3 Answers 3

8
$\begingroup$

If you test every number of children separately, you're running seven tests. According to Bonferroni, for making sure that you have a probability of smaller than 5% to find a significant result if in fact nothing is going on, you've got to run every single test at level 0.05/7. This is smaller than the p-value for 3 children. From this point of view nothing significant is going on in either situation. You may well keep all categories, but this doesn't give you a meaningful significance. (Obviously you could find that p-value suspicious and collect much more data to see whether it becomes even more significant, but it may not be worth the hassle.)

Added later: There was some discussion about whether it is wrong to treat the number of children as categorical, actually being a number, and what to do instead.

Generally I agree with the view that it makes sense to use the numerical meaning of the data. However this will usually involve assumptions about the functional form of the relationship. An important principle is that all available subject-matter knowledge should be used before actually analysing the data, as making modelling decisions based on the data will often produce selection bias effects and runs counter to theory behind some standard methods such as the significance tests for the coefficients.

So that first step before seeing the data should be to ask what kind of functional relationship could make sense here. The easiest assumed relationship would be linearity, but linearity may not be realistic, particularly because there may be a much bigger difference between 0 and any number of children than between two nonzero numbers of children (of course I can't tell without knowing the exact background). Also the relationship may be non-monotonic.

An additional problem is the question what can be identified from the data at what precision. Standard errors for 5, 6, 7 (maybe even 4) children suggest that there is not much data for these and whatever can be said will be very imprecise. In fact one could know this already from the plain numbers of observations falling into these categories (taking those into account will hardly bias later analyses as they don't imply information about the regression relationship).

This means that there may not be enough information to nail down confidently any relationship more complex (i.e., requiring more parameters) than a linear one, and making assumptions that require only a low number of parameters to be estimated is certainly desirable. This also means that treating the larger numbers of children as autonomous categories doesn't give useful information either. The standard errors of about 19 suggest that there is hardly any power to detect anything; for sure shown results cannot even exclude the possibility that there is a monotonic or even linear relationship (or none at all), even though the latter may not be realistic.

If relationships don't follow simple functional forms, in fact aggregation of categories can be a sensible choice; depending on the background comparing zero with existing children may be justifiable, and in general even something like "zero", "one or two", "more than two" can work better than either using all numbers as separate categories (particularly with very thin numbers on some categories) or assuming linearity or a more complex functional form. As said before, optimally such decisions should be made before seeing the data (or at least the regression relationship); however looking at the data may reveal a striking specific deviation from an initial assumption/decision, in which case model selection bias is probably less bad than sticking to an inappropriate initial decision at any cost.

From the numbers I currently see, I suspect however that clear evidence for any influence of this variable whatsoever cannot be found in the given data, and I'd be very skeptical if any of the ideas above applied at this time point would lead to a just about significant p-value, which in that case may well be explained by model selection bias.

Note also, as discussed elsewhere, that looking at p-values often isn't a very good way of doing variable selection.

$\endgroup$
6
  • $\begingroup$ This is all completely correct, but to be honest, I can't bring myself to upvoting it, because while it answers the question, it does not answer the question the OP should be asking as long as they categorize a numerical predictor in a rather obviously ecologically invalid way... $\endgroup$ Feb 2, 2023 at 10:39
  • $\begingroup$ @StephanKolassa so just to make sure I understand you correctly. Number of children is a kind of variable that should've never been treated as categorical. But is it warranted, upon looking at a numeric one and getting no significant results, to dichotomise it? Only for this case, when it seems ecologically plausible that children/no children are very different from each other. $\endgroup$
    – olke
    Feb 2, 2023 at 11:06
  • $\begingroup$ @StephanKolassa From the comments (which may be too many at this point), it seems that this is an XY question as the OP is doing univariate feature selection to choose predictors to include in the multivariate model they want to develop. $\endgroup$
    – dipetkov
    Feb 2, 2023 at 11:24
  • 1
    $\begingroup$ @olke: it depends. If you want to do inferential significance testing, then any kind of model selection after seeing the data invalidates the theory on which p values are predicated - you do need to specify your model before doing any fitting. (Hence the discussion between Christian and me.) If you are just doing exploratory analysis, or building predictive model, pretty much anything goes. There is a strong temptation to overfit, so you really need to include ecological understanding, per my comments - and then you need to validate on new data. $\endgroup$ Feb 2, 2023 at 12:22
  • 1
    $\begingroup$ @StephanKolassa I have added to my answer; hope that this is to your liking. ;-) $\endgroup$ Feb 2, 2023 at 14:52
7
$\begingroup$

This is a great example of why it is so difficult to 'hypothesize after the results are known' and why we should make analysis plans before we start, even for observational data. Every analysis we try now will just reflect something we've already seen, and we could probably find an analysis to give whatever answer we want.

IMO we can't say whether or not having children affects your outcome based on this data. Certainly your dataset is not inconsistent with 'no effect' so this is probably how you should explain it as that's how we normally think about statistical significance. Particularly if this also comes from a set of secondary outcomes that you are exploring. We also can't rule out a small effect.

From a programming point of view for an 'honest' quantification of your results you might try using emmeans to get the base vs all other group contrasts, and use whatever p-value and confidence interval correction is recommended for this kind of comparison. Or just report your initial model of 'number of kids' as a numeric, if this was supported by your theory and is what you tried initially.

Whether or not it makes sense to analyse the number of children in categories like this is up to you. Social factors might make 3 kids qualitatively different from 2, Its down to your population and your theory.

Statistical arguments aside you could also consider the Bradford Hill criteria to check if you believe a result is real or meaningful.

Finally, if you don't need have an opinion about what this data means then you don't need to overthink it. Just show your readers your model and your results and let them decide. A graph like this might be instructive:

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ Nice graph! ... $\endgroup$
    – Ben Bolker
    Feb 2, 2023 at 23:20
4
$\begingroup$

Why not run an ANOVA and see whether adding all the levels improves overall fit? If it does then you can worry about which level is driving it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.