15
$\begingroup$

A group of persons answers one question. The answer can be "yes" or "no". The researcher wants to know whether age is associated with the type of answer.

The association was assessed by doing a logistic regression where age is the explanatory variable and type of answer (yes, no) is the dependent variable. It was separately addressed by calculating the mean age of the groups that answered "yes" and "no", respectively, and by conducting a T test to compare means.

Both tests were performed following the advice of different persons, and neither of them is sure which is the right way to go. In view of the research question, which would be the better test?

For hypothesis testing the p values were not significant (regression) and significant (T test). The sample is less than 20 cases.

$\endgroup$
  • 2
    $\begingroup$ I'm not sure this gets are your real question. You already ran both of the analyses you're asking about. I'm guessing that what you really want to know is something about comparisons between or relations among those tests, for example which is better. Please edit your question to fix that. $\endgroup$ – John Jun 29 '15 at 3:22
  • $\begingroup$ Both tests were performed following the advice of different persons, and none of them is sure whether this is the right way to go. In view of the research questions (is age associated with the type of response?) which would be the better test, logistic regression of type of response on age or a T test comparing the mean age the persons who answered "yes" with the mean age of the persons who answered "no"? $\endgroup$ – Gwen Jun 29 '15 at 4:22
18
$\begingroup$

Both tests implicitly model the age-response relationship, but they do so in different ways. Which one to select depends on how you choose to model that relationship. Your choice ought to depend on an underlying theory, if there is one; on what kind of information you want to extract from the results; and on how the sample is selected. This answer discusses these three aspects in order.


I will describe the t-test and logistic regression using language that supposes you are studying a well-defined population of people and wish to make inferences from the sample to this population.

In order to support any kind of statistical inference we must assume the sample is random.

  • A t-test assumes the people in the sample responding "no" are a simple random sample of all no-respondents in the population and that the people in the sample responding "yes" are a simple random sample of all yes-respondents in the population.

    A t-test makes additional technical assumptions about the distributions of the ages within each of the two groups in the population. Various versions of the t-test exist to handle the likely possibilities.

  • Logistic regression assumes all people of any given age are a simple random sample of the people of that age in the population. The separate age groups may exhibit different rates of "yes" responses. These rates, when expressed as log odds (rather than as straight proportions), are assumed to be linearly related with age (or with some determined functions of age).

    Logistic regression is easily extended to accommodate non-linear relationships between age and response. Such an extension can be used to evaluate the plausibility of the initial linear assumption. It is practicable with large datasets, which afford enough detail to display non-linearities, but is unlikely to be of much use with small datasets. A common rule of thumb--that regression models should have ten times as many observations as parameters--suggests that substantially more than 20 observations are needed to detect nonlinearity (which needs a third parameter in addition to the intercept and slope of a linear function).

A t-test detects whether the average ages differ between no-and yes-respondents in the population. A logistic regression estimates how the response rate varies by age. As such it is more flexible and capable of supplying more detailed information than the t-test is. On the other hand, it tends to be less powerful than the t-test for the basic purpose of detecting a difference between the average ages in the groups.

It is possible for the pair of tests to exhibit all four combinations of significance and non-significance. Two of these are problematic:

  • The t-test is not significant but the logistic regression is. When the assumptions of both tests are plausible, such a result is practically impossible, because the t-test is not trying to detect such a specific relationship as posited by logistic regression. However, when that relationship is sufficiently nonlinear to cause the oldest and youngest subjects to share one opinion and the middle-aged subjects another, then the extension of logistic regression to nonlinear relationships can detect and quantify that situation, which no t-test could detect.

  • The t-test is significant but the logistic regression is not, as in the question. This often happens, especially when there is a group of younger respondents, a group of older respondents, and few people in between. This may create a great separation between the response rates of no- and yes-responders. It is readily detected by the t-test. However, logistic regression would either have relatively little detailed information about how the response rate actually changes with age or else it would have inconclusive information: the case of "complete separation" where all older people respond one way and all younger people another way--but in that case both tests would usually have very low p-values.

Note that the experimental design can invalidate some of the test assumptions. For instance, if you selected people according to their age in a stratified design, then the t-test's assumption (that each group reflects a simple random sample of ages) becomes questionable. This design would suggest relying on logistic regression. If instead you had two pools, one of no-responders and one of yes-responders, and selected randomly from those to ascertain their age, then the sampling assumptions of logistic regression are doubtful while those of the t-test will hold. That design would suggest using some form of a t-test.

(The second design might seem silly here, but in circumstances where "age" is replaced by some characteristic that is difficult, costly, or time-consuming to measure it can be appealing.)

$\endgroup$
  • $\begingroup$ Won't most of the non-linearity and separation concerns be alleviated by using a spline on the age variable? On that matter, apologies but I cannot see why the "pooled" design would invalidate the findings of the logistic regression. Sure, the assumption of random sample is gone but do we care given that we make this design choice? Are you alluding to selection bias? (The design you describe seems like a case-control study to me but I might be wrong...) (+1 obviously) $\endgroup$ – usεr11852 Jul 31 at 1:07
  • $\begingroup$ @usεr11852 Thank you for your thoughtful comments. I have rewritten some passages to clarify the points you bring up. Although splining the age can cope with nonlinearity in logistic regression, it can increase the possibility of complete separation. I'm not sure what you mean by "pooled design," but I would be suspicious of efforts to interpret the p-values of a logistic regression where a probability model cannot be justified (which is what random sampling enables us to do). $\endgroup$ – whuber Jul 31 at 12:05
  • $\begingroup$ Thank you for these. Yes, I fully appreciate the point you make about complete separation (Hauck-Donner effects), I did not consider them. OK, I see what you mean now on that about the two pools now. In that case we would have an agreed observational study concept (we observe/define the two pools) so we should off-the-bat look for way to control for it (propensity scores, etc.) $\endgroup$ – usεr11852 Jul 31 at 13:23
4
$\begingroup$

This doesn't really answer the question but may still be of some interest. The standard assumption of a two sample $t$-test is that the conditional normal distribution of $X$ given a binary variable $Y$, $$ X|Y=i \sim N(\mu_i,\sigma^2). $$ This together with the assumption that $Y \sim \operatorname{bernoulli}(p)$ marginally, implies that the conditional distribution of the binary variable $Y$ given $X=x$ is \begin{align} P(Y=1|X=x) &=\frac{f_{X|Y=1}(x)P(Y=1)}{\sum_{i=0}^1 f_{X|Y=i}(x)P(Y=i)} \\&=\frac{pe^{-\frac1{2\sigma^2}(x-\mu_1)^2}}{pe^{-\frac1{2\sigma^2}(x-\mu_1)^2} + (1-p)e^{-\frac1{2\sigma^2}(x-\mu_0)^2}} \\&=\frac1{1+\frac{1-p}pe^{-\frac1{2\sigma^2}(x-\mu_0)^2+\frac1{2\sigma^2}(x-\mu_1)^2}} \\&=\operatorname{logit}^{-1}(\beta_0 + \beta_1 x) \end{align} that is, a logistic regression model with intercept and slope \begin{align}\beta_0 &= \ln\frac p{1-p} -\frac1{2\sigma^2}(\mu_1^2-\mu_0^2) \\ \beta_1&=\frac1{\sigma^2}(\mu_1-\mu_0). \end{align}

So in this sense the two conditional models are compatible.

$\endgroup$
3
$\begingroup$

The better test is the the one that better addresses your question. Neither is just better on it's face. The differences here are equivalent to those found when regressing y on x and x on y and the reasons for different results are similar. The variance being assessed depends on which variable is being treated as the response variable in the model.

Your research question is terribly vague. Perhaps if you considered direction of causality you'd be able to come to a conclusion about which analysis you want to use. Is age causing people to respond "yes" or is responding "yes" causing people to get older? It's more likely the former, in which case the variance in the probability of a "yes" is what you wish to model and therefore the logistic regression is the best choice.

That said, you should examine assumptions of the tests. Those can be found online at wikipedia or in your text books on them. It may well be that you have good reasons not to perform the logistic regression and, when that happens you may need to ask a different question.

$\endgroup$
  • 1
    $\begingroup$ Do you mean "not to perform the logistic regression"? $\endgroup$ – mark999 Jun 29 '15 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.