14
$\begingroup$

A group of persons answers one question. The answer can be "yes" or "no". The researcher wants to know whether age is associated with the type of answer.

The association was assessed by doing a logistic regression where age is the explanatory variable and type of answer (yes, no) is the dependent variable. It was separately addressed by calculating the mean age of the groups that answered "yes" and "no", respectively, and by conducting a T test to compare means.

Both tests were performed following the advice of different persons, and neither of them is sure which is the right way to go. In view of the research question, which would be the better test?

For hypothesis testing the p values were not significant (regression) and significant (T test). The sample is less than 20 cases.

$\endgroup$
  • 2
    $\begingroup$ I'm not sure this gets are your real question. You already ran both of the analyses you're asking about. I'm guessing that what you really want to know is something about comparisons between or relations among those tests, for example which is better. Please edit your question to fix that. $\endgroup$ – John Jun 29 '15 at 3:22
  • $\begingroup$ Both tests were performed following the advice of different persons, and none of them is sure whether this is the right way to go. In view of the research questions (is age associated with the type of response?) which would be the better test, logistic regression of type of response on age or a T test comparing the mean age the persons who answered "yes" with the mean age of the persons who answered "no"? $\endgroup$ – Gwen Jun 29 '15 at 4:22
17
$\begingroup$

Both tests implicitly model the age-response relationship, but they do so in different ways. Which one to select depends on how you choose to model that relationship. Your choice ought to depend on an underlying theory, if there is one; on what kind of information you want to extract from the results; and on how the sample is selected. This answer discusses these three aspects in order.


I will describe the t-test and logistic regression using language that supposes you are studying a well-defined population of people and wish to make inferences from the sample to this population.

In order to support any kind of statistical inference we must assume the sample is random.

  • A t-test assumes the people in the sample responding "no" are a simple random sample of all no-respondents in the population and that the people in the sample responding "yes" are a simple random sample of all yes-respondents in the population.

    A t-test makes additional technical assumptions about the distributions of the ages within each of the two groups in the population. Various versions of the t-test exist to handle the likely possibilities.

  • Logistic regression assumes all people are a simple random sample of the population. When this population is broken down by age, the separate age groups exhibit different rates of "yes" responses. These rates, when expressed as log odds (rather than as straight proportions), are assumed to be linearly related with age.

    Logistic regression is easily extended to accommodate non-linear relationships between age and response. Such an extension can be used to evaluate the plausibility of the initial linear assumption. It is practicable with large datasets, which afford enough detail to display non-linearities, but is unlikely to be of much use with small datasets. A common rule of thumb--that regression models should have ten times as many observations as parameters--suggests that more than 20 observations are needed to detect nonlinearity (which needs a third parameter in addition to the intercept and slope of a linear function).

A t-test identifies whether the average ages differ between no-and yes-respondents in the population. A logistic regression estimates how the response rate varies by age. As such it is more flexible and capable of supplying more detailed information than the t-test is. On the other hand, it tends to be less powerful than the t-test for the basic purpose of detecting a difference between the average ages in the groups.

It is possible for the pair of tests to exhibit all four combinations of significance and non-significance. Two of these are problematic:

  • The t-test is not significant but the logistic regression is. When the assumptions of both tests are plausible, such a result is practically impossible, because the t-test is not trying to detect such a specific relationship as posited by logistic regression. However, when that relationship is sufficiently nonlinear to cause the oldest and youngest subjects to share one opinion and the middle-aged subjects another, then the extension of logistic regression to nonlinear relationships can detect and quantify that situation, which no t-test could detect.

  • The t-test is significant but the logistic regression is not, as in the question. This often happens, especially when there is a group of younger respondents, a group of older respondents, and few people in between. This creates a great separation between the groups of no- and yes-responders. It is readily detected by the t-test. However, logistic regression would either have relatively little detailed information about how the response rate actually changes with age or else it would have inconclusive information: the case of "complete separation" where all older people respond one way and all younger people another way--but in that case both tests would usually have very low p-values.

Note that the experimental design can invalidate some of the test assumptions. For instance, if you selected people according to their age in a stratified design, then the t-test's assumption (that each group reflects a simple random sample of ages) becomes questionable. This design would suggest relying on logistic regression. If instead you had two pools, one of no-responders and one of yes-responders, and selected randomly from those to ascertain their age, then the assumptions of logistic regression are doubtful while those of the t-test will hold. That design would suggest using some form of a t-test.

(The second design might seem silly here, but in circumstances where "age" is replaced by some characteristic that is difficult, costly, or time-consuming to measure it can be appealing.)

$\endgroup$
4
$\begingroup$

This doesn't really answer the question but may still be of some interest. The standard assumption of a two sample $t$-test is that the conditional normal distribution of $X$ given a binary variable $Y$, $$ X|Y=i \sim N(\mu_i,\sigma^2). $$ This together with the assumption that $Y \sim \operatorname{bernoulli}(p)$ marginally, implies that the conditional distribution of the binary variable $Y$ given $X=x$ is \begin{align} P(Y=1|X=x) &=\frac{f_{X|Y=1}(x)P(Y=1)}{\sum_{i=0}^1 f_{X|Y=i}(x)P(Y=i)} \\&=\frac{pe^{-\frac1{2\sigma^2}(x-\mu_1)^2}}{pe^{-\frac1{2\sigma^2}(x-\mu_1)^2} + (1-p)e^{-\frac1{2\sigma^2}(x-\mu_0)^2}} \\&=\frac1{1+\frac{1-p}pe^{-\frac1{2\sigma^2}(x-\mu_0)^2+\frac1{2\sigma^2}(x-\mu_1)^2}} \\&=\operatorname{logit}^{-1}(\beta_0 + \beta_1 x) \end{align} that is, a logistic regression model with intercept and slope \begin{align}\beta_0 &= \ln\frac p{1-p} -\frac1{2\sigma^2}(\mu_1^2-\mu_0^2) \\ \beta_1&=\frac1{\sigma^2}(\mu_1-\mu_0). \end{align}

So in this sense the two conditional models are compatible.

$\endgroup$
3
$\begingroup$

The better test is the the one that better addresses your question. Neither is just better on it's face. The differences here are equivalent to those found when regressing y on x and x on y and the reasons for different results are similar. The variance being assessed depends on which variable is being treated as the response variable in the model.

Your research question is terribly vague. Perhaps if you considered direction of causality you'd be able to come to a conclusion about which analysis you want to use. Is age causing people to respond "yes" or is responding "yes" causing people to get older? It's more likely the former, in which case the variance in the probability of a "yes" is what you wish to model and therefore the logistic regression is the best choice.

That said, you should examine assumptions of the tests. Those can be found online at wikipedia or in your text books on them. It may well be that you have good reasons not to perform the logistic regression and, when that happens you may need to ask a different question.

$\endgroup$
  • 1
    $\begingroup$ Do you mean "not to perform the logistic regression"? $\endgroup$ – markseeto Jun 29 '15 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.