# Regression with a 0-1 variable. Should be the same as running a t-test, but I get a different p-value [duplicate]

I have a dataset in R where I have volatility estimates (in my case, just standard deviation of minute returns on that day) for different days:

Date, Volatility, DayType


The variable DayType is 0 or 1 (no other options - no NA either), and it denotes whether a date meets some specific criteria or not - let's say that a day can either be normal or special. I want to run a simple test to see whether volatility on special day is different from a volatility on a normal day.

Since DayType can only have two values, the simple regression analysis should be equivalent to a t-test. But I get two different p-values, and I am wondering what I've misunderstood.

First, I can run a simple regression:

MyModel1=lm(Volatility ~ DayType, data=MyData)
summary(MyModel1)


This will give me a p-value for DayType. Alternatively, I can split the dataset into two datasets and run a t-test.

library(dplyr)
MyDataSpecial=MyData %>% filter(DayType==1)
MyDataNormal=MyData %>% filter(DayType==0)

t.test(MyDataSpecial$Volatility, MyDataNormal$Volatility, alternative="two.sided")


This will give a different p-value. Can you help me understand what is wrong here?

Here is also another example with mtcars dataset:

library(dplyr)
MyData=mtcars

#Regression
MyModel1=lm(mpg ~ vs, data=MyData)
summary(MyModel1)

#t-test
MyData1=MyData %>% filter(vs==1)
MyData2=MyData %>% filter(vs==0)

t.test(MyData1$mpg, MyData2$mpg, alternative="two.sided")


Thank you!

• please always share the data. use dput Commented Jul 26 at 17:59

The interesting thing is that, while the previous answers are undoubtedly technically correct (set var.equal=TRUE), they do not explain the reasons behind this.
There are 2 "versions" of the 2-sample t-test; the "original recipe", aka Student's t-test, which relies on a assumption of equal variances. And the Welch t-test, which does not need this assumption. The Student's t-test can either significantly exceed its nominal $$\alpha$$ error rate, when the variances are different, or be much more conservative than its nominal $$\alpha$$ level, depending on which sample has the largest variance (the one with the smallest or largest sample size). The Welch t-test remedies this. In fact, many authors/textbooks advise to always use the Welch t-test, even when the variances are similar and the samples balanced; for example here or here. This is because the Welch t-test works almost identically to the Student's t test when the conditions are met (equal variances, balanced samples), and much better when they are not. Because of this, many software default to using the Welch t-test. This is why t.test() defaults to "unequal variances", i.e. Welch. But lm() assumes equal variances.
Now, in your case, because you got different p-values (one assumes different enough that you had to ask a question), you have different variances, and possibly unequal sample sizes, you should use var.equal=FALSE (or leave it at default). That will give you the proper p-value. Now, if they are both significant (or both not significant) the difference does not matter much; but if they are quite different, believe t.test(), not lm().

• And OP should account for variance heterogeneity in the regression model. The gls function from the nlme package enables this. Commented Jul 29 at 7:30

If you set var.equal=TRUE in the t.test() call, they should be equivalent. Eg:

set.seed(1)
x <- sample(0:1, 100, rep=TRUE)
y <- rnorm(length(y)) + x
dtf <- data.frame(x, y)

summary(lm(y ~ x, data=dtf))
#
# Call:
# lm(formula = y ~ x, data = dtf)
#
# Residuals:
#      Min       1Q   Median       3Q      Max
# -5.53966 -1.73653 -0.25531  1.60254  7.42146
#
# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)
# (Intercept) -0.21661    0.39233 -0.5521  0.58214
# x            1.34139    0.56629  2.3687  0.01981 *
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#
# Residual standard error: 2.8292 on 98 degrees of freedom
# Multiple R-squared:  0.054154,    Adjusted R-squared:  0.044503
# F-statistic:  5.611 on 1 and 98 DF,  p-value: 0.01981
#

t.test(y ~ x, data=dtf, var.equal=TRUE)
#
#   Two Sample t-test
#
# data:  y by x
# t = -2.36875, df = 98, p-value = 0.01981
# alternative hypothesis: true difference in means between
# group 0 and group 1 is not equal to 0
# 95 percent confidence interval:
#  -2.4651653 -0.2176130
# sample estimates:
# mean in group 0 mean in group 1
#     -0.21660528      1.12478389
#


The methods in lm() and t.test() are different. Try the t.test with var.equal = TRUE:

t.test(MyData1$mpg, MyData2$mpg, alternative = "two.sided", var.equal = TRUE)

Two Sample t-test

data:  MyData1$mpg and MyData2$mpg
t = 4.8644, df = 30, p-value = 3.416e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
4.606732 11.274221
sample estimates:
mean of x mean of y
24.55714  16.61667

# extract p.value from the t.test

p_value_t_test <-
t.test(MyData1$mpg, MyData2$mpg,
alternative = "two.sided",
var.equal = TRUE
) |>
{\(x) x$p.value}() # Linear Regression MyModel1 = lm(mpg ~ vs, data = MyData) model_summary = summary(MyModel1) # Extract p-value for the DayType (vs) coefficient p_value_lm <- model_summary$coefficients["vs", "Pr(>|t|)"]


p.values are now the same

> p_value_t_test
[1] 3.415937e-05
> p_value_lm
[1] 3.415937e-05