Assuming that Y is continuous (rather than binary) and the first predictor X1 is binary, you could perform either of the following:
1. A t-test which allows you to compare the mean value of Y when X1 = 1 against the mean value of Y when X1 = 0;
2. A simple linear regression model regressing Y on X1.
I am guessing most people use a t-test because it has the flexibility of allowing the variability of the Y values when X1 = 0 to be different from the variability of the Y values when X1 = 1. But if that needs to be allowed for, it can easily be accommodated in the context of a simple linear regression model too by allowing the error variance to depend on the values of X1.
Assuming that Y is continuous and the second predictor X2 is categorical with k categories, where k > 2, you could perform either of the following:
i. A one-way analysis of variance, which allows you to compare the mean value of Y when X1 = 1 across the categories of X2;
ii. A simple linear regression model regressing Y on X2. (When fitted to the data, this model will actually include k - 1 dummy variables as predictors, all of which help encode the effect of X2 on Y.)
By default, the one-way anova in i. and the simple regression model in ii. assume that the variability of the Y values is roughly the same across the categories of the X2 variable. If that is not the case, each of these methods can be modified accordingly.