What does it mean to perform regression using a specific distribution?

Question

When we specify the family= argument inside glm() in R, how is the distribution being used to regress the dependent variable on the independent variable? In simple linear regression(lm() in R), we simply calculate the mean of Y for each X and that becomes the predicted value. How different is this when we mention a family?

I do know that each distribution has a few parameters that describes it (mean, shape, scale, etc). So how are they used to get predictions? But I would like to learn what it means to fit a distribution to a data.

---

To clarify, let's say I trained my model on a dataset of 1lac obs with 2 independent variables and the coefficients are 1,2. That is, beta1 = 1 and beta2 = 2

          Y    x1   x2
1st obs.  2    .5    1
2nd obs.  1    .25  .25

So if I choose the Poisson distribution, then:

1st obs. mean(mu) = exp(.5*1+1*2) = 12.18

Similarly, we can get a mean for the second obs and so on. Now how is this mean related to what you are going to predict? I am not able to connect this.

Another major concern is also how the shape / scale (i.e., sigma, nu, tau for gamlss) is being modeled. However they are secondary, and for now I want to focus on GLMs.

Answer 1

Regression models aim to estimate a parameter (typically a mean) for a response variable conditional on a (set of) regressor variable(s). To do that, we generally need to specify the nature of the response variable, i.e., its distribution. Standard linear regression assumes the response is conditionally normal and coefficients for the covariates can be found using a simple, closed-form solution, which is the ordinary least squares (OLS) estimator. For many types of distributions, however, OLS will not yield an appropriate model. More generally, we use maximum likelihood estimation to fit models. (It turns out that when the response distribution is normal, OLS corresponds to the maximum likelihood estimate.) The generalized linear model is a class of regression models that can be fit by MLE. The generalized linear model requires that the distribution be a member of the exponential family, but having appropriately stipulated such a distribution, MLE can be used to fit the model.

In other words, we need to specify a response distribution to fit a model using MLE. That is because the likelihood is determined by the distribution. Consider a logistic regression model assuming the response is Bernoulli. Then there is a probability, $p_i$, that a given observation will be $1$ (and $1-p_i$ that it will be $0$). A given model (i.e., a set of candidate betas) will specify a set of $p_i$s / $1-p_i$s for your realized $y_i$ values. The log likelihood is the sum of the logs of those $p_i$s / $1-p_i$s. This can be understood as a measure of the quality of the model, and it can be used to find the best model for a given dataset. That wouldn't work if the response isn't Bernoulli, though. For example, if the response were normal the likelihood for a given datum would instead be:
$$ \ell(y_i) = \frac{1}{\sqrt{2\pi\sigma^2}}\ \exp\!\bigg(-\frac{(y_i-\mu_i)^2}{2\sigma^2}\bigg) $$ Computing that for every datum, taking logs, and adding them up, would yield the log likelihood for a model of your (conditionally normal) data. These examples show that specifying the distribution is needed to create one of the building blocks required to fit the model. That is what is meant by using a specific distribution to fit a regression.

The likelihood is also used in inference, as well as estimation. This can show up in many ways, but perhaps the easiest to see is in a likelihood ratio test. The difference between the log of the likelihood of a full model and the log likelihood of a nested, restricted model (where one or more parameters are set to fixed values, e.g., $0$—the variables are dropped) can be used to perform inference on those restricted parameters. That is, you are testing if those variables are significant. But to do that, you need the likelihoods, for which you need to specify the distribution. So that is an example of what is meant by using a specific distribution to perform inference with a regression model.

The distribution can be used in lots of other ways, too. A quick example might be that various 'pseudo-$R^2$s' are based ultimately on the distribution. Etc.

Answer 2

These concepts fall under the subject of Generalized Linear Models. Generalized linear models contain two components

• A random component. The observed data is distributed according to some distribution. The distribution that generates a single data point comes from a family of distributions.
• The systematic component. The specific distribution from the family which is assumed to generate a single data point is chosen by some function of the independent variables.

You can summarize many of the common generalized linear models, and see the pattern that emerges:

Linear Model

$$ y \mid x \sim \text{Normal} \left( \mu = \beta_0 + \beta_1 x_1 + \cdots + \beta_n x_n \right) $$

Logistic Model

$$ y \mid x \sim \text{Bernoulli} \left( p = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \cdots + \beta_n x_n)}} \right) $$

Poisson Model

$$ y \mid x \sim \text{Poisson} \left( \lambda = e^{\beta_0 + \beta_1 x_1 + \cdots + \beta_n x_n} \right) $$

Each model is fit by the method of maximum likelihood. In practice, we maximize the logarithm of the likelihood of the observed data using Newton's method.