# How do models, parameters, specifications, restrictions and assumptions relate?

So this has been something I've been struggling with for a long time:

The specification of a particular model is subjective. However, there seems to be objective ('true') values of the parameters we wish to find, given that specification. What are these 'true' parameters?

And given this, how can we make sense of 'model mis-specification', 'bias', and other such terms?

Finally, how does model specification differ from restriction of parameters or the assumptions taken for estimation routines?

I've been trying to understand this for a few months and have read quite extensively, I think I am quite close to resolving the ambiguities, but I thought it would be good to run it by some experts and see if it's completely in the wrong direction.

Please be nice - I am entirely self-taught :)

I'm going to assume a Frequentist perspective throughout this.

The 'True' Distribution

• Suppose there are $d$ variables of interest. Consider a sample space, $\mathscr S \subseteq R^d$.Suppose $\Bbb P(\mathscr S)$ is the set of all possible probability distributions on $\mathscr S$.
• There is a point $p^* \in \Bbb P$ which is the 'true' population distribution. This distribution is likely to be quite complex.
• $p^*$ can be represented formally as a set of moments (and conditional moments; i.e. the $E(Y|X=x)$, $Var(Y|X=x)$, and so forth for infinite moments) that fully specify the probability surface.
• The aim of the statistical inference is to approximate $p^*$ with some other (likely simpler) $p ∈ Q(S)$. The distance between $p$ and $p^*$ can be thought of (in principle) the outcome of some f-divergence function. $p$ is intended as a trade-off between parsimony and accuracy.
• In principle $p$ can be perfectly identical to $p^*$ (in which case it is the ‘true’ model), which maximizes accuracy in sacrifice of parsimony

Models

From a standard definition of models:

A statistical model is a set of probability distributions on the sample space $\mathscr S$.

When it comes to parametric models, we simplify the space of possible probability distributions by proposing some structural constraints between the $d$ variables of interest. [For example, in a simple linear model, the conditional expected values of $Y$ are some deterministic, linear function of some (if not all) of the covariates.] This gives rise to a family of distributions.

Formally, by specifying a parametric model $F$, we are proposing some subset $F(\mathscr S)\subset\Bbb P(\mathscr S)$, each member of which (a distribution, $p$) is indexed by a unique finite-dimensional parameter $θ_F ∈ Θ_F$, where $Θ_F$ is the feasible region of parameters. That is, a parametric model $F$ is a function that maps each $θ_F ∈ Θ_F$ uniquely onto some distribution $p ∈ F(\mathscr S) \subset \Bbb P(\mathscr S)$.

Note that $F(\mathscr S)$ does not necessarily (and is unlikely to) contain $p^*$. $F(\mathscr S)$ is filled with distributions which we think approximate $p^*$ reasonably well.

Parameters

For any parametric model $F(\mathscr S)$, there is some $θ'_F ∈ Θ$ and therefore corresponding $p’$, which by some f-divergence function, most closely resembles $p^*$. These are the ‘true values’ of the parameters that we are attempting to infer from a given sample.

Importantly, the value $θ’_F$ is conditional on the parameterisation $F$. If the statistician specified a different class of models (some $G ≠ F$), then the corresponding $θ’_G$ would be different. (If it was the same it would be entirely by coincidence).

The implication for this is that if the statistician specifies a different model, the parameters he or she is attempting to find, $θ’_F$, are different. There is no ‘objective’ $θ^*$, it is a derivative of the parameterization function.

Restrictions

Given a parameterization $F$, a restriction of the model forces some of the parameters $\theta$ indexing $F$ to take on certain values (i.e. = 0).

The result is a different model $F'$, where $F'(\mathscr S) \subset F(\mathscr S) \subset \Bbb P(\mathscr S)$. Following the logic of the preceding section, since $F'$ is a different model to $F$, the parameter space $\Theta_{F'}$ indexing $F'$ does not necessarily equal $\Theta_F$, and as a result the 'true value' parameters we are trying to find $\theta'_{F'}$ does not necessarily equal $\theta'_F$.

Having said that, for a broad class of models, and a broad class of restrictions, we are only interested in a subset of the parameters $\theta$, and these may remain invariant to the restrictions. For example, with linear regressions we are usually only interested in $\beta$, which (AFAIK) do not change if one were to go from a heteroskedastic regression model to the homoskedastic restriction (which forces the $Var(Y|X=x) = k; \forall x$). The 'true' values of the other parameters, such as those describing the errors (i.e. $\sigma$), presumably, would have to change under $F'$ instead of $F$.

Assumptions

When it comes to estimators, the first step is to specify a model, then impose certain restrictions (as implied above). There are now a set of free parameters $\theta'_{F'}$ to estimate.

However, certain estimation procedures require certain assumptions to hold. By this, I mean that we assume (not impose) that the true value of some of the parameters (i.e. some elements of $\theta'_{F'}$), take on certain values. If they do not take on these values, then this will affect the fidelity of the estimation procedure. If they do, we can be confident in the fidelity of the estimators.

A natural example is OLS estimators. If you are working with a model that allows for heteroskedastic error, then the standard deviation of the OLS estimators ($\hat \beta_F$) will be biased. That is, we allow for a heteroskedasticity (so there is a parameter allowing for this), but assume that there is none. If this assumption holds, the estimator will function as desired.

Illustrations

There are a plethora of different models for two variables of interest, $(X, Y)$, such as the poisson regression, multivariate beta distribution, etc. Each of those are starting points for the parameterization $F$. On a very high level, this points towards the fact that there can't be 'objectively true' (i.e. model invariant) values of parameters. A multivariate beta model has completely different relevant parameters to a normal regression and as such they can't have the same 'objectively true' parameters to be estimated!

Let's use a the normal linear regression model as an example.

A linear regression model allowing for heteroskedasticity and dependent regressors: $$Y = \beta_0 + \beta_1 X + \epsilon, \epsilon \sim N(0, X^\alpha \sigma), cov(X, \epsilon) = \rho$$

The free parameters here are $\theta_F = (\beta_0,\beta_1,\sigma,\alpha,\rho$). The 'true' parameters we are attempting to estimate are $\theta'_F = (\beta'_{0_F},\beta'_{1_F},\sigma_F',\alpha'_F,\rho'_F$).

1. We could restrict this to the heteroskedastic model ($F'$), i.e. set $\alpha = 0$.

Now the free parameters are $\theta_{F'} = (\beta_0,\beta_1,\sigma,\rho$). The 'true' parameters we are attempting to estimate are now $\theta'_{F'} = (\beta'_{0_{F'}},\beta'_{1_{F'}},\sigma_{F'}',\rho'_{F'}$).

However while $\sigma'$ and $\rho'$ may have changed under the new model $F'$, it may be reasonable to expect $(\beta'_{0_{F'}},\beta'_{1_{F'}})=(\beta'_{0_F},\beta'_{1_F})$, which is what we are interested in.

It is now a separate question as to how the estimators will behave in light of this new model. On this I am not clear.

1. We could keep the model as is ($F$), but instead assume that $\alpha = 0$.

In this case we have not changed the model so the true parameters are still $\theta'_F$. If our assumption holds, (for the purpose of illustration suppose $\rho = 0$ also for now), then the OLS estimation will produce unbiased, consistent estimates of the $\beta'_F$ and $\sigma'_F$ parameters (note that we are attempting to find $\sigma'_F$ instead of $\sigma'_{F'}$, as the model hasn't changed. If it doesn't, then var($\hat \beta_{F}$) will be biased.

Note: In the framework thus specified, the above two points are not incompatible, but I assume only one of them is a good representation of how we deal with model specification vs taking assumptions. If someone could verify how to handle this I would be extremely grateful.

The above framework also nicely handles the distinction between predictive regression models and regression models with causal interpretations.

1. Suppose we are working with the restricted homoscedastic model $F’$ (only for ease of exposition). Suppose we impose $\rho = 0$, and this model's associated distributions $F’’(\mathscr S)$ is now subset of $F’(\mathscr S)$.

How we specify the parameters around the error define what the error is, and by extension what the beta parameters are meant to illustrate. In this case, by defining $\rho = 0$, the error necessarily becomes the uncorrelated residual once the expected value of Y is obtained (rather than, say, the contribution of various unobserved factors, which could very well be correlated with X).

As such, the true values of $\beta_{F’’}$ being estimated are the values defining the least squares line in the population distribution $p^*$, which is ($\beta_1=\sigma_{X,Y}/\sigma^2_X$). [Referring here to the population values of the covariance and variance].

1. In contrast, suppose we allow $\rho_{F’}$ to vary.

In this case the error is best interpreted as the contribution of unobserved effects, not as a residual to the least squares in the population. As such, it’s perfectly reasonable for $X$ to correlate with the unobserved effects.

Suppose we use the OLS estimator for $\beta_{F’}$ again. If $\rho_{F’} = 0$ then the estimator will produce an unbiased estimate of $\beta_{F’}$. But it’s unlikely to be, in which case our estimator is biased.

In fact, given the linear model we proposed we can’t estimate all the parameters. The best practice would therefore be add a set of $k$ covariates (i.e. specify a new model, on a larger sample space $\mathscr S’\subset \Bbb R^{d+k}$), such that we are more certain that $\rho=0$. We could then use the value of $\beta$ resulting from that model, restrict this model with that value, and use this to estimate $\rho_{F’}$.

There's a really important distinction there - restricting the values of $\rho$ means that the true values of $\beta$ being uncovered are drastically different, if restricted they are for predictive purposes, if not restricted but only assumed they offer an interpretation which aligns more closely with the common thought that they represent partial derivatives.

Written with StackEdit.