Relationship between system of equations and statistical models

Question

I am studying statistics and I am starting to see some parallels between statistical models and systems of equations I used to solve in high school. For example, suppose that on one hand we have a linear system of equations x + 3y = 4, 2x + 4 =2 and on the other hand we have 10 observations of two normal variables W and Z that we assume come from the model W = b0 + b1Z + error (standard linear regression). In both cases we have:

• In both cases there are unknowns that we are trying to find (x, y, b0, b1, var(error))
• In both cases there ares some constraints for the unknowns (W, Z ~ Normal, x,y belong to the real or complex numbers)
• Once the system is formulated there exist deterministic algorithms to solve it.

My question is the following: is there a general framework that encompasses both problems? For example:

1. Is a system of equations a particular case of a statistical model, if we consider degenerated distributions (e.g., if we interpret X=3 as P(X=3)=1)?
2. Is there a parallel between a system of equations having infinite solutions and a statistical model to be identifiable? What about a system of equations having no solution, what would be the equivalent in a statistical model?
3. I have the feeling that the concept of degrees of freedom is important in this parallel, but I don't know exactly how.

Answer 1

Obligatory xkcd:

The difference is that with system of equations you are looking for a solution, while in statistics you are trying to estimate and approximate something given the data. There is a notion of random variables and noise, contrary to solving equations. You are not looking for a solution, but for an estimate that is assumed to be a random variable. Same applies to the constraints you mention, $X \sim \mathcal{N}(\mu, \Sigma)$ is a fuzzy constraint (under the "constraint" any value on real line is possible, some are just more likely), as compared to hard constraints in solving equations. Those are subtle, but important differences.

For a general framework, as you noticed, you can treat everything as a random variable and use statistical methods even for things that are deterministic. One example is Laplace who over two hundred years ago used Bayesian approach to estimate the mass of Saturn.

Finally, notice that the problems are not exchangeable. When solving the equations, you are given the equations, with statistics you are given the data, but the equations are not known, so your problem is broader than solving them.

Answer 2

1. Is a system of equations a particular case of a statistical model, ...

The estimation of a statistical model based on a linear system of equations is called linear regression.

2. Is there a parallel between a system of equations having infinite solutions and a statistical model to be identifiable? What about a system of equations having no solution, what would be the equivalent in a statistical model?

You can regard the number of parameters $p$ and the number of measurements $n$.

(these $n$ and $p$ can be seen as parallel to the number of equations $n$ and number of unknowns $p$)

Often we have more measurements than parameters. For the equivalent linear equation there might be no solution. But in the case of least squares regression this is no problem. We search for the solution that fits the best.

If there are an infinite number of solutions, $p\geq n$, then additional constraints are used. See for instance ridge regression where the solution with the smallest length of the parameter vector is chosen.

3. I have the feeling that the concept of degrees of freedom is important in this parallel, but I don't know exactly how.

I don't know how to answer this.