I am studying statistics and I am starting to see some parallels between statistical models and systems of equations I used to solve in high school. For example, suppose that on one hand we have a linear system of equations x + 3y = 4, 2x + 4 =2 and on the other hand we have 10 observations of two normal variables W and Z that we assume come from the model W = b0 + b1Z + error (standard linear regression). In both cases we have:
- In both cases there are unknowns that we are trying to find (x, y, b0, b1, var(error))
- In both cases there ares some constraints for the unknowns (W, Z ~ Normal, x,y belong to the real or complex numbers)
- Once the system is formulated there exist deterministic algorithms to solve it.
My question is the following: is there a general framework that encompasses both problems? For example:
- Is a system of equations a particular case of a statistical model, if we consider degenerated distributions (e.g., if we interpret X=3 as P(X=3)=1)?
- Is there a parallel between a system of equations having infinite solutions and a statistical model to be identifiable? What about a system of equations having no solution, what would be the equivalent in a statistical model?
- I have the feeling that the concept of degrees of freedom is important in this parallel, but I don't know exactly how.