In frequentist statistics, in regression analysis, errors, like random variables, have a distribution. Errors, like parameters, can be estimated and the residuals of the model are their estimates. So it seems that errors have a nature between the definition of random variable and the definition of parameter.
So what is the true nature of errors? What are they exactly?
Can they be considered as bayesian parameters even in frequentist statistics?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ You are right. So errors can be seen as unobserved random variables in frequentist statistics and can be seen as parameters in Bayesian statistics. Is it correct? $\endgroup$– John MCommented May 21, 2015 at 12:40
-
$\begingroup$ Why should errors be parameters in bayesian statistics---they will still be unobservable random variables, I think. $\endgroup$– kjetil b halvorsen ♦Commented May 21, 2015 at 12:50
-
1$\begingroup$ One clarification. Parameters are "unknown constants" in both classical and bayesian statistics. Our model says it is a constant, but the value is not known. In classical statistics, we leave it at that. But in bayesian statistics, the parameters are "modelled as random variables". But, they are still constants we do not know, what have a distribution is not really the parameters themselves, it is our knowledge about them. So the status of the random variable is different---it is "epistemological". $\endgroup$– kjetil b halvorsen ♦Commented May 21, 2015 at 12:55
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Random variables are theoretical concept. They do come from probability distributions, which are usually unknown. So, we try to estimate the parameters of those distributions or other properties such as tails.
In regression modeling both errors and parameter estimates are random variables. The true parameters are not random, but they're unknown usually, so we have to estimate them from the samples. The estimates become random variables then.
Consider a data set with daily measurement of a person's weight: $w_t$. You may want to find out what is the trend $r$ in a daily weight change, and postulate a model $w_t=w_0+rt+\varepsilon_t$. Here, you're assuming that the weight changes by the same amount every day. You estimate the rate $\hat r$ and the variance of the errors $\hat\sigma_\varepsilon$. You can also estimate the errors $e_t$. If your model is right, then these estimates will have some good properties, such as consistency.
However, if your model is misspecified, and the true weight process is $\Delta w_t=r+\varepsilon_t$, then your estimates of rate $\hat r$ and $\hat \sigma_\varepsilon$ will be messed up.
Observable random variables are always looking back. For instance, the weight $w_t$ in my example is an observable random variable. We can measure it. We can describe what was happening to it. However, the moment we have to look forward we must start making assumption, and go into the wild world of unobservables. So, let's say I know your weight history for entire past year. What can I say about your weight tomorrow? I'll have to make some assumptions. For instance, I'll postulate that your weight is at an equilibrium and doesn't change too much. In this case I can take your weight today as my best estimate of your weight tomorrow.
$\begingroup$
$\endgroup$
Linear regression model can be described as
$$ y = f(x) + \varepsilon $$
where you have a fixed part $f(x)$ and random part $\varepsilon$. Instead of single random part, you can have multiple random effects - like in random intercept linear mixed models. Random effects have mean of zero, some variance and are normally distributed. In mixed models random effects are predicted rather then estimated as fixed effects, what makes this kind of models called by some to be "somewhere in between" frequentionist and Bayesian approaches. The "frequentionist" part of such models assumes fixed parameters, while the "Bayesian" part is the random effects (see here for description of differences between Bayesian and freuentionist statistics). So from the estimation viewpoint this distinction makes sense, while in general this is a philosophical debate on the nature of the parameters and statistics in general.
