21
$\begingroup$

I am learning right now about regression analysis and the analysis of variance.

In regression analysis you have one variable fixed and you want to know how the variable goes with the other variable.

In analysis of variance you want to know for example: If this specific animal food influences the weight of animals... SO one fixed var and the influence on the others...

Is that right or wrong, pls help me...

|cite|improve this question
$\endgroup$
25
$\begingroup$

Suppose your data set consists of a set $(x_i,y_i)$ for $i=1,\ldots,n$ and you want to look at the dependence of $y$ on $x$.

Suppose you find the values $\hat\alpha$ and $\hat\beta$ of $\alpha$ and $\beta$ that minimize the residual sum of squares $$ \sum_{i=1}^n (y_i - (\alpha+\beta x_i))^2. $$ Then you take $\hat y = \hat\alpha+ \hat\beta x$ to be the predicted $y$-value for any (not necessarily already observed) $x$-value. That's linear regression.

Now consider decomposing the total sum of squares $$ \sum_{i=1}^n (y_i - \bar y)^2 \qquad\text{where }\bar y = \frac{y_1+\cdots+y_n}{n} $$ with $n-1$ degrees of freedom, into "explained" and "unexplained" parts: $$ \underbrace{\sum_{i=1}^n ((\hat\alpha+\hat\beta x_i) - \bar y)^2}_{\text{explained}}\ +\ \underbrace{\sum_{i=1}^n (y_i - (\hat\alpha+\hat\beta x_i))^2}_{\text{unexplained}}. $$ with $1$ and $n-2$ degrees of freedom, respectively. That's analysis of variance, and one then considers things like F-statistics $$ F = \frac{\sum_{i=1}^n ((\hat\alpha+\hat\beta x_i) - \bar y)^2/1}{\sum_{i=1}^n (y_i - (\hat\alpha+\hat\beta x_i))^2/(n-2)}. $$ This F-statistic tests the null hypothesis $\beta=0$.

One often first encounters the term "analysis of variance" when the predictor is categorical, so that you're fitting the model $$ y = \alpha + \beta_i $$ where $i$ identifies which category is the value of the predictor. If there are $k$ categories, you'd get $k-1$ degrees of freedom in the numerator in the F-statistic, and usually $n-k$ degrees of freedom in the denominator. But the distinction between regression and analysis of variance is still the same for this kind of model.

A couple of additional points:

  • To some mathematicians, the account above may make it appear that the whole field is only what is seen above, so it may seem mysterious that both regression and analysis of variance are active research areas. There is much that won't fit into an answer appropriate for posting here.
  • There is a popular and tempting mistake, which is that it's called "linear" because the graph of $y=\alpha+\beta x$ is a line. That is false. One of my earlier answers explains why it's still called "linear regression" when you're fitting a polynomial via least squares.
|cite|improve this answer
$\endgroup$
  • 5
    $\begingroup$ @MichaelHardy While the decomposition of variance into components in regression is often referred to as an analysis of variance table. That is not what statisticians commonly mean by ANOVA. The methods 1) linear regression, 2) analysis of variance and 3) analysis of covariance are categories under the general heading of the general linear model, linear regression involves continuous covariates, ANOVA includes discrete groups only and ANCOVA is a combination of continuous covariates and discrete groups. $\endgroup$ – Michael R. Chernick Aug 17 '12 at 18:25
  • 1
    $\begingroup$ Informally one sometimes speaks that way, and my answer didn't say that, but one should know that (1) least-squares estimation of coefficients is done in either of the two problems (continuous or categorical predictors) and a decomposition of the sum of squares with their corresponding degrees of freedom---an anova table---is also done in either of the two problems. $\endgroup$ – Michael Hardy Aug 17 '12 at 18:51
  • 5
    $\begingroup$ With that concession then you have to conced that there is nothing wrong with my answer. Also the terms ANOVA, ANCOVA and regression are not informal terms. They are very distinctly formal and it is incorrect to tell the OP that ANOVA is the decomposition of variance in regression. The fact that a statistical procedure that someone named anova can do any linear model doesn't prove anything. In SAS proc reg deals only with regression, proc anova deals only with the analysis of variance as I defined it and proc glm is the one that does both. $\endgroup$ – Michael R. Chernick Aug 17 '12 at 19:00
  • 1
    $\begingroup$ ....and in R, "lm(....)" gives regression coefficients in both situations, and "anova(lm(....))" gives the decomposition of the sum of square and degrees of freedom, in both situations. As far as "have to concede" goes, I've put some further comments below your answer. Certainly if you're going to mention logistic regression, it would be clearer if you said that as soon as you're not talking about linear regression, the word "regression" is a very broad terms that can include many things. $\endgroup$ – Michael Hardy Aug 18 '12 at 1:44
  • $\begingroup$ @MichaelHardy Feel free to comment on my question raised on the stats.SE site. I think that your answer and my answer to this question are both correct in a way. I certainly object to my answer being downvoted. I wanted to get the opinions of others in the statistics community about this. $\endgroup$ – Michael R. Chernick Aug 18 '12 at 15:12
5
$\begingroup$

The main difference is the response variable. While logistic regression deals with a binary response in linear regression analysis and also nonlinear regression the response variable is continuous. You have a variable(s) (aka covariate(s)) that have a functional relationship to the continuous response variable. In the analysis of variance the response is continuous but belongs to a few different categories (e.g. treatment group and control group). In the analysis of variance you look for difference in the mean response between groups. In linear regression you look at how the response changes as the covariates change. Another way to look at the difference is to say that in regression the covariates are continuous whereas in analysis of variance they are a discrete set of groups.

|cite|improve this answer
$\endgroup$
  • 6
    $\begingroup$ I'd have taken the question to mean the difference between linear regression and analysis of variance; bringing in logistic regression seems to get away from the topic. However, your last sentence is wrong. Analysis of variance can be done regardless of whether the predictors are discrete or continuous. $\endgroup$ – Michael Hardy Aug 17 '12 at 17:56
  • 1
    $\begingroup$ There are indeed predictors in the analysis of variance. In your example, the predictor is categorical, but it need not be so. Analysis of variance does not only consider problems involving "discrete groups". $\endgroup$ – Michael Hardy Aug 17 '12 at 18:27
  • 3
    $\begingroup$ @MichaelHardy I am taking a step back because when I check my statistical encyclopedias I find reference to the analysis of variance in terms of the decomposition of variance in the general linear model. But the term has two meanings and quite often ANOVA is distinguished from ANCOVA and regression in the way I described. So the OP should be aware of both terms the one that refers to infernece about variance components in the general linear model and the one that refers to the subclass of linear models that involve only discrete groups. $\endgroup$ – Michael R. Chernick Aug 17 '12 at 19:17
  • 2
    $\begingroup$ I think of the usage you're using as informal. It seems strange to mention logistic regression without saying it's just one of a variety of "regressions", when that term is used in the broad sense of estimating an average or predicted value of one variable given another, and then distinguishing that from analysis of variance. But the question of the difference between linear regression models and analysis of variance seems like a more sensible question. But there are often uncertainties about what the original poster intended. $\endgroup$ – Michael Hardy Aug 17 '12 at 21:52
  • 7
    $\begingroup$ Whatever your intentions might have been, I find the "I have a PhD in statistics,..." commentary to be inappropriate. First of all, it does nothing to resolve the issue at hand. Appealing to authority is an oft-used, but very misguided approach to proving things. Appealing to your own authority is even more problematic. It also can be interpreted as showing (inadvertently or otherwise) a lack of respect for @MichaelHardy (the personal you are addressing), who also happens to have a PhD in statistics from a very reputable program. $\endgroup$ – cardinal Aug 18 '12 at 14:41
2
$\begingroup$

The analysis of variance (ANOVA) is a body of statistical method of analyzing observations assumed to be of the structure

$y_i=\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_px_{ip}+e_i,~i=1(1)n$,which are constituted of linear combinations of $p$ unknown quantities $\beta_1,\beta_2,\dots,\beta_p$ plus errors $e_1,e_2,\dots,e_n$ and the {$x_{ij}$} are known constant coefficients with the r.v's {$e_i$} are uncorrelated and have the same mean $0$ and the variance $\sigma^2$(unknown).

i.e. $E(y^{n \times 1})=X\beta,D(y)=\sigma^2I_n$ Where D is dispersion matrix or variance-covariance matrix.

,where the coefficients {$x_{ij}$} are the values of counter variables or indicator variables which refer to the presence or absence of the effects {$\beta_j$} in the conditions under which the observations are taken:{$x_{ij}$} is the number of times $\beta_j$ occurs in the i-th observation,and this is usually $0$ or $1$.In general,in the analysis of variance all the factors are treated qualitatively.

If the {$x_{ij}$} are values taken on in the observations not by counter variables but by continuous variables like $t$=time ,$T$=temperature,$t^2,e^{-T}$,etc,then we have a case of *regression analysis.In general,in regression analysis all factors are quantitative and treated quantitatively.

Mainly,these two are two kinds of Analysis.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ What does the notation $i=1(1)n$ mean? $\endgroup$ – user1157 Aug 18 '12 at 17:05
  • 1
    $\begingroup$ $i=1(1)n$ means $i=1,2,\dots,n$ $\endgroup$ – Argha Aug 19 '12 at 3:54
-1
$\begingroup$

In regression analysis you have one variable fixed and you want to know how the variable goes with the other variable.

In analysis of variance you want to know for example: If this specific animal food influences the weight of animals... SO one fixed var and the influence on the others.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Hello Aiza, welcome to SE. You need to edit this to give more context and make it clear what the question actually is. $\endgroup$ – Jesper Hybel Dec 30 '18 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.