The Stack Overflow podcast is back! Listen to an interview with our new CEO.

New answers tagged

0

If I understand the question correctly, one situation in which OLS breaks down is when covariance matrix is singular (i.e. its determinant is 0), which indicates linear dependency between columns (i.e. variables) in your dataset. Note that if the determinant of a singular matrix is zero, matrix inverse would fail to be calculated.


0

No. One way of thinking of a multiple regression is as the best fitting line in a multidimensional space. There is always such a line, even if the fit is terrible. Look at all possible lines - one will be best. Unless there is exact collinearity, in which case, more than one will be best.


0

You probably mean "unique" solution not inconsistent which is a different concept. Yes, linear regression problem can have degenerated solution, i.e. multiple solutions equally good in a sense of the lowest sum of squared residuals. A simple example is to have two identical variables in the equation, such as a temperature in Fahrenheit and Celsius. The ...


4

A linear system $Ax=b$ can have no-solution case due to the inconsistencies introduced in its equations. But, linear regression does not aim to satisfy $Ax=b$, it merely tries to minimize $||Ax-b||_2$. This is a simple optimization problem, and you'll have your solution, either unique or infinitely many. Intuitively, one can always draw a line as best fit to ...


0

I would recommend you Pedhazur, E. J. (1997). Multiple regression in behavioural research Fort Worth. TX: Harcourt Brace College Publishers. Of all books on regression that I have had a chance to read, the one by Pedhazur is the easiest to understand. He covers the basics of regression, explaining each concept by using easy to follow examples. Also, he ...


1

(On a side note, this is a case of correlated errors -residuals are always correlated). You are considering a very specific form of correlation, equicorrelation. Then note that $$S=\sigma^2[(1-\rho)I+\rho\mathbf i \mathbf i']$$ where $I$ is the identity matrix and $\mathbf i = (1,...,1)'$. It follows that $$\text{Var}\left(\hat \beta_{OLS}\mid X\...


3

I'm not a statistician, but I'll try to weigh-in. Note: edited somewhat in response to comments by OP, to make this response more relevant for the linear regression case. For the p value to be valid, the assumptions for the test need to be met. These may include normality of errors for some tests, but other tests will have different assumptions. That is, ...


0

1) No. I recall normality condition is needed only for small sample (finite sample) cases. For large samples, asymptotic normality holds given some assumptions. (See Halbert White, Asymptotic theory for econometricians. Sorry for non-Statistic biased recommendation) "Large" is ambiguous, but often say something like N>30. I suspect this came from t-dist and ...


1

Reading the whole book on just linear models may be an overkill. This will be especially inefficient if the book is all about "traditional" thinking and does not present a detailed overview of regularization. In many situations, regularization allows one to improve estimation methods for linear models, generating techniques like lasso, ridge regression, ...


1

The key is the use of an offset. Offsets are simple algebraic manipulations to a linear model. In this case, fitting the reduced model with an offset is done by fitting an intercept-only model (since the intercept is a free parameter) and specifying offset(mpg) in the options. This is algebraically equivalent to calculating a new response variable as price-...


1

1) The two approaches actually work in the same way, albeit in the first (multiple covariates) case the system of linear equations that are being solved is greater. 2) The outputs would be different but how different would depend on the correlation. The model with more covariates will invariably explain more variation, although this is not to say that it ...


2

Firstly, this is a very strange model --- it is a binomial GLM with an identity link function, which is a strange link function to choose in this context. In any case, even taking that model as fixed, the explanation in the book seems quite strange to me. Re-arranging the regression equation, and taking the variance of both sides (treating the explanatory ...


0

I wanted to suggest something similar to the approach pointed by @Carsten in the comment (I also never used it, nor heard about it ealier). He refers to the paper by Saqr and Khan (2012) who describe weighted reduced major axis regression. The method assumes that there is a regression model defined in terms of latent variables $\eta_i$ and $\xi_i$ $$\begin{...


0

You often will get multicollinnearity issue with cubes but not the perfect kind. In your case a perfect multicollinnearity can be defined as: $\alpha x+x^3=c$. This is not true by definition. However, when $x<<1$ you get $x-x^3\approx 0$ because $x^3\approx x$. Therefore, sometimes you may get perfect multicollinearity warning or design matrix ...


3

Multicollinearity refers to the situation in which the regressor matrix $Z$ does not have full column rank $k$. This is the case if it is possible to linearly combine the columns $z_1,\ldots,z_k$ into the zero vector with a vector $a=(a_1,\ldots,a_k)'$ other than the trivial zero vector $0$, i.e., $$ a_1z_1+\ldots+a_kz_k=0 $$ for $a\neq0$. If, say, $z_1\...


2

Focus on your second line. First term: $$\sum_j\frac{1}{n^2}=\frac{n}{n^2}=\frac{1}{n}$$ Second term: $$\sum_j\frac{2}{n}\frac{(y_i-\bar{y})(y_j-\bar{y})}{S_{yy}^2}=\frac{2(y_i-\bar{y})}{nS_{yy}^2}\sum_j(y_j-\bar{y})=0$$ Third term: $$\sum_j\frac{(y_i-\bar{y})^2(y_j-\bar{y})^2}{S_{yy}^2}=\frac{(y_i-\bar{y})^2}{S_{yy}^2}\underbrace{\sum_j (y_j-\bar{y})^2}_{S_{...


0

As explained on this page the calculation is not simple but it is typically performed for you by standard statistical software. In general you need to know the relationships among all of the predictors. Write the linear regression model in matrix form: $Y=X\beta+\epsilon$ with $X$ the matrix of data values (rows for observations, columns for predictors ...


1

Treating ordered categorical predictors as continuous is discussed extensively on this page. With a fairly large number of ordered levels for your trial numbers you have the following two extreme approaches: Treating trial number as a continuous predictor will only use up one degree of freedom in the analysis, but at the cost of assuming that a 1-unit ...


1

Parameters of the model that you describe are indeed only $a$ (slope) and $b$ (intercept). Other, more complicated models, have other parameters. By training (fitting) of the model you mean finding the values of these parameters that make the model explain your data (in the sense of minimizing some objective function). Parameters of the model can be ...


2

Overview I don't think that solving classification problems using linear regression is usually the best approach (see notes below), but it can be done. For multiclass problems, multinomial logistic regression would typically be used rather than a combination of multiple regular logistic regression models. By analogy, one could instead use least squares ...


1

Yes, they are equivalent, at least if the following numerical test is generally representative. library(lsmeans) library(dplyr) library(ggplot2) set.seed(1) n <- 5000 ## per condition dat <- data.frame(Sex=rep(c("M", "F"), 2*n), Drug=rep(c("C", "T"), each=2*n)) dat$Sex.Drug <- paste(dat$Sex, dat$Drug, sep=".") meanz <- c("M.C"=0, "M.T"=1, "F.C"...


0

As a newby here, I cannot add a comment, so my answer will be more like an elaborated comment. Think of a single variable in $X$ and $y$, then your estimate $\hat{\beta} = \frac{cov(x,y)}{Var(x)}$. This captures covariation between $x$ and $y$ standardized in terms of variance of $x$. And as you have said, the estimate $\beta$ is very much related to the ...


0

So, you discovered that some of the data you collected cannot be used, because you learnt that for some of the routes you sampled, bikes where not allowed. It would certainly not be meaningful to continue sampling useless data, so your proposal of restricting the sampling frame is the right thing to do, and cannot cause any problems. Previous data is not ...


1

Constant Variance of Error Term (no Heteroscedasticity) holds. Regardless of the fitted values, you observe a constant spread around 0. Heteroscedasticity would look like this: Mean of the error equals 0 also holds, as you rightly point out.


0

Ultimately all observations have been used for feature selection, so the final LOOCV will be biased, as explained in the linked post (highest response, first sentence): "If you perform feature selection on all of the data and then cross-validate, then the test data in each fold of the cross-validation procedure was also used to choose the features and this ...


1

The 95% prediction interval for a new value $Y^\prime = \hat\beta_0 + \hat\beta_1x^\prime$ (not used to find the regression line) corresponding to a new value $x^\prime$ is $$Y^\prime \pm t_.975\,S_{Y|x}\sqrt{1 + \frac 1n + \frac{(x^\prime - \bar x)^2}{S_{xx}}},$$ where $\bar x$ is the mean of the $x_i,$ $S_{xx} = \sum(x_i - \bar x)^2$ (the numerator of the ...


Top 50 recent answers are included