Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least ...

learn more… | top users | synonyms (1)

0
votes
0answers
48 views

Why is the MEAN taken in Simple Linear Regression?

Question Why take the mean of the squared residuals? wouldn't it be simpler and produce the same result ( parameters $\theta_0$ and $\theta_1$ ) if you just minimised the sum of the square ...
1
vote
0answers
9 views

Continuous, sometimes negative residuals from Poisson-distributed variable - how to analyze?

It's been years since I've taken my grad school stats courses, and it's a subject I struggle with, so bear with me. I am attempting to analyze a dataset containing two Poisson-distributed variables ...
0
votes
0answers
9 views

OLS and GLM - residuals plots and lack/goodness of fit statistics in JMP

It's been years since I've taken my grad school stats courses, and it's a subject I struggle with, so bear with me. I'm attempting to fit some sort of linear model to several response variables. In ...
13
votes
1answer
833 views

Why does this regression NOT fail due to perfect multicollinearity, although one variable is a linear combination of others?

Today, I was playing around with a small dataset and performed a simple OLS regression which I expected to fail due to perfect multicollinearity. However, it didn't. This implies that my ...
0
votes
0answers
12 views

References for MMSE estimator for forecasting future value

As the title says, I have been looking for articles on minimum mean squared error (MMSE) estimator for forecasting. But so far I cannot find any. Does anyone know some articles on this subject?
1
vote
1answer
34 views

Total least squares minimization

I've seen some lecture notes about the Total Least Squares, which state the following: Suppose we have a linear system $Y=XB$, which may be inconsistent. Now change this system to ...
3
votes
3answers
73 views

Why does the sum of residuals equal 0 from a graphical perspective?

I've seen the proof for why in least squares regression the sum of residuals is always equal to 0, and I kind of understand why from that algebraic perspective. Basically, you're finding the minimum ...
1
vote
3answers
43 views

What type of data would have non-normal errors?

I'm trying to understand the assumptions for an OLS model. I get that the error term should be normally distributed if we want easy-to-calculate confidence intervals for our coefficient estimates. ...
0
votes
0answers
14 views

Geometric interpretation of MLR

$\vec{y}=\mathbf{X_{n\times p}}\vec{\beta}+\vec{\epsilon}$ We know that $\hat{\beta}^{\text{LSE}}=(X^TX)^{-1}X^Ty$ If all the dimensions are orthogonal, we can obtain that $\beta_j=\frac{\langle ...
0
votes
0answers
21 views

Minimum mean squared error linear combination of random variables

Consider the following objective function: $$ \mathbb{E}((Y-X\beta)^2)\rightarrow \min_\beta $$ where $Y$ and $X$ are (generally not independent) random variables and $\beta$ is a constant. That is, ...
0
votes
3answers
69 views

Why do we do hypothesis testing on estimates of linear regression?

I was reading about linear regression and what I understood is that once we minimize OLS equation we get the beta parameters. Its just like solving a normal equation to get the unknowns. Then why do ...
0
votes
0answers
10 views

Intuition for Normalized Squared Loss error function?

In terms of optimization squared loss is perhaps the most common error function used for regression. I've seen another function named "Normalized Squared Loss" mentioned, described as The ...
0
votes
0answers
12 views

Including a constant in the regression when there is no constant in the DGP [duplicate]

Suppose I generate the variable y as follows: y=1*x1+2*x2+3*x3+u Should a constant be included in the OLS regression, even if there is no constant in the DGP, and if so why? Thanks.
0
votes
0answers
22 views

Generating B-Splines for LSQ-Estimation; which spline to remove

In Least Square Regression we need k-1 parameters per factor (plus intercept) The same should be true for B-Spline multiple regression in additive models. Thus I need k-1 splines, so that the splines ...
0
votes
0answers
33 views

assumptions OLS regression met?

When I was checking the assumptions for a normal distribution these plots appeared: I was wondering what I could conclude from these... the dependent variable is winsorized between 0 and 1 ( and ...
1
vote
1answer
26 views

cross-sectional data : regression model

I'm using a cross-sectional dataset ( just one point in time) and want to use a regression. In literature, mostly an OLS regression or a regression with fixed or random effects were used. However, I ...
5
votes
2answers
130 views

For least squares estimation, what is the difference between using the estimator $\hat{\beta} = X^{T}Y$ vs $\hat{\beta} = (X^{T}X)^{-1}X^{T}Y$

For least squares estimation, the estimator $\hat{\beta} = X^{T}Y$ is an unbiased estimator while $\hat{\beta} = (X^{T}X)^{-1}X^{T}Y$ is also an unbiased estimator given that $X$ is well-defined. Is ...
3
votes
1answer
39 views

Is it possible to determine how many effects I can estimate in a least squares problem just by looking at the correlation matrix?

I currently have a model matrix $X$ with $6$ columns, which is being used for a factorial design problem, with each column associated with an effect. The ultimate goal is to be able to estimate as ...
0
votes
3answers
69 views

Express correlation matrix of $X$ in terms of $X^{T}X$ (in the OLS context)

In least squares estimation where $Y = \beta X$, how can we find the correlation matrix of $X$ in terms of $X^{T}X$? It seems that $X^{T}X$ is very close in structure to the correlation matrix, but ...
0
votes
0answers
17 views

dummy variable with only few observations of value 1

I'm intending to do an OLS regression but when looking at the kurtosis of my dummy variable ( and another categorical variable) it's way too high (about 127). Looking at the data I can see that the ...
1
vote
1answer
51 views

When to use PCA vs. OLS

Let's say I want to build a factor model to explain (excess) stock returns--think Fama-French, for instance. Obviously one could use OLS to fit a model, but I've seen PCA used as well. What are the ...
4
votes
0answers
22 views

Best-fit plane through a set of lines?

A linear model estimates the best fit line from a set of points, often through minimization of the sum of squared residuals. By analogy, is there any established method (possibly implemented in R) ...
0
votes
0answers
10 views

Prediction bias correction factor for OLS versus REML regression

Can Newman's correction factor for prediction bias following logarithmic transformation of variables under least squares regression be applied to log-transformed models derived with restricted maximum ...
2
votes
1answer
70 views

Binary treatment with covariates

I am stuck on problem, asking me to show that in the model: $$ Y = \beta_0 + \beta_1T + \Gamma X + u $$ Where Y is the outcome, T is a treatment indicator and X are a set of controls (pre ...
0
votes
1answer
33 views

What type of regression for qualitative dependent variable

What econometric analysis can be used to study the impact of quantitative independent variables on a qualitative dependent variable? Basically, how do I do conduct a multiple variable regression where ...
3
votes
1answer
138 views

ANOVA vs multiple linear regression? Why is ANOVA so commonly used in experimental studies?

ANOVA vs multiple linear regression? I understand that both of these methods seem to use the same statistical model. However under what circumstances should I use which method? What are the ...
3
votes
0answers
51 views

Why use quantile regression instead of splitting the data in quantiles and calculating multiple linear regressions?

Why use quantile regression instead of splitting the data in quantiles and calculating multiple linear regressions? What are the advantages and disadvantages of these methods? As far as I understand ...
1
vote
2answers
76 views

Regression and Time Series

I was posed a problem by a colleague that i am struggling with. He is interested in the relationship between 10 variables and a single dependent, continuous variable. This could simply be an OLS ...
3
votes
1answer
34 views

OLS interpretation - if Y increases due to X increasing, can I say that if X decreases, Y will decrease with the same magnitude?

$$ \ln(y)=b + 0.25\ln(X) + \epsilon $$, i.e. for $10\%$ increase in $X$, we observe about $2.5\%$ increase in $Y$. Can I claim that if I reduce $X$ by $10\%$, then Y will drop $2.5\%$? Can such ...
0
votes
1answer
27 views

Choosing IRLS over gradient descent in logistic regression

I am currently reading Bishop [1] and got confusion on why should we take IRLS (Iterative Re-weighted Least Square) as it seems that using gradient descent that with one derivative at a time would ...
3
votes
1answer
39 views

Why should we use IRLS in logistic regression?

I am really confused on why should we take IRLS as it seems that using gradient that with one derivatives at a time would solve the problem, what is the meaning of introducing Hessian matrix? Or did I ...
3
votes
0answers
25 views

Linear model with biased estimator

Consider a linear regression model. Suppose that the estimator $\hat{\beta}$ for the vector of the parameters of the model $\beta$ is, for some reasons, biased. As a consequence: ...
1
vote
2answers
76 views

What is bias in Bias-Variance Tradeoff?

In the book An Introduction to Statistical Learning, chapter 2, it is mentioned that Expected MSE has 3 components: ...
2
votes
2answers
86 views

Putting quadratic effect into regression model

So I have a least squares model which tries to identify changes in income due to different levels of experience, education, etc. But in my model the dependent variable is 'income' and among the ...
3
votes
3answers
141 views

What is/are the “mechanical” difference between multiple linear regression with lags and time series?

I'm a graduate from business and economics who's currently studying for a master's degree in data engineering. While studying linear regression (LR) and then time series analysis (TS), a question ...
5
votes
2answers
87 views

How do regression results change after standardization, as a general rule?

Based on the simulation below, it appears that standardizing all variables in a data set affects OLS results in the following ways: Coefficient estimates change Standard errors change P-values ...
1
vote
0answers
32 views

conditional variance is equal to the variance

I know that a question with the same topic has been asked a few years ago (How do you derive the conditional variance for $s^2$, the OLS estimator of $\sigma^2$?) however, My question is kind of ...
1
vote
1answer
52 views

Estimating Cointegration vector

I am learning about the concept of cointegration and I found in various places the following claim about estimating the cointegration vector using OLS which is: Despite the fact that the OLS estimator ...
1
vote
0answers
14 views

Can I run an OLS Regression on the system generated VECM Equation?

This I am planning to do to obtain the p values associated with the coefficients , also to run a Wald Test to check restrictions on the coefficients . Is this correct?
1
vote
0answers
30 views

High $R^2$ on Ordinary least squares model with violated assumptions. Is it good?

Recently I tried to fit some points which (from the plot) seems linearly distributed. The fit result (in R) is: ...
0
votes
1answer
46 views

Heteroscedasticity in linear regression, there is a a pattern. What to do?

I'm modelling the behaviour of two variables with a linear regression. Since I saw (and believe) there is a multiplicative behaviour I transformed the dependent and independent variables taking the ...
1
vote
0answers
6 views

Multiplicative Measurement Error in Response Variable

I understand that taking log of the multiplicative error model transforms it into the additive error model. Let $y'$ be the observed response variable, with $y$ being the true response variable and ...
0
votes
0answers
14 views

TReading Residual Plot: Omitted Variable Bias of Dummy Variable

I have a plot of the residuals versus fitted values of an OLS model such that the shape of the plot are two identical randomly scattered clouds, one above and one below the $\hat{e} = 0$ line. Can I ...
0
votes
1answer
26 views

Same estimate of least squares regression equation

This was one of the question in my test which I was not able to attempt. Question: Let $\left\{ \left. \left(x_{i,1}, x_{i,2}, \ldots, x_{i,d}, y_i\right) \; \right\rvert \; i = 1, \ldots, n ...
2
votes
3answers
104 views

Comparing OLS versus WLS residual standard error, is the smaller the better?

For a linear regression model I tried on a dataset, when I fitted OLS, the output is as follows: ...
1
vote
1answer
34 views

Finding optimal beta when there are multiple different errors

I am working on an econometrics model that I'm not sure how to approach. I've made a utility function where the weights have noise as well. In short it's: $$ y_i = (\beta + \epsilon_i)x_i + u_i $$ ...
0
votes
1answer
36 views

How to solve multicollinearity in OLS regression with correlated dummy variables and collinear continuous variables?

I have predicted an ecological variable using OLS regression which showed the model accounts for more than 72% of the variance in the dependent variable (DV). However, I am also interested in which ...
1
vote
0answers
16 views

What does it mean when least squares and instrumental variables give similar results?

What can said when for some data set the result of a least squares regression is extremely close to the result of an instrumental variable regression?
0
votes
0answers
7 views

How dummy variable interactions affect least squares estimates

I have a dataset on earnings data and running various regressions have found that ethnicity is not a strong indicator for women but it is for men whilst BMI is significant for women but not for men. ...
0
votes
1answer
41 views

Least Squared Regressions on Partitioned Data

Say I have 800 (X,Y) data points, and I do a LSQ fit and get y = mx+b Then I think to myself, of the 800 data points, 500 are males and 300 are females, so I ...