Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

1
vote
0answers
14 views

Adjusting for ARCH effect in regression analysis

I have the following regression: yt = b0 + b1X1t + b2X2t + b3X3t + e I then saw that e is serially correlated so I modeled the regression with ARIMA errors. However, then I saw that there was an ARCH ...
2
votes
2answers
38 views

OLS with ARMA errors, tip or two

I need a tip or two. I am performing OLS with dynamic factors (4x1 factors each representing a PANEL of 24 series, hence 4 time series). My OLS has autocorrelation in the error so I want to use OLS ...
0
votes
0answers
27 views

Independent variables and moderator variables correlate (three-way interaction)

let's assume I hypothesize: H1: A direct effect of X1 on Y H2: A direct effect of X2 on Y H3: A two-way interaction between X1 and X2 H4: A three-way interaction between X1, X2 and X3 H5: A three-...
3
votes
0answers
21 views

Use of logs in a regression analysis [duplicate]

I'm doing a paper on the impact of FDI on GDP and unemployment in different countries and I'm doing an OLS regression analysis for my data. Would it make sense to use logs for only one variable? Or ...
0
votes
1answer
36 views

Statistically Estimating Fixed and Variable Costs Using Least Squares Regression

I have been tasked with exploring the possibility of estimating the fixed and variable price of a hotel room using a statistical approach. I am by no means an expert in this field, so please ignore or ...
2
votes
0answers
27 views

Normal equations issue

Let $A\mathbf{x}=\mathbf{b}$ be an overdetermined system, with $A$ being an $n \times m $ full-column rank rectangular matrix. Are these minimization problems equivalent? $$ 1) \;\underset{\mathbf{x}...
8
votes
2answers
689 views

Why not use instrumental variable directly as a covariate in the regression?

I know this is a silly question, as I know the theory of instrumental variables and two stage regression. Still, I never saw a clear answer to the following: assume you have endogeneity due to ...
1
vote
1answer
25 views

What happends if we standarise with a slightly shifted mean?

For deep neural networks or any ML algorithms , lets say the mean of my dataset is 5 and instead I normalize with 6 or 4, something slightly off. What affect does this have on learning?
1
vote
0answers
37 views

Least Squares Fit Covariance Matrix

How do you perform a least squares fit for $\Sigma$ in the equation $v = u^T\Sigma u$? The term $v$ is a vector of observed variances of a projected Gaussian distribution. The matrix $u$ is made of ...
2
votes
0answers
22 views

Loess preprocessing for robust quadratic regression?

It seems to me that we could fit a quadratic curve to the predictions of a LOESS curve in order to obtain a parametric model approximating the non-parametric LOESS model. Is this something that is ...
4
votes
1answer
67 views

Gauss Original Paper

I am looking for Gauss's 1809 paper in which he introduced least squares regression, MLE and the gaussian distribution. I cannot find it online. Can someone tell me where I may find it?
0
votes
1answer
33 views

Need help with Least Squares Estimator

I am interested in a One-Way ANOVA model:                           &...
2
votes
1answer
25 views

Meaning of $r^2n$ for a large dataset

I have a large astronomical dataset, showing the OLS regression value $r$ between two continuous variables, and the number of observations $n$. My research supervisor has told me to include the value ...
2
votes
0answers
35 views

LASSO solution when $p \gg n$

We know that when $p \gg n$ we don't have a unique least squares solution. But, in the same scenario, does LASSO have a solution? LASSO uses the least squares term as part of its cost function (i.e.,...
0
votes
1answer
45 views

In OLS regression in R, how can I tell my model an observation is in my dataset more than once? [closed]

First of all an example dataset, both a picture and the R code to generate it. Here, X1 represents the ID of the individual / observation, ...
1
vote
0answers
9 views

Sample selection in difference-in-differences

I have a dependent variables with a significant amount of zeros (and the share of zeros is different between the control and treatment groups, and changes between the pre- and post-treatment periods). ...
1
vote
2answers
32 views

Question about expectation in OLS?

Consider the linear model $$y_i = x_i^T\beta + \epsilon_i.$$ In ordinary least squares it is assumed that the errors satisfy $E[\epsilon_i]=0$. This implies that that $\dfrac{X^T\epsilon}{n} \to 0$ ...
0
votes
2answers
43 views

Proof for “The sum of the observed values $Y_i$ equals the sum of the estimated / fitted values $\hat Y_i$”

I needed some help trying to understand why the sum of the observed values $Y_i$ equals the sum of the estimated values $\hat{Y}_i$.
0
votes
0answers
41 views

OLS assumptions: prediction vs inference

Taking an OLS model (actually, is it a "model" or an "estimator"?) as an example, there are several assumptions (such as strict exogeneity and spherical errors) which are important for the consistency ...
1
vote
1answer
32 views

Multiple linear regression: observations with 2 or more values per factor / categories - a problem?

Is it a problem for linear regression (lm in R) to have observations that have multiple values for a given factor? For example, I have the weekly average sales <...
2
votes
0answers
52 views

Does Ridge regression always yield lower MSE value compared to OLS?

First time asking question in StackExchange after being a long time lurker. I am trying to analyze some simple data using R. I found the best lambda for Ridge regression using cross-validation, then ...
0
votes
0answers
25 views

Intuition of variance (in the context of linear regression)

I was studying linear regression lately and checking the assumptions for Ordinary Least Squares method for the regression problem. I was not sure about the intuition behind the difference of squares ...
0
votes
0answers
18 views

$R^2$ associated with a restricted LS estimator is never larger that that of the unrestricted LS estimator

Prove that the $R^2$ associated with a restricted least squares estimator is never larger than that associated with the unrestricted least square estimator. So, I tried doing this question but I can'...
4
votes
2answers
148 views

Can Adjusted R squared be equal to 1?

I have a dataset with around 15 independent variables. I am using a multi-regression model to fit the dataset. For model selection, I am using a backward elimination procedure based on the p-values. ...
0
votes
0answers
7 views

“Attribution” of Changes in Fitted Value - Log-Log OLS

I have a regression that follows the below form: $\ln(y) = \alpha + \beta_1 x_1 + ... \beta_n x_n + \epsilon$ The independent variables are a mix of non-transformed and log-transformed series. For ...
0
votes
0answers
36 views

Interpreting OLS regression with continuous interaction?

I'm a beginner when it comes to statistics and I have a question about how to interpret the results of this OLS model and specifically the interaction of two continuous variables. The following ...
1
vote
0answers
32 views

Why are polynomials of every degree significant in my linear regression?

My dataset consists of 1 million observations. I am running a linear regression of Y on X (mean-centered and scaled variable), trying to demonstrate that there is a curvilinear relationship between ...
2
votes
1answer
116 views

How bad is Cholesky decomposition for OLS?

I've heard that I should never use Cholesky decomposition for OLS as it's super unstable, so I thought I'd try a very simple example to see how bad it could be. First let's set up the problem in R: <...
0
votes
1answer
38 views

Selecting appropriate likelihood during non-linear regression

When performing regression to fit a function, $f (x,{\bf \beta})$, to a set of observed data, $y_i(x_i)$, we are seeking to optimize the parameters, $\beta$, of the fitting function, to minimize some ...
0
votes
1answer
33 views

Why isn't the least squares predictor $\Phi(\Phi^\top\Phi)^{-1}\Phi^\top$ simply the identity matrix? [duplicate]

Given target vector $y$. Want to predict it using linear regression $h(w) = w^Tx$ Let $\Phi$ be the least squares matrix, i.e., $\Phi = \begin{bmatrix} x_1^\top \\ \vdots\\ x_n^\top \end{bmatrix}$ ...
1
vote
3answers
121 views

In Bishop's textbook, is the example of overfitting exaggerated?

Here, the data $x$ are randomly generated, and $t$ are generated by running $x$ through a function $\sin(2\pi x)$, then Gaussian noise is added. Bishop's text then tries to fit those data using a ...
0
votes
1answer
27 views

Is there an alternative to R squared to compare goodness of fits of different datasets? Slope makes them incomparable

I'm fitting the degradation of a signal. Some instruments degrade faster than others, so the slope varies a bit. This makes it difficult to compare how good the fits are relative to eachother. See ...
2
votes
1answer
90 views

Is it valid to iterate over every permutation of a regression specification and compute an “average significance?”

I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner. Let's say I run an OLS model and the results come back significant. There is discussion as to ...
0
votes
1answer
65 views

Derivation of Formula for linear regression with data points forming a circle

Suppose we have one dimensional data with following conditions.1. The value of this single feature lies in the closed interval [-a,a].2. The associated target value with each data also lies in the ...
0
votes
0answers
16 views

How would I figure out - TSS of Y, Yi and XiYi from the following information?

I came across an analogous question when revising and have no clue how to approach it. The Given information is ΣXi= 20, ΣYi=40 Σ(Xi-x̅)²= 40, Σ(Xi-x̅)(Yi-nȳ)=20 and n=20. The question requires ...
0
votes
0answers
36 views

Comparing the output and fit of an OLS and a Tobit model, and comparing Tobit models

I am trying to estimate a quite complicated model (many variables with different structures), with a limited dependent variable, which ranges from 0-100% with about 45% of the sample having an ...
0
votes
0answers
25 views

Cannot replicate a 95% CI using: Beta ±1.96*SE [duplicate]

I am having a lot of difficulty figuring out why I cannot calculate a 95% CI by hand with one set of data; but I can using the same process with a different set of data. Drawing on OLS results; I am ...
0
votes
1answer
28 views

Cluster-robust standard errors in panel data analysis

In a simple panel data analysis with data on 64 firms over 8 years, I use cluster-robust standard errors (at the firm level) to evaluate significance of coefficients. I observe important differences ...
0
votes
0answers
19 views

upward/downward bias of negative variable

If I have a variable that, considering some omitted factor, should have fallen by a higher amount than when it is not there - would that be a downward bias? I.e. the decrease is not large enough, so ...
1
vote
0answers
37 views

Degrees of freedom in OLS regresison vs Bootstrap

I understand that in OLS, the degrees of freedom for estimating the variance of the residuals is n-q-1. We loose q+1 degrees because they are "used" to analytically determine the q parameters and 1 ...
1
vote
1answer
26 views

How to motivate a POLS?

How would you justify the usage of Pooled OLS regression instead of Fixed effects? If I am calculating just correlation between two phenomena, may I get rid of these fixed effects? May I choose to ...
2
votes
1answer
68 views

Does $\hat{\beta}=(X′X)^{−1}X′ \ Y$ simplify further?

As far as I know $(AB)^{-1}=B^{-1}A^{-1}$ and matrix multiplication is associative so: $$(X′ X)^{−1}X′=(X^{-1})(X')^{-1}(X')=X^{-1}.$$ What am I doing wrong?
1
vote
1answer
40 views

How to obtain the inverse of the variance covariance matrix of GLS (Random Effects Model)

In the standard GLS set up how do you find the inverse of the variance covariance matrix? $$y _ { i t } = \beta _ { 0 } + x _ { i t } ^ { \prime } \beta + \alpha _ { i } + u _ { i t } \hspace{35pt} u ...
0
votes
1answer
31 views

Log-normalization of predictors

I have the following dependent and independent variables for my linear regression model. Since they are all in different scales (some of the are % others continuous variables), I was suggested to take ...
0
votes
1answer
29 views

Does a panel regression model make sense for my data?

This might be a bit of a newbish question, but I recently picked up a forecasting project at my job, and I'm trying to figure out whether it makes sense to run a panel regression like a Fixed Effects ...
0
votes
0answers
32 views

Why does the variance of the OLS estimator goes to zero as n increases while it is a random variable?

I have a question regarding the OLS estimator. In my class, the teacher asked us to critic this comment: “We write the unconditional variance of the OLS as $V(\hat{b}^{ols}) = 1/N 𝜎^2 E(X_i'X_i )^{−...
0
votes
0answers
18 views

Dynamic Pooled OLS: does it make sense?

I have data on 25 countries for about 35 years. I want to estimate a POLS as I am not looking for a causal interpretation but just sttistical correlation between two phenomena. That's why I will not ...
0
votes
0answers
10 views

On dichotomizing a continuous variable and how it affects MSE of OLS

Recently I have learned about the practice of dichotomizing a continuous independent variable (or maybe even discretize it into more than 2 categories), and then run a predictive model (e.g. multiple ...
0
votes
1answer
13 views

Significance of the sum of the main effect and interaction term

Consider a simple linear regression with an interaction term: $Y=b_0 + b_1X +b_2Z+b_3XZ$ where $X$ is continuous and $Z$ is a dummy. I want to find out whether $X$ has a significant impact on $Y$ ...
0
votes
0answers
32 views

How do we compute OLS coefficients from Sum of Squares in multiple linear regression?

Suppose we have a variable y which is dependent on 2 variables say x1 and x2. Then I can understand how we can compute Sum of squares due to x1 and x2. For instance we may have Type I Sum of squares ...